We've been making changes to how we organize and display our docs. Our work isn't done but we'd love your feedback.
Getting started
Basics
Domains & Origins
Performance

Configuration
Basics
Conditions
Dictionaries
Domains & Origins
Request settings
Cache settings
Headers
Responses
Performance
Custom VCL
Image optimization
Video

Security
Access Control Lists
Monitoring and testing
Securing communications
TLS
Web Application Firewall

Integrations
Logging endpoints
Non-Fastly services

Diagnostics
Streaming logs
Debugging techniques
Common errors

Account info
Account management
Billing
User access and control

Reference

    Log streaming: Google Cloud Storage

      Last updated May 31, 2019

    Fastly's Real-Time Log Streaming feature can send log files to Google Cloud Storage (GCS). GCS is an online file storage service used for storing and accessing data on Google's infrastructure. One advantage of using GCS is that you can use Google BigQuery to analyze the log files.

    Prerequisites

    Before adding GCS as a logging endpoint for Fastly services, you will need to:

    Creating a GCS bucket

    You can create a new GCS bucket to hold the logs, or you can use an existing bucket. Be sure to note the name of the bucket as you will need it later. To learn how to create a GCS bucket, see Google's guide on creating a bucket.

    Creating a service account

    GCS uses service accounts for third-party application authentication. You will need to create a new service account on Google's website. To learn how to create a service account, see Google's guide on generating a service account credential. When you create the service account, be sure to set the Key Type to JSON.

    Obtaining the private key and client email

    After you create the service account, a JSON file will be downloaded to your computer. This file contains the credentials for the GCS service account you just created. Open the file with a text editor and make a note of the private_key and client_email.

    Enabling the Google Cloud Storage JSON API

    To ensure the Fastly logs are sent to your GCS bucket, you need to enable the Google Cloud Storage JSON API. For more information, see Google's instructions for activating the API.

    Adding GCS as a logging endpoint

    Follow these instructions to add GCS as a logging endpoint:

    1. Review the information in our Setting Up Remote Log Streaming guide.
    2. Click the Google Cloud Services logo. The Create a Google Cloud Storage (GCS) endpoint page appears.

      The  Google Cloud Storage endpoint page

    3. Fill out the Create a Google Cloud Storage (GCS) endpoint fields as follows:
      • In the Name field, type a human-readable name for the endpoint.
      • In the Log format field, optionally type an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. See our guidance on format strings for more information.
      • In the Timestamp format field, optionally type a timestamp format for log files. The default is an strftime compatible string. Our guide on changing where log files are written provides more information.
      • In the Email field, type the client_email address listed in the JSON file associated with the service account you created on Google's website.
      • In the Bucket name field, type the name of the GCS bucket in which to store the logs.
      • In the Secret key field, type the private_key value listed in the JSON file associated with the service account you created on Google's website. We strip out the JSON newline escape characters for you so don't worry about removing them.
      • In the PGP public key field, optionally type a PGP public key that Fastly will use to encrypt your log files before writing them to disk. You will only be able to read the contents by decrypting them with your private key. The PGP key should be in PEM (Privacy-Enhanced Mail) format. See our guide on log encryption for more information.
      • In the Period field, optionally type an interval (in seconds) to control how frequently your log files are rotated. This value defaults to 3600 seconds.
    4. Click the Advanced options link of the Create a Google Cloud Storage (GCS) endpoint page and decide which of the optional fields to change, if any.

      the advanced options on the create a GCS endpoint page

    5. Fill out the Advanced options of the Create a Google Cloud Storage (GCS) endpoint page as follows:
      • In the Path field, optionally type the path within the bucket to store the files. Specify a directory by ending the path with a trailing slash (/). Leaving this field empty saves the files in the bucket's root path. Our guide on changing where log files are written provides more information.
      • In the Select a log line format area, select the log line format for your log messages. Our guide on changing log line formats provides more information.
      • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about waf_debug_log.
      • In the Gzip Level field, optionally type the level of gzip compression you want applied to the log files. You can specify any whole number from 1 (fastest and least compressed) to 9 (slowest and most compressed). This value defaults to 0 (no compression).
    6. Click Create to create the new logging endpoint.
    7. Click the Activate button to deploy your configuration changes.
    Back to Top