LOG IN SIGN UP
Documentation

Log streaming: Google Cloud Storage

Fastly's Real-Time Log Streaming feature can send log files to Google Cloud Storage (GCS). GCS is an online file storage service used for storing and accessing data on Google's infrastructure. One advantage of using GCS is that you can use Google BigQuery to analyze the log files.

Prerequisites

Before adding GCS as a logging endpoint in the Fastly application, you will need to register for a GCS account, create a bucket and service account on Google's website, and obtain the private_key and client_email from the JSON file associated with the service account.

Creating a GCS bucket

You can create a new GCS bucket to hold the logs, or you can use an existing bucket. Be sure to note the name of the bucket as you will need it later. To learn how to create a GCS bucket, see Google's guide on creating a bucket.

Creating a service account

GCS uses service accounts for third-party application authentication. You will need to create a new service account on Google's website. To learn how to create a service account, see Google's guide on generating a service account credential. When you create the service account, be sure to set the Key Type to JSON.

Obtaining the private key and client email

After you create the service account, a JSON file will be downloaded to your computer. This file contains the credentials for the GCS service account you just created. Open the file with a text editor and make a note of the private_key and client_email.

Adding GCS as a logging endpoint

After Fastly support has enabled the GCS endpoint for your account, follow these instructions to add GCS as a logging endpoint:

  1. Review the information in our Setting Up Remote Log Streaming guide.
  2. On the Logging page, find the Google Cloud Storage endpoints area and click New.

    The new GCS endpoint button

    The New Google Cloud Storage endpoint window appears.

    The New Google Cloud Storage endpoint window

  3. Fill out the New Google Cloud Storage endpoint fields as follows:

    • In the Name field, type a human-readable name for the endpoint.
    • In the Bucket Name field, type the name of the GCS bucket in which to store the logs.
    • In the Email field, type the client_email address listed in the JSON file associated with the service account you created on Google's website.
    • In the Secret Key field, type the private_key value listed in the JSON file associated with the service account you created on Google's website. We strip out the JSON newline escape characters for you so don't worry about removing them.
    • In the Path field, optionally type the path within the bucket to store the files. Specify a directory by ending the path with a trailing slash (/). Leaving this field empty saves the files in the bucket's root path.
    • In the Period field, optionally type an interval (in seconds) to control how frequently your log files are rotated. This value defaults to 3600 seconds.
    • In the Gzip Level field, optionally type the level of gzip compression you want applied to the log files. You can specify any whole number from 1 (fastest and least compressed) to 9 (slowest and most compressed). This value defaults to 0 (no compression).
    • In the Format String field, optionally type an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. See our guidance on format strings for more information.
  4. Click Create to create the new logging endpoint.