Compute@Edge log streaming: Google Cloud Storage

Fastly's Real-Time Log Streaming feature for Compute@Edge services can send log files to Google Cloud Storage (GCS). GCS is an online file storage service used for storing and accessing data on Google's infrastructure. One advantage of using GCS is that you can use Google BigQuery to analyze the log files.

Prerequisites

Before adding GCS as a logging endpoint for Fastly Compute@Edge services, you will need to:

• Register for a GCS account.
• Create a bucket and service account on Google's website.
• Obtain the private_key and client_email from the JSON file associated with the service account.
• Enable the Google Cloud Storage JSON API.

Creating a GCS bucket

You can create a new GCS bucket to hold the logs, or you can use an existing bucket. Be sure to note the name of the bucket as you will need it later. To learn how to create a GCS bucket, see Google's guide on creating a bucket.

Creating a service account

GCS uses service accounts for third-party application authentication. You will need to create a new service account on Google's website with the role of Storage Object Creator and make sure you've added it as a member of the GCS bucket you created. To learn how to create a service account, see Google's guide on generating a service account credential. When you create the service account, be sure to set the Key Type to JSON.

If you elect to use Google service account impersonation in order to avoid storing keys with Fastly you may use this same service account for that purpose. Our guide to creating an Google IAM role provides further details on this feature.

Obtaining the private key and client email

After you create the service account, a JSON file will be downloaded to your computer. This file contains the credentials for the GCS service account you just created. Open the file with a text editor and make a note of the private_key and client_email.

Enabling the Google Cloud Storage JSON API

To ensure the Fastly logs are sent to your GCS bucket, you need to enable the Google Cloud Storage JSON API. For more information, see Google's instructions for activating the API.

Adding GCS as a logging endpoint

Follow these instructions to add GCS as a logging endpoint. As part of configuration, you can elect to configure Google IAM role-based service account impersonation to avoid storing secrets. Read our guide on creating a Google IAM role for more information on this feature.

1. Review the information in our guide to setting up remote log streaming for Compute@Edge. Additionally, our developer documentation provides more information about logging with Compute@Edge code written in Rust, AssemblyScript, and JavaScript.
2. Click the Google Cloud Services Create endpoint button. The Create a Google Cloud Storage (GCS) endpoint page appears.
3. Fill out the Create a Google Cloud Storage (GCS) endpoint fields as follows:
   • In the Name field, enter the endpoint name you specified in your Compute@Edge code. For example, in our Rust code example, the name is my_endpoint_name.
   • In the Timestamp format field, optionally enter a timestamp format for log files. The default is an strftime compatible string. Our guide on changing where log files are written provides more information.
   • In the Bucket name field, enter the name of the GCS bucket in which to store the logs.
   • In the Access Method area, select how Fastly will access Google resources for purposes of log delivery. Valid values are User Credentials and IAM Role.
   • If you selected User Credentials, enter the following fields:
     • In the Email field, enter the client_email address listed in the JSON file associated with the service account you created on Google's website.
     • In the Secret key field, enter the private_key value listed in the JSON file associated with the service account you created on Google's website. We strip out the JSON newline escape characters for you so don't worry about removing them.
   • If you selected IAM Role, enter the following field:
     • In the Service Account Name field, enter the name of the service account email address you selected when configuring Google IAM service account impersonation.
   • In the PGP public key field, optionally enter a PGP public key that Fastly will use to encrypt your log files before writing them to disk. You will only be able to read the contents by decrypting them with your private key. The PGP key should be in PEM (Privacy-Enhanced Mail) format. Read our guide on log encryption for more information.
   • In the Period field, optionally enter an interval (in seconds) to control how frequently your log files are rotated. Rotation entails the finalization of one file object and the start of a new one, never removing any previously created file object. This value defaults to 3600 seconds.
4. Click the Advanced options link of the Create a Google Cloud Storage (GCS) endpoint page and decide which of the optional fields to change, if any.
5. Fill out the Advanced options of the Create a Google Cloud Storage (GCS) endpoint page as follows:
   • In the Path field, optionally enter the path within the bucket to store the files. The path ends with a trailing slash. If this field is left empty, the files will be saved in the bucket's root path. Our guide on changing where log files are written provides more information.
   • In the Select a log line format area, select the log line format for your log messages. Our guide on changing log line formats provides more information.
   • In the Compression field, optionally select the compression format you want applied to the log files. Our guide on changing log compression options provides more information.
6. Click the Create button to create the new logging endpoint.
7. Click the Activate button to deploy your configuration changes.

Recommended log format

Log messages can take on any format you choose as long as they can be processed from GCS.