Log streaming: Google BigQuery

Fastly's Real-Time Log Streaming feature can send log files to BigQuery, Google's managed enterprise data warehouse.

Prerequisites

Before adding BigQuery as a logging endpoint for Fastly services you will need to:

• Register for a Google Cloud Platform (GCP) account.
• Create a service account on Google's website.
• Use Google IAM service account impersonation or key-based access to configure the access method that Fastly uses to send logs.
• Enable the BigQuery API.
• Create a BigQuery dataset.
• Add a BigQuery table.

Creating a service account

Google Cloud Platform (GCP) uses service accounts for third-party application authentication and role-based access to Google Cloud resources. To create a new service account, follow the instructions in the Google Cloud documentation.

Configure access for the service account

Fastly requires access to write logs to your BigQuery table. You can use either Google IAM service account impersonation or key-based access to allow Fastly to send logs to BigQuery.

Configuring Google IAM service account impersonation

If you elect to use Google IAM service account impersonation to avoid storing keys with Fastly, you'll need to specify a service account name and a project ID when adding GCS as your logging endpoint. The service account ID comes before the @ in the service account email and the project ID immediately after. For example, if your service account email is my-name@projectid.iam.gserviceaccount.com, the service account ID is my-name and the project ID is projectid. Our guide to creating an Google IAM role provides specific details on configuring this feature.

Configuring key-based access

If you elect to use key-based access, refer to Google's guide on generating a service account credential.

When configuring key-based access, you must select the JSON format. Upon saving the configuration, a JSON file will be downloaded to your computer, containing the credentials for the GCS service account you just created. Open the file with a text editor and make a note of the private_key and client_email.

Enabling the BigQuery API

To send your Fastly logs to your BigQuery table, you'll need to enable the BigQuery API in the Google Cloud Platform API Manager.

Creating the BigQuery dataset

After you've enabled the BigQuery API, follow these instructions to create a BigQuery dataset:

1. Open the BigQuery page in the Cloud Console.
2. In the Explorer panel, select the project where you want to create the dataset.
3. In the details panel, click Create dataset.
4. In the Dataset ID field, enter a name for the dataset (e.g., fastly_bigquery).
5. Click Create dataset.

Adding a BigQuery table

After you've created the BigQuery dataset, you'll need to add a BigQuery table. There are four ways of creating the schema for the table:

• Edit the schema using the BigQuery web interface.
• Edit the schema using the text field in the BigQuery web interface.
• Use an existing table.
• Set the table to automatically detect the schema.

Follow these instructions to add a BigQuery table:

1. Open the BigQuery page in the Cloud Console.

2. In the Explorer panel, expand your project and select the BigQuery dataset you created previously.

3. In the Source section, select Empty Table from the Create table from: menu.

   

4. In the Table name field, enter a name for the table (e.g., logs).

5. In the Schema section of the BigQuery website, use the interface to add fields and complete the schema. Check out the example schema section for details.

6. Click Create Table.

Adding BigQuery as a logging endpoint

Follow these instructions to add BigQuery as a logging endpoint. As part of configuration, you can elect to configure Google IAM role-based service account impersonation to avoid storing secrets. Read our guide on creating a Google IAM role for more information on this feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

{
  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S"\}, time.start)}V",
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "geo_country": "%{client.geo.country_name}V",
  "geo_city": "%{client.geo.city}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "url": "%{json.escape(req.url)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_protocol": "%{json.escape(req.proto)}V",
  "request_referer": "%{json.escape(req.http.referer)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_state": "%{json.escape(fastly_info.state)}V",
  "response_status": %{resp.status}V,
  "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
  "response_body_size": %{resp.body_bytes_written}V,
  "fastly_server": "%{json.escape(server.identity)}V",
  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
}

timestamp:TIMESTAMP,client_ip:STRING,geo_country:STRING,geo_city:STRING,host:STRING,url:STRING,request_method:STRING,request_protocol:STRING,request_referer:STRING,request_user_agent:STRING,response_state:STRING,response_status:STRING,response_reason:STRING,response_body_size:STRING,fastly_server:STRING,fastly_is_edge:BOOLEAN