Log streaming: Google Cloud Pub/Sub

Fastly's Real-Time Log Streaming feature can send log files to Cloud Pub/Sub, Google's global messaging and event data ingestion product.

Prerequisites

Before adding Cloud Pub/Sub as a logging endpoint for Fastly services, you will need to register for a Google Cloud Platform (GCP) account and then:

• Create a service account on Google's website.
• Navigate to the Pub/Sub section of the Google Cloud console. Follow the prompts to enable the API.
• Create a Pub/Sub topic.
• Use Google IAM service account impersonation or key-based access to configure the access method that Fastly uses to send logs.

Creating a service account

Google Cloud Platform (GCP) uses service accounts for third-party application authentication and role-based access to Google Cloud resources. To create a new service account, follow the instructions in the Google Cloud documentation.

Configure access for the service account

Fastly requires access to write logs to your Pub/Sub topic. You can use either role-based, service account impersonation, or key-based access to allow Fastly to send logs to Google Pub/Sub.

Configuring Google IAM service account impersonation

If you elect to use Google IAM service account impersonation to avoid storing keys with Fastly, you'll need to specify a service account name and a project ID when adding GCS as your logging endpoint. The service account ID comes before the @ in the service account email and the project ID immediately after. For example, if your service account email is my-name@projectid.iam.gserviceaccount.com, the service account ID is my-name and the project ID is projectid. Our guide to creating an Google IAM role provides specific details on configuring this feature.

Configuring key-based access

If you elect to use key-based access, refer to Google's guide on generating a service account credential.

When configuring key-based access, you must select the JSON format. Upon saving the configuration, a JSON file will be downloaded to your computer, containing the credentials for the GCS service account you just created. Open the file with a text editor and make a note of the private_key and client_email.

Adding Cloud Pub/Sub as a logging endpoint

Follow these instructions to add Cloud Pub/Sub as a logging endpoint. As part of configuration, you can elect to configure Google IAM role-based service account impersonation to avoid storing secrets. Read our guide on creating a Google IAM role for more information on this feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

{
  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "geo_country": "%{client.geo.country_name}V",
  "geo_city": "%{client.geo.city}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "url": "%{json.escape(req.url)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_protocol": "%{json.escape(req.proto)}V",
  "request_referer": "%{json.escape(req.http.referer)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_state": "%{json.escape(fastly_info.state)}V",
  "response_status": %{resp.status}V,
  "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
  "response_body_size": %{resp.body_bytes_written}V,
  "fastly_server": "%{json.escape(server.identity)}V",
  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
}