Log streaming: Google Cloud Pub/Sub

Fastly's Real-Time Log Streaming feature can send log files to Cloud Pub/Sub, Google's global messaging and event data ingestion product.

NOTE

Fastly does not provide direct support for third-party services. Read Fastly's Terms of Service for more information.

Prerequisites

Before adding Cloud Pub/Sub as a logging endpoint for Fastly services, you will need to register for a Google Cloud Platform (GCP) account and then:

  • Create a service account on Google's website.
  • Navigate to the Pub/Sub section of the Google Cloud console. Follow the prompts to enable the API.
  • Create a Pub/Sub topic.
  • Use Google IAM service account impersonation or key-based access to configure the access method that Fastly uses to send logs.

Creating a service account

Google Cloud Platform (GCP) uses service accounts for third-party application authentication and role-based access to Google Cloud resources. To create a new service account, follow the instructions in the Google Cloud documentation.

NOTE

When creating the service account for logging to Google Pub/Sub, the service account must be assigned the Pub/Sub Publisher role to publish to the topic you created for Fastly logging.

Create service account panel

Configure access for the service account

Fastly requires access to write logs to your Pub/Sub topic. You can use either role-based, service account impersonation, or key-based access to allow Fastly to send logs to Google Pub/Sub.

Configuring Google IAM service account impersonation

If you elect to use Google IAM service account impersonation to avoid storing keys with Fastly, you may use the same service account created above. Our guide to creating an Google IAM role provides further details on configuring this feature.

Configuring key-based access

If you elect to use key-based access, refer to Google's guide on generating a service account credential.

When configuring key-based access, you must select the JSON format. Upon saving the configuration, a JSON file will be downloaded to your computer, containing the credentials for the GCS service account you just created. Open the file with a text editor and make a note of the private_key and client_email.

NOTE

Read more about Cloud Pub/Sub in Google’s documentation.

Adding Cloud Pub/Sub as a logging endpoint

Follow these instructions to add Cloud Pub/Sub as a logging endpoint. As part of configuration, you can elect to configure Google IAM role-based service account impersonation to avoid storing secrets. Read our guide on creating a Google IAM role for more information on this feature.

  1. Deliver services
  2. Compute services
  1. Review the information in our guide to setting up remote log streaming.
  2. In the Google Cloud Pub/Sub area, click Create endpoint.
  3. Fill out the Create a Google Cloud Pub/Sub endpoint fields as follows:
    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. Read our guide on changing log placement for more information.
    • In the Log format field, enter the data to send to Google Cloud Pub/Sub. See the example format section for details.
    • In the Project ID field, enter the ID of your Google Cloud Platform project.
    • In the Email field, enter the email address of the service account configured for your Pub/Sub topic.
    • In the Topic field, enter the Pub/Sub topic to which logs should be sent.
    • In the Access Method area, select how Fastly will access Google resources for purposes of log delivery. Valid values are User Credentials and IAM Role.
    • If you selected User Credentials, fill out the following fields:
      • In the Email field, enter the email address of the service account configured for your Pub/Sub topic.
      • In the Secret Key field, enter the exact value of the private_key associated with the service account configured for your Pub/Sub topic.
    • If you selected IAM Role, enter fill out following field:
      • In the Service Account Name field, enter the name of the service account email address you selected when configuring Google IAM service account impersonation.
  4. Click Create to create the new logging endpoint.
  5. Click Activate to deploy your configuration changes.

Example format

Data sent to Cloud Pub/Sub must be serialized as a JSON object, and every field in the JSON object must map to a string in your table's schema. The JSON can have nested data in it (e.g., the value of a key in your object can be another object). Here's an example format string for sending data:

1{
2 "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
3 "client_ip": "%{req.http.Fastly-Client-IP}V",
4 "geo_country": "%{client.geo.country_name}V",
5 "geo_city": "%{client.geo.city}V",
6 "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
7 "url": "%{json.escape(req.url)}V",
8 "request_method": "%{json.escape(req.method)}V",
9 "request_protocol": "%{json.escape(req.proto)}V",
10 "request_referer": "%{json.escape(req.http.referer)}V",
11 "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
12 "response_state": "%{json.escape(fastly_info.state)}V",
13 "response_status": %{resp.status}V,
14 "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
15 "response_body_size": %{resp.body_bytes_written}V,
16 "fastly_server": "%{json.escape(server.identity)}V",
17 "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
18}
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.