Log streaming: Google Cloud Pub/Sub

Fastly's Real-Time Log Streaming feature can send log files to Cloud Pub/Sub, Google's global messaging and event data ingestion product.

Prerequisites

Before adding Cloud Pub/Sub as a logging endpoint for Fastly services, you will need to register for a Google Cloud Platform (GCP) account and then:

• Create a service account on Google's website.
• Navigate to the Pub/Sub section of the Google Cloud console. Follow the prompts to enable the API.
• Create a Pub/Sub topic.
• Obtain the private key from the JSON file associated with the service account configured for your Pub/Sub topic.

Adding Cloud Pub/Sub as a logging endpoint

Follow these instructions to add Cloud Pub/Sub as a logging endpoint:

1. Review the information in our Setting Up Remote Log Streaming guide.

2. Click the Google Cloud Pub/Sub Create endpoint button. The Create a Google Cloud Pub/Sub endpoint page appears.

   

3. Fill out the Create a Google Cloud Pub/Sub endpoint fields as follows:
   • In the Name field, enter a human-readable name for the endpoint.
   • In the Log format field, enter the data to send to Google Cloud Pub/Sub. See the example format section for details.
   • In the Project ID field, enter the ID of your Google Cloud Platform project.
   • In the Email field, enter the email address of the service account configured for your Pub/Sub topic.
   • In the Topic field, enter the Pub/Sub topic to which logs should be sent.
   • In the Secret Key field, enter the exact value of the private key associated with the service account configured for your Pub/Sub topic.

4. Click the Advanced options link of the Create a Cloud Pub/Sub endpoint page. The Advanced options appear.

   

5. In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about waf_debug_log.
6. Click the Create button to create the new logging endpoint.
7. Click the Activate button to deploy your configuration changes.

Example format

Data sent to Cloud Pub/Sub must be serialized as a JSON object, and every field in the JSON object must map to a string in your table's schema. The JSON can have nested data in it (e.g., the value of a key in your object can be another object). Here's an example format string for sending data:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
  "time_elapsed":%{time.elapsed.usec}V,
  "is_tls":%{if(req.is_ssl, "true", "false")}V,
  "client_ip":"%{req.http.Fastly-Client-IP}V",
  "geo_city":"%{client.geo.city}V",
  "geo_country_code":"%{client.geo.country_code}V",
  "request":"%{req.method}V",
  "host":"%{req.http.Fastly-Orig-Host}V",
  "url":"%{json.escape(req.url)}V",
  "request_referer":"%{json.escape(req.http.Referer)}V",
  "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
  "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
  "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
  "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V"
}