Log streaming: Google Cloud Pub/Sub

      Last updated September 30, 2020

    Fastly's Real-Time Log Streaming feature can send log files to Cloud Pub/Sub, Google's global messaging and event data ingestion product.

    Prerequisites

    Before adding Cloud Pub/Sub as a logging endpoint for Fastly services, you will need to register for a Google Cloud Platform (GCP) account and then:

    Adding Cloud Pub/Sub as a logging endpoint

    Follow these instructions to add Cloud Pub/Sub as a logging endpoint:

    1. Review the information in our Setting Up Remote Log Streaming guide.
    2. Click the Google Cloud Pub/Sub Create endpoint button. The Create a Google Cloud Pub/Sub endpoint page appears.

      The Google Cloud Pub/Sub endpoint page

    3. Fill out the Create a Google Cloud Pub/Sub endpoint fields as follows:
      • In the Name field, enter a human-readable name for the endpoint.
      • In the Log format field, enter the data to send to Google Cloud Pub/Sub. See the example format section for details.
      • In the Project ID field, enter the ID of your Google Cloud Platform project.
      • In the Email field, enter the email address of the service account configured for your Pub/Sub topic.
      • In the Topic field, enter the Pub/Sub topic to which logs should be sent.
      • In the Secret Key field, enter the value of the private key associated with the service account configured for your Pub/Sub topic.
    4. Click the Advanced options link of the Create a Cloud Pub/Sub endpoint page. The Advanced options appear.

      the advanced options on the create a new Google Cloud Pub/Sub endpoint page

    5. In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about waf_debug_log.
    6. Click the Create button to create the new logging endpoint.
    7. Click the Activate button to deploy your configuration changes.

    Example format

    Data sent to Cloud Pub/Sub must be serialized as a JSON object, and every field in the JSON object must map to a string in your table's schema. The JSON can have nested data in it (e.g., the value of a key in your object can be another object). Here's an example format string for sending data:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    
    {
      "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
      "time_elapsed":%{time.elapsed.usec}V,
      "is_tls":%{if(req.is_ssl, "true", "false")}V,
      "client_ip":"%{req.http.Fastly-Client-IP}V",
      "geo_city":"%{client.geo.city}V",
      "geo_country_code":"%{client.geo.country_code}V",
      "request":"%{req.method}V",
      "host":"%{req.http.Fastly-Orig-Host}V",
      "url":"%{json.escape(req.url)}V",
      "request_referer":"%{json.escape(req.http.Referer)}V",
      "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
      "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
      "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
      "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V"
    }
    
    Back to Top