Log streaming: Google Cloud Pub/Sub
Last updated 2021-09-01
Fastly's Real-Time Log Streaming feature can send log files to Cloud Pub/Sub, Google's global messaging and event data ingestion product.
Fastly does not provide direct support for third-party services. See Fastly's Terms of Service for more information.
Prerequisites
Before adding Cloud Pub/Sub as a logging endpoint for Fastly services, you will need to register for a Google Cloud Platform (GCP) account and then:
- Create a service account on Google's website.
- Navigate to the Pub/Sub section of the Google Cloud console. Follow the prompts to enable the API.
- Create a Pub/Sub topic.
- Obtain the private key from the JSON file associated with the service account configured for your Pub/Sub topic.
Read more about Cloud Pub/Sub in Google’s documentation.
Adding Cloud Pub/Sub as a logging endpoint
Follow these instructions to add Cloud Pub/Sub as a logging endpoint:
- Review the information in our Setting Up Remote Log Streaming guide.
- Click the Google Cloud Pub/Sub Create endpoint button. The Create a Google Cloud Pub/Sub endpoint page appears.
- Fill out the Create a Google Cloud Pub/Sub endpoint fields as follows:
- In the Name field, enter a human-readable name for the endpoint.
- In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. See our guide on changing log placement for more information.
- In the Log format field, enter the data to send to Google Cloud Pub/Sub. See the example format section for details.
- In the Project ID field, enter the ID of your Google Cloud Platform project.
- In the Email field, enter the email address of the service account configured for your Pub/Sub topic.
- In the Topic field, enter the Pub/Sub topic to which logs should be sent.
- In the Secret Key field, enter the exact value of the private key associated with the service account configured for your Pub/Sub topic.
- Click the Create button to create the new logging endpoint.
- Click the Activate button to deploy your configuration changes.
Example format
Data sent to Cloud Pub/Sub must be serialized as a JSON object, and every field in the JSON object must map to a string in your table's schema. The JSON can have nested data in it (e.g., the value of a key in your object can be another object). Here's an example format string for sending data:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
"client_ip": "%{req.http.Fastly-Client-IP}V",
"geo_country": "%{client.geo.country_name}V",
"geo_city": "%{client.geo.city}V",
"host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
"url": "%{json.escape(req.url)}V",
"request_method": "%{json.escape(req.method)}V",
"request_protocol": "%{json.escape(req.proto)}V",
"request_referer": "%{json.escape(req.http.referer)}V",
"request_user_agent": "%{json.escape(req.http.User-Agent)}V",
"response_state": "%{json.escape(fastly_info.state)}V",
"response_status": %{resp.status}V,
"response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
"response_body_size": %{resp.body_bytes_written}V,
"fastly_server": "%{json.escape(server.identity)}V",
"fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
}