Log streaming: Amazon Kinesis Data Streams

Fastly's Real-Time Log Streaming feature can send log files to Amazon Kinesis Data Streams. Amazon Kinesis Data Streams (KDS) is a real-time data streaming service that can continuously capture data from a variety of sources.

NOTE

Fastly does not provide direct support for third-party services. Read Fastly's Terms of Service for more information.

How Amazon Kinesis Data Streams work with Fastly log streaming

Amazon KDS sends data records to a stream. Each stream comprises one or more shards. A shard represents a fixed amount of processing capacity and the total processing capacity of a stream is determined by the number of shards. The number of shards may be increased or decreased over the lifetime of a stream. This is important because the Fastly Kinesis logging endpoint monitors the number of shards and attempts to uniformly distribute the log data records across the available shards. When the number of shards for a stream changes, the Fastly Kinesis logging endpoint automatically adjusts in response. The goal is to make the best use of the throughput capability of the stream while minimizing the configuration overhead required for our customers.

If the log volume exceeds the throughput capacity of the stream, Amazon KDS will return errors to our system that indicate that the stream is being throttled and that may prevent some logs from being delivered. AWS CloudWatch provides a metric for Kinesis Data Streams, WriteProvisionedThroughputExceeded, that can be used to monitor this so that adjustments to the stream capacity can be made as necessary.

TIP

For more information about working with Amazon KDS and understanding the capacity limits, refer to the Kinesis Developer Guide.

Prerequisites

Before adding Amazon KDS as a logging endpoint for Fastly services, we recommend creating Identity and Access Management (IAM) credentials in your AWS account specifically for Fastly. Our recommended way for doing this is by creating an AWS IAM role, which lets you grant temporary credentials. For more information, see Creating an AWS IAM Role for Fastly Logging. Alternatively, create an IAM user and grant the user kinesis:PutRecords and kinesis:ListShards permissions for the logging stream. For more information, see Amazon's guidance on understanding and getting your AWS credentials.

Adding Amazon Kinesis as a logging endpoint

After you've registered for an AWS account and created an IAM user in Amazon Kinesis, follow these instructions to add Amazon KDS as a logging endpoint:

  1. Deliver services
  2. Compute services
  1. Review the information in our guide to setting up remote log streaming.
  2. In the Amazon Kinesis Data Streams area, click Create endpoint.

  3. Fill out the Create an Amazon Kinesis Data Streams endpoint fields as follows:

    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. Read our guide on changing log placement for more information.
    • In the Log format field, optionally enter an Apache-style string or VCL variables to use for log formatting. Consult the example format section for details.
    • In the Access method field, select either User Credentials or IAM Role.
    • If you select User Credentials, enter the access key and secret key associated with the IAM user you created in your AWS account specifically for Fastly. Check out Amazon's documentation on security credentials for more information.
    NOTE

    Password management software may mistakenly treat the Secret Key field as a password field because of the way your web browser works. As such, that software may try to auto-fill this field with your Fastly account password. If this happens to you, the AWS integration with Fastly services won't work and you will need to enter Secret Key manually instead.

    • If you select IAM Role, enter the Amazon Resource Name (ARN) for the IAM role granting Fastly access to KDS. For more information, check out Creating an AWS IAM Role for Fastly Logging.
    • In the Stream name field, enter the name of the Kinesis stream to which log data will be sent.
    • From the Region menu, select the region to stream logs to. This must match the region where you created your Kinesis stream.
  4. Click Create to create the new logging endpoint.
  5. Click Activate to deploy your configuration changes.

Example format

The following is an example format string for sending data to Amazon KDS. Our discussion of format strings provides more information.

1{
2 "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
3 "client_ip": "%{req.http.Fastly-Client-IP}V",
4 "geo_country": "%{client.geo.country_name}V",
5 "geo_city": "%{client.geo.city}V",
6 "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
7 "url": "%{json.escape(req.url)}V",
8 "request_method": "%{json.escape(req.method)}V",
9 "request_protocol": "%{json.escape(req.proto)}V",
10 "request_referer": "%{json.escape(req.http.referer)}V",
11 "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
12 "response_state": "%{json.escape(fastly_info.state)}V",
13 "response_status": %{resp.status}V,
14 "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
15 "response_body_size": %{resp.body_bytes_written}V,
16 "fastly_server": "%{json.escape(server.identity)}V",
17 "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
18}
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.