Custom log formats

Fastly provides two versions of custom log formats for use when you set up remote log streaming. All new logging endpoints use the version 2 custom log format by default. You can upgrade version 1 logging endpoints to the version 2 custom log format. You can also make version 2 look like version 1 for the sake of continuity. We've described the key advantages of the version 2 custom log format below.

Version 2 log format

This table details version 2 of Fastly's custom log formats. All variables should be prefixed by a percent sign (%), as indicated in the table.

Version 1 log format

This table details version 1 of Fastly's custom log formats. All variables should be prefixed by a percent sign (%), as indicated in the table.

Upgrading endpoints to use version 2 log format

Follow these instructions to upgrade a logging endpoint to the version 2 custom log format:

1. Log in to the Fastly web interface.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.

4. Click Logging.

   

5. Click the name of a logging endpoint you want to edit.

   

6. Click Convert to Log Format Version 2.

   

7. Select an output format:

   • Use compatible output is the recommended setting. This setting won't modify your timestamp format string, but your logs will be formatted differently. The new format will be compatible with Apache's log format.
   • Maintain legacy output uses the version 2 parser, but the generated log string will be the same. This means that any instances of %t need to be turned into %{now}V, any instances of %r need to be turned into %{req.url}V, and any instances of %b need to be turned into %{resp.http.Content-Length}V.

8. Click Select.

9. Click Update to upgrade the logging endpoint to the version 2 custom log format.

10. Click Activate to deploy your configuration changes.

Using the API to upgrade

To upgrade a logging endpoint using the Fastly API, either clone the active version of the service you need upgraded or choose a version of the service that is unlocked and not active, then run the following command on that in development version:

$ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<logging endpoint>/<log name>' --data-binary '{"format_version":"2"}'

Keep in mind that the format_version field is a per-object field. Updating it on one logging object will not change it on any other objects.

For example, to update a Google Cloud Storage logging endpoint named "GCS Test," the curl command would look something like this:

$ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/1/logging/gcs/GCS%20Test' --data-binary '{"format_version":"2"}'

Determining which logging version is being used

To determine which logging version your service currently uses, issue the following curl command:

$ curl -X GET -H 'Fastly-Key: FASTLY_API_TOKEN'  'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<logging endpoint>/<log name>'

The curl command will produce JSON output detailing the configuration of your service version. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
    "address": "logs.papertrailapp.com",
    "created_at": "2016-04-01T15:37:30+00:00",
    "deleted_at": null,
    "format": "time.start.msec time.to_first_byte time.elapsed req.body_bytes_read req.bytes_read resp.http.content-length server.region client.ip %>s \"req.method req.url req.proto\" \"req.http.referer\" \"req.http.user-agent\"",
    "format_version": "2",
    "hostname": "logs.papertrailapp.com",
    "name": "fastly",
    "port": "11111",
    "public_key": null,
    "response_condition": "LOG /",
    "service_id": "1a2b3c4d5e6f7g8h9j0k",
    "updated_at": "2016-04-01T19:47:47+00:00",
    "version": "123"
}

The format_version field displays either a 1 or a 2 as appropriate for the custom log format being used.

Advantages of using the version 2 custom log format

The key advantages of using the version 2 custom log format include the following:

• Log lines are generated in vcl_log instead of vcl_deliver to allow us to accurately set the various size variables because vcl_log is run after the object has been delivered to the browser.
• The %t time directive is compatible with Apache log format. In version 1, we used a non-standard time format.
• The %r "first line of request" directive is compatible with Apache log format. In version 1, we incorrectly left off the protocol.
• When using the %b directive, which represents the size of a response in bytes, excluding HTTP headers is more accurate. In version 1, we used the reported Content-Length from the origin, which could be inaccurate (especially with ESI).
• We've added all Apache logging directives that make sense. In version 1, we used a smaller subset.

Making version 2 logs look like version 1

The default logging format for version 1 is as follows:

%h %l %u %t %r %>s

Most of the directives in version 2 are exactly the same - only %t and %r are different. After you upgrade to version 2 log formats, you can recreate the appearance of version 1 logs using the new %{...}V directive, which allows you to specifically include VCL in logging directives:

%h %l %u %{now}V %{req.method}V %{req.url}V %>s

In addition, if you are using the %b directive in version 1, then you can use this directive instead:

%{resp.http.Content-Length}V