Custom log formats

Fastly provides two versions of custom log formats for use when you set up remote log streaming. All new logging endpoints use the version 2 custom log format by default. You can upgrade version 1 logging endpoints to the version 2 custom log format. You can also make version 2 look like version 1 for the sake of continuity. We've described the key advantages of the version 2 custom log format below.

Version 2 log format

This table details version 2 of Fastly's custom log formats. All variables should be prefixed by a percent sign (%), as indicated in the table.

Format StringDescription
%%The percent sign.
%aThe client IP address of the request. Equivalent to req.http.Fastly-Client-IP.
%AThe local IP address. Equivalent to server-ip.
%BThe size of response in bytes, excluding HTTP headers. Equivalent to resp.body_bytes_written.
%bThe size of response in bytes, excluding HTTP headers. In Common Log Format (CLF), that means a "-" rather than a 0 when no bytes are sent.
%{Foobar}CThe contents of cookie Foobar in the request sent to the server.
%DThe time taken to serve the request, in microseconds. Equivalent to time.elapsed.usec.
%{FOOBAR}eNot supported. Always returns "-".
%fThe filename. Equivalent to req.url with any query string removed, because Varnish has no notion of a file being served.
%hThe remote IP address. Equivalent to req.http.Fastly-Client-IP.
%HThe request protocol. Equivalent to req.proto.
%{Foobar}iThe contents of Foobar: header lines in the request sent to the server.
%IBytes received, including request and headers. Cannot be zero Equivalent to req.bytes_read.
%kThe number of keepalive requests handled on this connection. Always returns 0.
%lNot supported. Always returns "-".
%mThe request method. Equivalent to req.request.
%{Foobar}nNot supported. Always returns "-".
%{Foobar}oThe contents of Foobar: header lines in the reply.
%OBytes sent, including headers. Cannot be zero. Equivalent to resp.bytes_written.
%pThe canonical port of the server serving the request. Always returns 80. Equivalent to server.port.
%{format}pThe canonical port of the server serving the request. Valid formats are canonical, local, or remote. Returns 80 for HTTP requests and 443 for HTTPS requests.
%PNot supported. Always returns "-".
%{format}PNot supported. Always returns "-".
%qThe query string (prepended with a ? if a query string exists, otherwise an empty string). Equivalent to req.url.
%rThe first line of the request.
%RNot supported. Always returns "-".
%sThe status. For requests that got internally redirected, this is the status of the original request. Use %>s for the final status. Equivalent to resp.status.
%tThe time the request was received, in Standard English format (e.g., 01/Jan/1970:00:00:00 -0700). The last number indicates the timezone offset from GMT.
%{format}tThe time, in the form given by format, which should be in strftime(3) format (potentially localized). If the format starts with begin: (the default) the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by strftime(3), the following format tokens are supported: sec (number of seconds since the Epoch), msec (number of milliseconds since the Epoch), usec (number of microseconds since the Epoch), msec_frac (millisecond fraction), and usec_frac (microsecond fraction). Equivalent to time.start.
%TThe time taken to serve the request, in seconds. Equivalent to time.elapsed.sec.
%uNot supported. Always returns "-".
%UThe URL path requested, not including any query string. Equivalent to req.url.path.
%vThe domain name of the request. Equivalent to req.http.host.
%VThe same as %v. Equivalent to req.http.host.
%{vcl}VThe literal VCL to include without quoting. This can be used to write VCL variables to your logs (e.g., %{client.geo.country_code}V or %{tls.client.cipher}V). This %-directive is a Fastly extension and is not found in Apache.
%XThe connection status when response is completed. Always set as + (connection may be kept alive after the response is sent).

Version 1 log format

This table details version 1 of Fastly's custom log formats. All variables should be prefixed by a percent sign (%), as indicated in the table.

Format StringDescription
%bThe content size of the response, calculated using the Content-Length header rather than actually checking the length of the response (and may therefore be wrong).
%hThe remote IP address.
%lThe remote log name. Always returns the hardcoded value "-".
%rThe HTTP verb and request path (e.g., GET /index.html). Unlike Apache and version 2 log formats, the protocol version is not included.
%>sThe status of the last request.
%tThe time the request was received, in Unix ctime format (e.g., Thu, 01 Jan 1970 00:00:00 GMT) rather than Apache's Standard English format (e.g., 01/Jan/1970:00:00:00 -0700).
%uAlways returns "-".

Upgrading endpoints to use version 2 log format

WARNING

Upgrading is a permanent change. Logging objects using version 2 formatting cannot be downgraded to version 1.

Follow these instructions to upgrade a logging endpoint to the version 2 custom log format:

  1. Log in to the Fastly web interface.
  2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
  3. Click the Edit configuration button and then select the option to clone the active version. The Domains page appears.
  4. Click the Logging link. The Logging endpoints page appears. If you have any logging endpoints using the version 1 custom log format, a message appears indicating that they can be updated.

    the logging endpoints page with upgrade message

  5. Click the name of a logging endpoint you want to edit. The Edit this endpoint page appears.

    the edit logging endpoint page with upgrade message

  6. Click the Convert to Log Format Version 2 button. The Convert to log format version 2 window appears.

    the convert to log format version 2 window

  7. Select an output format:

    • Use compatible output is the recommended setting. This setting won't modify your timestamp format string, but your logs will be formatted differently. The new format will be compatible with Apache's log format.
    • Maintain legacy output uses the version 2 parser, but the generated log string will be the same. This means that any instances of %t need to be turned into %{now}V, any instances of %r need to be turned into %{req.url}V, and any instances of %b need to be turned into %{resp.http.Content-Length}V.
  8. Click the Select button. The Edit this endpoint page appears.

  9. Click the Update button to upgrade the logging endpoint to the version 2 custom log format.

  10. Click the Activate button to deploy your configuration changes.

Using the API to upgrade

To upgrade a logging endpoint using the Fastly API, either clone the active version of the service you need upgraded or choose a version of the service that is unlocked and not active, then run the following command on that in development version:

$ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<logging endpoint>/<log name>' --data-binary '{"format_version":"2"}'

Keep in mind that the format_version field is a per-object field. Updating it on one logging object will not change it on any other objects.

For example, to update a Google Cloud Storage logging endpoint named "GCS Test," the curl command would look something like this:

$ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/1/logging/gcs/GCS%20Test' --data-binary '{"format_version":"2"}'
NOTE

The log name included a space that needed to be URL encoded (the space into %20).

Determining which logging version is being used

To determine which logging version your service currently uses, issue the following curl command:

$ curl -X GET -H 'Fastly-Key: FASTLY_API_TOKEN' 'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<logging endpoint>/<log name>'

The curl command will produce JSON output detailing the configuration of your service version. For example:

1{
2 "address": "logs.papertrailapp.com",
3 "created_at": "2016-04-01T15:37:30+00:00",
4 "deleted_at": null,
5 "format": "time.start.msec time.to_first_byte time.elapsed req.body_bytes_read req.bytes_read resp.http.content-length server.region client.ip %>s \"req.method req.url req.proto\" \"req.http.referer\" \"req.http.user-agent\"",
6 "format_version": "2",
7 "hostname": "logs.papertrailapp.com",
8 "name": "fastly",
9 "port": "11111",
10 "public_key": null,
11 "response_condition": "LOG /",
12 "service_id": "1a2b3c4d5e6f7g8h9j0k",
13 "updated_at": "2016-04-01T19:47:47+00:00",
14 "version": "123"
15}

The format_version field displays either a 1 or a 2 as appropriate for the custom log format being used.

Advantages of using the version 2 custom log format

The key advantages of using the version 2 custom log format include the following:

  • Log lines are generated in vcl_log instead of vcl_deliver to allow us to accurately set the various size variables because vcl_log is run after the object has been delivered to the browser.
  • The %t time directive is compatible with Apache log format. In version 1, we used a non-standard time format.
  • The %r "first line of request" directive is compatible with Apache log format. In version 1, we incorrectly left off the protocol.
  • When using the %b directive, which represents the size of a response in bytes, excluding HTTP headers is more accurate. In version 1, we used the reported Content-Length from the origin, which could be inaccurate (especially with ESI.
  • We've added all Apache logging directives that make sense. In version 1, we used a smaller subset.

Making version 2 logs look like version 1

The default logging format for version 1 is as follows:

%h %l %u %t %r %>s

Most of the directives in version 2 are exactly the same - only %t and %r are different. After you upgrade to version 2 log formats, you can recreate the appearance of version 1 logs using the new %{...}V directive, which allows you to specifically include VCL in logging directives:

%h %l %u %{now}V %{req.method}V %{req.url}V %>s

In addition, if you are using the %b directive in version 1, then you can use this directive instead:

%{resp.http.Content-Length}V
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support.