LOG IN SIGN UP
Documentation

Custom log formats

  Last updated February 17, 2017

Fastly provides two versions of custom log formats. All new logging endpoints use the version 2 custom log format by default. You can upgrade version 1 logging endpoints to the version 2 custom log format. You can also make version 2 look like version 1 for the sake of continuity. We've described the key advantages of the version 2 custom log format below.

Version 2 log format

This table details version 2 of Fastly's custom log formats.

String Description
% The percent sign.
a The client IP address of the request.
A The local IP address.
B The size of response in bytes, excluding HTTP headers.
b The size of response in bytes, excluding HTTP headers. In Common Log Format (CLF), that means a "-" rather than a 0 when no bytes are sent.
{Foobar}C The contents of cookie Foobar in the request sent to the server.
D The time taken to serve the request, in microseconds.
{FOOBAR}e The contents of the environment variable FOOBAR. Always returns NIL.
f The filename.
h The remote hostname.
H The request protocol.
{Foobar}i The contents of Foobar: header lines in the request sent to the server.
I Bytes received, including request and headers. Cannot be zero.
k The number of keepalive requests handled on this connection. Always returns 0.
l The remote logname (from identd, if supplied). This will return a dash unless mod_ident is present and IdentityCheck is set On. Always returns "-".
m The request method.
{Foobar}n The contents of note Foobar from another module. Always returns NIL.
{Foobar}o The contents of Foobar: header lines in the reply.
O Bytes sent, including headers. Cannot be zero.
p The canonical port of the server serving the request. Always returns 80.
{format}p The canonical port of the server serving the request. Valid formats are canonical, local, or remote. Returns 80 for HTTP requests and 443 for HTTPS requests.
P The process ID of the child that serviced the request. Always returns NIL.
{format}P The process ID or thread ID of the child that serviced the request. Valid formats are pid, tid, and hextid. Always returns NIL.
q The query string (prepended with a ? if a query string exists, otherwise an empty string).
r The first line of the request.
R The handler generating the response (if any).
s The status. For requests that got internally redirected, this is the status of the original request. Use %>s for the final status.
t The time the request was received, in Standard English format (e.g., 01/Jan/1970:00:00:00 -0700). The last number indicates the timezone offset from GMT.
{format}t The time, in the form given by format, which should be in strftime(3) format (potentially localized). If the format starts with begin: (the default) the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by strftime(3), the following format tokens are supported: sec (number of seconds since the Epoch), msec (number of milliseconds since the Epoch), usec (number of microseconds since the Epoch), msec_frac (millisecond fraction), and usec_frac (microsecond fraction).
T The time taken to serve the request, in seconds.
u The remote user if the request was authenticated. May be bogus if return status (%s) is 401 (unauthorized). Always returns "-".
U The URL path requested, not including any query string.
v The canonical ServerName of the server serving the request.
V The server name according to the UseCanonicalName setting.
{vcl}V The literal VCL to include without quoting (a Fastly extension).
X The connection status when response is completed. Statuses include X (connection aborted before the response completed), + (connection may be kept alive after the response is sent), and - (connection will be closed after the response is sent).

Version 1 log format

This table details version 1 of Fastly's custom log formats.

String Description
b The content size of the response, calculated using the Content-Length header rather than actually checking the length of the response (and may therefore be wrong).
h The remote IP address.
l The remote log name. Always returns the hardcoded value "-".
r The HTTP verb and request path (e.g., GET /index.html). Unlike Apache and version 2 log formats, the protocol version is not included.
>s The status of the last request.
t The time the request was received, in Unix ctime format (e.g., Thu, 01 Jan 1970 00:00:00 GMT) rather than Apache's Standard English format (e.g., 01/Jan/1970:00:00:00 -0700).
u The remote user. Always returns the hardcoded value "-".

Upgrading endpoints to use version 2 log format

Follow these instructions to upgrade a logging endpoint to the version 2 custom log format:

  1. Log in to the Fastly web interface and click the Configure link.
  2. From the service menu, select the appropriate service.
  3. Click the Edit configuration button and then select Clone active. The service version page appears.
  4. Click the Logging tab. The Logging endpoints page appears. If you have any logging endpoints using the version 1 custom log format, a message appears indicating that they can be updated.

    the logging endpoints page with upgrade message

  5. Click the name of a logging endpoint to edit it. The Edit this endpoint page appears.

    the edit logging endpoint page with upgrade message

  6. Click the Convert to Log Format Version 2 button. The Convert to log format version 2 window appears.

    the convert to log format version 2 window

  7. Select an output format:
    • Use compatible output is the recommended setting. This setting won't modify your timestamp format string, but your logs will be formatted differently. The new format will be compatible with Apache's log format.
    • Maintain legacy output uses the version 2 parser, but the generated log string will be the same. This means that any instances of %t need to be turned into %{now}V, any instances of %r need to be turned into %{req.url}V, and any instances of %b need to be turned into %{resp.http.Content-Length}V.
  8. Click the Select button. The Edit this endpoint page appears.
  9. Click the Update button to upgrade the logging endpoint to the version 2 custom log format.
  10. Click the Activate button to deploy your configuration changes.

Using the API to upgrade

To upgrade a logging endpoint using the Fastly API, either clone the active version of the service you need upgraded or choose a version of the service that is unlocked and not active, then run the following command on that in development version:

curl -X PUT -H 'Fastly-Key: <your Fastly API key>' -H 'Content-Type: application/json' 'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<log type>/<log name>' --data-binary '{"format_version":"2"}'

where log type is type of the endpoint you want to upgrade:

Keep in mind that the format_version field is a per-object field. Updating it on one logging object will not change it on any other objects. For example, to upgrade a Google Cloud Storage endpoint the cURL command would look something like this if the endpoint was named "GCS Test":

curl -X PUT -H 'Fastly-Key: d3cafb4dde4dbeef' -H 'Content-Type: application/json' 'https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/1/logging/gcs/GCS%20Test' --data-binary '{"format_version":"2"}'

Determining which logging version is being used

To determine which logging version your service currently uses, issue the following cURL command:

curl -X GET -H 'Fastly-Key: <your Fastly API key>'  'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<log type>/<log name>'

where <log type> is ftp, heroku, logentries, loggly, logshuttle, papertrail, s3, scalyr, sftp, sumologic, or syslog. The cURL command will produce JSON output detailing the configuration of your service version. For example:

{
    "address": "logs.papertrailapp.com",
    "created_at": "2016-04-01T15:37:30+00:00",
    "deleted_at": null,
    "format": "time.start.msec time.to_first_byte time.elapsed req.body_bytes_read req.bytes_read resp.http.content-length server.region client.ip %>s \"req.request req.url req.proto\" \"req.http.referer\" \"req.http.user-agent\"",
    "format_version": "2",
    "hostname": "logs.papertrailapp.com",
    "name": "fastly",
    "port": "11111",
    "public_key": null,
    "response_condition": "LOG /",
    "service_id": "1a2b3c4d5e6f7g8h9j0k",
    "updated_at": "2016-04-01T19:47:47+00:00",
    "version": "123"
}

The format_version field displays either a 1 or a 2 as appropriate for the custom log format being used.

Advantages of using the version 2 custom log format

The key advantages of using the version 2 custom log format include the following:

Making version 2 logs look like version 1

The default logging format for version 1 is as follows:

%h %l %u %t %r %>s

Most of the directives in version 2 are exactly the same - only %t and %r are different. After you upgrade to version 2 log formats, you can recreate the appearance of version 1 logs using the new %{...}V directive, which allows you to specifically include VCL in logging directives:

%h %l %u %{now}V %{req.request}V %{req.url}V %>s

In addition, if you are using the %b directive in version 1, then you can use this directive instead:

%{resp.http.Content-Length}V


Additional resources:


Back to Top