- English
- 日本語
Custom log formats
Last updated 2023-04-14
Fastly provides two versions of custom log formats for use when you set up remote log streaming. All new logging endpoints use the version 2 custom log format by default. You can upgrade version 1 logging endpoints to the version 2 custom log format. You can also make version 2 look like version 1 for the sake of continuity. We've described the key advantages of the version 2 custom log format below.
Version 2 log format
This table details version 2 of Fastly's custom log formats. All variables should be prefixed by a percent sign (%
), as indicated in the table.
Format String | Description |
---|---|
%% | The percent sign. |
%a | The client IP address of the request. Equivalent to req.http.Fastly-Client-IP . |
%A | The local IP address. Equivalent to server-ip . |
%B | The size of response in bytes, excluding HTTP headers. Equivalent to resp.body_bytes_written . |
%b | The size of response in bytes, excluding HTTP headers. In Common Log Format (CLF), that means a "-" rather than a 0 when no bytes are sent. |
%{Foobar}C | The contents of cookie Foobar in the request sent to the server. |
%D | The time taken to serve the request, in microseconds. Equivalent to time.elapsed.usec . |
%{FOOBAR}e | Not supported. Always returns "-" . |
%f | The filename. Equivalent to req.url with any query string removed, because Varnish has no notion of a file being served. |
%h | The remote IP address. Equivalent to req.http.Fastly-Client-IP . |
%H | The request protocol. Equivalent to req.proto . |
%{Foobar}i | The contents of Foobar: header lines in the request sent to the server. |
%I | Bytes received, including request and headers. Cannot be zero Equivalent to req.bytes_read . |
%k | The number of keepalive requests handled on this connection. Always returns 0 . |
%l | Not supported. Always returns "-" . |
%m | The request method. Equivalent to req.request . |
%{Foobar}n | Not supported. Always returns "-" . |
%{Foobar}o | The contents of Foobar: header lines in the reply. |
%O | Bytes sent, including headers. Cannot be zero. Equivalent to resp.bytes_written . |
%p | The canonical port of the server serving the request. Always returns 80 . Equivalent to server.port . |
%{format}p | The canonical port of the server serving the request. Valid formats are canonical , local , or remote . Returns 80 for HTTP requests and 443 for HTTPS requests. |
%P | Not supported. Always returns "-" . |
%{format}P | Not supported. Always returns "-" . |
%q | The query string (prepended with a ? if a query string exists, otherwise an empty string). Equivalent to req.url . |
%r | The first line of the request. |
%R | Not supported. Always returns "-" . |
%s | The status. For requests that got internally redirected, this is the status of the original request. Use %>s for the final status. Equivalent to resp.status . |
%t | The time the request was received, in Standard English format (e.g., 01/Jan/1970:00:00:00 -0700 ). The last number indicates the time zone offset from GMT. |
%{format}t | The time, in the form given by format , which should be in strftime(3) format (potentially localized). If the format starts with begin: (the default) the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request processing. In addition to the formats supported by strftime(3) , the following format tokens are supported: sec (number of seconds since the Epoch), msec (number of milliseconds since the Epoch), usec (number of microseconds since the Epoch), msec_frac (millisecond fraction), and usec_frac (microsecond fraction). Equivalent to time.start . |
%T | The time taken to serve the request, in seconds. Equivalent to time.elapsed.sec . |
%u | Not supported. Always returns "-" . |
%U | The URL path requested, not including any query string. Equivalent to req.url.path . |
%v | The domain name of the request. Equivalent to req.http.host . |
%V | The same as %v . Equivalent to req.http.host . |
%{vcl}V | The literal VCL to include without quoting. This can be used to write VCL variables to your logs (e.g., %{client.geo.country_code}V or %{tls.client.cipher}V ). This %-directive is a Fastly extension and is not found in Apache. |
%X | The connection status when response is completed. Always set as + (connection may be kept alive after the response is sent). |
Version 1 log format
This table details version 1 of Fastly's custom log formats. All variables should be prefixed by a percent sign (%
), as indicated in the table.
Format String | Description |
---|---|
%b | The content size of the response, calculated using the Content-Length header rather than actually checking the length of the response (and may therefore be wrong). |
%h | The remote IP address. |
%l | The remote log name. Always returns the hardcoded value "-" . |
%r | The HTTP verb and request path (e.g., GET /index.html ). Unlike Apache and version 2 log formats, the protocol version is not included. |
%>s | The status of the last request. |
%t | The time the request was received, in Unix ctime format (e.g., Thu, 01 Jan 1970 00:00:00 GMT ) rather than Apache's Standard English format (e.g., 01/Jan/1970:00:00:00 -0700 ). |
%u | Always returns "-" . |
Upgrading endpoints to use version 2 log format
WARNING
Upgrading is a permanent change. Logging objects using version 2 formatting cannot be downgraded to version 1.
Follow these instructions to upgrade a logging endpoint to the version 2 custom log format:
- Log in to the Fastly web interface.
- From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
- Click Edit configuration and then select the option to clone the active version.
Click Logging.
Click the name of a logging endpoint you want to edit.
Click Convert to Log Format Version 2.
Select an output format:
- Use compatible output is the recommended setting. This setting won't modify your timestamp format string, but your logs will be formatted differently. The new format will be compatible with Apache's log format.
- Maintain legacy output uses the version 2 parser, but the generated log string will be the same. This means that any instances of
%t
need to be turned into%{now}V
, any instances of%r
need to be turned into%{req.url}V
, and any instances of%b
need to be turned into%{resp.http.Content-Length}V
.
Click Select.
Click Update to upgrade the logging endpoint to the version 2 custom log format.
- Click Activate to deploy your configuration changes.
Using the API to upgrade
To upgrade a logging endpoint using the Fastly API, either clone the active version of the service you need upgraded or choose a version of the service that is unlocked and not active, then run the following command on that in development version:
$ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<logging endpoint>/<log name>' --data-binary '{"format_version":"2"}'
Keep in mind that the format_version
field is a per-object field. Updating it on one logging object will not change it on any other objects.
For example, to update a Google Cloud Storage logging endpoint named "GCS Test," the curl command would look something like this:
$ curl -X PUT -H 'Fastly-Key: FASTLY_API_TOKEN' -H 'Content-Type: application/json' 'https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/1/logging/gcs/GCS%20Test' --data-binary '{"format_version":"2"}'
NOTE
The log name
included a space that needed to be URL encoded (the space into %20
).
Determining which logging version is being used
To determine which logging version your service currently uses, issue the following curl command:
$ curl -X GET -H 'Fastly-Key: FASTLY_API_TOKEN' 'https://api.fastly.com/service/<your Fastly service ID>/version/<version number>/logging/<logging endpoint>/<log name>'
The curl command will produce JSON output detailing the configuration of your service version. For example:
1{2 "address": "logs.papertrailapp.com",3 "created_at": "2016-04-01T15:37:30+00:00",4 "deleted_at": null,5 "format": "time.start.msec time.to_first_byte time.elapsed req.body_bytes_read req.bytes_read resp.http.content-length server.region client.ip %>s \"req.method req.url req.proto\" \"req.http.referer\" \"req.http.user-agent\"",6 "format_version": "2",7 "hostname": "logs.papertrailapp.com",8 "name": "fastly",9 "port": "11111",10 "public_key": null,11 "response_condition": "LOG /",12 "service_id": "1a2b3c4d5e6f7g8h9j0k",13 "updated_at": "2016-04-01T19:47:47+00:00",14 "version": "123"15}
The format_version
field displays either a 1
or a 2
as appropriate for the custom log format being used.
Advantages of using the version 2 custom log format
The key advantages of using the version 2 custom log format include the following:
- Log lines are generated in
vcl_log
instead ofvcl_deliver
to allow us to accurately set the various size variables becausevcl_log
is run after the object has been delivered to the browser. - The
%t
time directive is compatible with Apache log format. In version 1, we used a non-standard time format. - The
%r
"first line of request" directive is compatible with Apache log format. In version 1, we incorrectly left off the protocol. - When using the
%b
directive, which represents the size of a response in bytes, excluding HTTP headers is more accurate. In version 1, we used the reported Content-Length from the origin, which could be inaccurate (especially with ESI). - We've added all Apache logging directives that make sense. In version 1, we used a smaller subset.
Making version 2 logs look like version 1
The default logging format for version 1 is as follows:
%h %l %u %t %r %>s
Most of the directives in version 2 are exactly the same - only %t
and %r
are different. After you upgrade to version 2 log formats, you can recreate the appearance of version 1 logs using the new %{...}V
directive, which allows you to specifically include VCL in logging directives:
%h %l %u %{now}V %{req.method}V %{req.url}V %>s
In addition, if you are using the %b
directive in version 1, then you can use this directive instead:
%{resp.http.Content-Length}V
Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.