Useful log formats

Different systems have standardized on different logging formats over time. Fastly believes logging should be as customizable as possible, working with whichever infrastructure you already have in place. This guide details some of the more complicated custom logging strings (e.g., JSON, Key/Value, CSV, and URL-encoded) you can use to implement the logging formats mentioned in the Apache logging module.

NOTE

Fastly provides two versions of custom log formats. The version 2 logging formats, used by default when you create a new logging endpoint, improve compatibility with Apache’s logging directives.

IMPORTANT

Be sure to take into account security, privacy, and compliance requirements when making configuration and endpoint decisions for the data you intend to include in streamed logs.

TIP

You can log any Varnish variable or Fastly's extensions to VCL. Consider reading our guide to useful variables to log.

Common Log Format (CLF)

%h %l %u %t "%r" %>s %b

This is the default for many of our logging providers.

Common Log Format with Virtual Host

%v %h %l %u %t "%r" %>s %b

NCSA extended/combined log format

%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"

Referer log format

%{Referer}i -> %U

Agent (Browser) log format

%{User-agent}i

Custom Tags to Loggly or RFC 5424 to another provider

You’ll have to create a regular Syslog logging object pointing at your RFC 5424 compatible endpoint. For example, to send custom tags to Loggly, create a new Syslog object and set hostname to logs-01.loggly.com, port to 6514, use_tls to true, and the message format field to blank. You should also make sure the token field is blank. Then, in the format field, you would put the following:

<134>1 %{%Y-%m-%dT%TZ}t %{server.datacenter}V <log name> - - [<token>@<PEN> tag="fastly" tag="other-tag" id="12345" key="some-value" <tags>] <regular format string>

The various fields you need to replace are:

  • log name - this can be whatever you want but we recommend using the same name as you’ve used for the logging object in Fastly.
  • token - the private token for your RFC5424 endpoint (if sending to Loggly this is your Customer Token).
  • PEN - this is a Private Enterprise Number. For example the Loggly PEN (Private Enterprise Number) is 41058. If you want to send to another provider then you can look up their PEN on the IANA registry and use that.
  • tags - these can be any key/value pairs you want. Two appear in the above example: fastly and other-tag. Valid tag values include all alpha-numeric characters, the dash (-), the period (.), and the underscore (_). Tag values with spaces aren't valid and will be dropped by many logging providers such as Loggly. Additional information about tags, their restrictions, and details about how Loggly parses tags, can be found in the Loggly documentation but is useful information for sending data to all RFC 5424 compatible endpoints.
  • regular format string - this is regular Fastly logging directives, put whatever you want here (for example: the Common Log Format mentioned above).

Structured data

The examples below demonstrate different representations of the same variables and variable types:

NameVCL ValueTypeDescription
Protocolreq.protocolstringThe HTTP protocol version.
Epoch Secondstime.start.secnumberThe time at the start of the request in seconds.
Start Timebegin:%Y-%m-%dT%H:%M:%S%ztimeThe time at the start of the request in ISO 8601. format
User Agentreq.http.User-Agentescaped stringThe User-Agent request header.
Is IPv6req.is_ipv6booleanWhether the request was made over IPv6 or not.
IDdeadbeefliteral stringA generic ID.
Some Stringdwayne "the rock" johnsonescaped literal stringA string with quotation marks in it.
Version1.1literal numberA generic version number.

JSON

1{
2 "protocol" : "%H",
3 "epoch_seconds" : %{time.start.sec}V,
4 "time_start" : "%{begin:%Y-%m-%dT%H:%M:%S%z}t",
5 "user_agent" : "%{User-Agent}i",
6 "is_ipv6" : %{if(req.is_ipv6, "true", "false")}V,
7 "some_string":"%{json.escape(\{"dwayne "the rock" johnson"\})}V",
8 "id" : "deadbeef",
9 "version" : 1.1
10}
IMPORTANT

When logging to BigQuery, any TIMESTAMP field requires using the timestamp format in the JSON above. When using the DATETIME type instead, the timezone designator should be omitted (i.e., use `"%{begin:%Y-%m-%dT%H:%M:%S}t",` instead).

CSV

%H, %{time.start.sec}V, %{begin:%Y-%m-%dT%H:%M:%S%z}t, %{regsub(req.http.User-Agent, \{"""\}, \{"""""\})}V, %{if(req.is_ipv6, "true", "false")}V, deadbeef, %{regsub(\{"dwayne "the rock" johnson"\}, \{"""\}, \{"""""\})}V, 1.1

Key/Value

protocol:%H, epoch_seconds:%{time.start.sec}V, time_start:%{begin:%Y-%m-%dT%H:%M:%S%z}t, user_agent:%{User-Agent}i, is_ipv6:%{if(req.is_ipv6, "true", "false")}V, id:deadbeef, some_string:%{json.escape(\{"dwayne "the rock" johnson"\})}V, version:1.1

URL Encoded

protocol=%H&epoch_seconds=%{time.start.sec}V&time_start=%{begin:%Y-%m-%dT%H:%M:%S%z}t&user_agent=%{urlencode(req.http.User-Agent)}i&is_ipv6=%{if(req.is_ipv6, "true", "false")}V&some_string=%{urlencode(\{"dwayne "the rock" johnson"\})}V&id=deadbeef&version=1.1
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support@fastly.com.