Useful log formats

      Last updated January 26, 2021

    Different systems have standardized on different logging formats over time. Fastly believes logging should be as customizable as possible, working with whichever infrastructure you already have in place. This guide details some of the more complicated custom logging strings (e.g., JSON, Key/Value, CSV, and URL-encoded) you can use to implement the logging formats mentioned in the Apache logging module.

    Common Log Format (CLF)

    1
    
    %h %l %u %t "%r" %>s %b
    

    This is the default for many of our logging providers.

    Common Log Format with Virtual Host

    1
    
    %v %h %l %u %t "%r" %>s %b
    

    NCSA extended/combined log format

    1
    
    %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"
    

    Referer log format

    1
    
    %{Referer}i -> %U
    

    Agent (Browser) log format

    1
    
    %{User-agent}i
    

    Custom Tags to Loggly or RFC 5424 to another provider

    You’ll have to create a regular Syslog logging object pointing at your RFC 5424 compatible endpoint. For example, to send custom tags to Loggly, create a new Syslog object and set hostname to logs-01.loggly.com, port to 6514, use_tls to true, and the message format field to blank. You should also make sure the token field is blank. Then, in the format field, you would put the following:

    1
    
    <134>1 %{%Y-%m-%dT%TZ}t %{server.datacenter}V <log name> - - [<token>@<PEN> tag="fastly" tag="other-tag" id="12345" key="some-value" <tags>] <regular format string>
    

    The various fields you need to replace are:

    Structured data

    The examples below demonstrate different representations of the same variables and variable types:

    Name VCL Value Type Description
    Protocol req.protocol string The HTTP protocol version.
    Epoch Seconds time.start.sec number The time at the start of the request in seconds.
    Start Time begin:%Y-%m-%dT%H:%M:%S%z time The time at the start of the request in ISO 8601. format
    User Agent req.http.User-Agent escaped string The User-Agent request header.
    Is IPv6 req.is_ipv6 boolean Whether the request was made over IPv6 or not.
    ID deadbeef literal string A generic ID.
    Some String dwayne "the rock" johnson escaped literal string A string with quotation marks in it.
    Version 1.1 literal number A generic version number.

    JSON

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    
    {
      "protocol" : "%H",
      "epoch_seconds" : %{time.start.sec}V,
      "time_start" : "%{begin:%Y-%m-%dT%H:%M:%S%z}t",
      "user_agent" : "%{User-Agent}i",
      "is_ipv6" : %{if(req.is_ipv6, "true", "false")}V,
      "some_string":"%{json.escape(\{"dwayne "the rock" johnson"\})}V",
      "id" : "deadbeef",
      "version" : 1.1
    }
    

    CSV

    1
    
    %H, %{time.start.sec}V, %{begin:%Y-%m-%dT%H:%M:%S%z}t, %{regsub(req.http.User-Agent, \{"""\}, \{"""""\})}V, %{if(req.is_ipv6, "true", "false")}V, deadbeef, %{regsub(\{"dwayne "the rock" johnson"\}, \{"""\}, \{"""""\})}V, 1.1
    

    Key/Value

    1
    
    protocol:%H, epoch_seconds:%{time.start.sec}V, time_start:%{begin:%Y-%m-%dT%H:%M:%S%z}t, user_agent:%{User-Agent}i, is_ipv6:%{if(req.is_ipv6, "true", "false")}V, id:deadbeef, some_string:%{json.escape(\{"dwayne "the rock" johnson"\})}V, version:1.1
    

    URL Encoded

    1
    
    protocol=%H&epoch_seconds=%{time.start.sec}V&time_start=%{begin:%Y-%m-%dT%H:%M:%S%z}t&user_agent=%{urlencode(req.http.User-Agent)}i&is_ipv6=%{if(req.is_ipv6, "true", "false")}V&some_string=%{urlencode(\{"dwayne "the rock" johnson"\})}V&id=deadbeef&version=1.1
    
    Back to Top