LOG IN SIGN UP
Documentation

Log streaming: Datadog

  Last updated August 29, 2018

Fastly's Real-Time Log Streaming feature can be configured to send logs in a format readable by Datadog. Datadog is a cloud-based monitoring and analytics solution that allows you to see inside applications within your stack and aggregate the results.

Prerequisites

Before adding Datadog as a logging endpoint for Fastly services, you will need to:

Adding Datadog as a logging endpoint

After you've created a Datadog account, copied the CA certificate, and noted your Datadog API key, follow the steps below to add Datadog as a logging endpoint for Fastly services.

  1. Review the information in our Setting Up Remote Log Streaming guide.
  2. Click the syslog icon. The Create a Syslog endpoint page appears.

    the create a Syslog endpoint page

  3. Fill out the Create a Syslog endpoint fields as follows:
    • In the Name field, type a human-readable name for the endpoint.
    • In the Log format field, set the format to be the Datadog API key you noted earlier, followed by a space and then your log string in JSON format. Be sure to add the JSON as a single line to ensure proper parsing. We've described the use of this format below with additional suggestions.
    • In the Syslog address field, type intake.logs.datadoghq.com in the domain field before the colon, and in the port field after the colon type 10516.
    • From the TLS menu, select Yes to enable encryption for the syslog endpoint. The TLS Hostname and TLS CA Certificate fields will both appear.
    • In the TLS Hostname field, type intake.logs.datadoghq.com. This is the hostname Fastly will use to verify the syslog server's certificate.
    • In the TLS CA certificate field, enter the contents of the Datadog Root Certificate file you noted.
  4. Click the Advanced options link of the Create a Syslog endpoint page. The Advanced options appear.

    the advanced options on the create a Syslog endpoint page

  5. In the Select a log line format area, select Blank.
  6. Click the Create button to create the new logging endpoint.
  7. Click the Activate button to deploy your configuration changes.

Logs should begin appearing in your Datadog account a few seconds after you've created the endpoint and deployed your service changes. These logs can then be accessed via the Datadog Log Explorer on your Datadog account.

Using the JSON logging format

Datadog automatically parses log files created in JSON format, making this format the easiest way to get data into Datadog because no customized logging rules are required. In addition, Datadog recognizes several reserved fields, such as service and date.

For example, in the JSON below we've set service to the ID of the Fastly service that sent the log but you could also use a human-readable name or you could group all logs under a common name such as fastly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
{
  "ddsource": "fastly",
  "service": "%{req.service_id}V",
  "date": "%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
  "time_start": "%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
  "time_end": "%{end:%Y-%m-%dT%H:%M:%S%Z}t",
  "http": {
    "request_time_ms": %D,
    "method": "%m",
    "url": "%{json.escape(req.url)}V",
    "useragent": "%{User-Agent}i",
    "referer": "%{Referer}i",
    "protocol": "%H",
    "request_x_forwarded_for": "%{X-Forwarded-For}i",
    "status_code": "%s"
  },
  "network": {
    "client": {
     "ip": "%h",
     "name": "%{client.as.name}V",
     "number": "%{client.as.number}V",
     "connection_speed": "%{client.geo.conn_speed}V"
    },
   "destination": {
     "ip": "%A"
    },
  "geoip": {
  "geo_city": "%{client.geo.city.utf8}V",
  "geo_country_code": "%{client.geo.country_code}V",
  "geo_continent_code": "%{client.geo.continent_code}V",
  "geo_region": "%{client.geo.region}V"
  },
  "bytes_written": %B,
  "bytes_read": %{req.body_bytes_read}V
  },
  "host": "%{Fastly-Orig-Host}i",
  "origin_host": "%v",
  "is_ipv6": %{if(req.is_ipv6, "true", "false")}V,
  "is_tls": %{if(req.is_ssl, "true", "false")}V,
  "tls_client_protocol": "%{json.escape(tls.client.protocol)}V",
  "tls_client_servername": "%{json.escape(tls.client.servername)}V",
  "tls_client_cipher": "%{json.escape(tls.client.cipher)}V",
  "tls_client_cipher_sha": "%{json.escape(tls.client.ciphers_sha)}V",
  "tls_client_tlsexts_sha": "%{json.escape(tls.client.tlsexts_sha)}V",
  "is_h2": %{if(fastly_info.is_h2, "true", "false")}V,
  "is_h2_push": %{if(fastly_info.h2.is_push, "true", "false")}V,
  "h2_stream_id": "%{fastly_info.h2.stream_id}V",
  "request_accept_content": "%{Accept}i",
  "request_accept_language": "%{Accept-Language}i",
  "request_accept_encoding": "%{Accept-Encoding}i",
  "request_accept_charset": "%{Accept-Charset}i",
  "request_connection": "%{Connection}i",
  "request_dnt": "%{DNT}i",
  "request_forwarded": "%{Forwarded}i",
  "request_via": "%{Via}i",
  "request_cache_control": "%{Cache-Control}i",
  "request_x_requested_with": "%{X-Requested-With}i",
  "request_x_att_device_id": "%{X-ATT-Device-Id}i",
  "content_type": "%{Content-Type}o",
  "is_cacheable": %{if(fastly_info.state~"^(HIT|MISS)$", "true","false")}V,
  "response_age": "%{Age}o",
  "response_cache_control": "%{Cache-Control}o",
  "response_expires": "%{Expires}o",
  "response_last_modified": "%{Last-Modified}o",
  "response_tsv": "%{TSV}o",
  "server_datacenter": "%{server.datacenter}V",
  "req_header_size": %{req.header_bytes_read}V,
  "resp_header_size": %{resp.header_bytes_written}V,
  "socket_cwnd": %{client.socket.cwnd}V,
  "socket_nexthop": "%{client.socket.nexthop}V",
  "socket_tcpi_rcv_mss": %{client.socket.tcpi_rcv_mss}V,
  "socket_tcpi_snd_mss": %{client.socket.tcpi_snd_mss}V,
  "socket_tcpi_rtt": %{client.socket.tcpi_rtt}V,
  "socket_tcpi_rttvar": %{client.socket.tcpi_rttvar}V,
  "socket_tcpi_rcv_rtt": %{client.socket.tcpi_rcv_rtt}V,
  "socket_tcpi_rcv_space": %{client.socket.tcpi_rcv_space}V,
  "socket_tcpi_last_data_sent": %{client.socket.tcpi_last_data_sent}V,
  "socket_tcpi_total_retrans": %{client.socket.tcpi_total_retrans}V,
  "socket_tcpi_delta_retrans": %{client.socket.tcpi_delta_retrans}V,
  "socket_ploss": %{client.socket.ploss}V
}

Using logging formats other than JSON

The log format you specify doesn't have to be JSON. If you use other formats however, you'll need to parse the log manually instead. When selecting a format other than JSON, choose the format that best suits your needs (such as Apache Common Log Format) and then use Datadog's Grok Parser to extract the fields you want.

Fastly's guide to logging formats provides more information about using custom log formats with streaming logs. We also provide additional request and response variables for use with logging beyond the standard logging directives. Our guide to useful logging variables describes these in more detail.

When you use a logging format other than JSON, consider using Datadog's logging pipelines feature, which allows you to create a filtered subset of incoming logs. For example, you could use the User-Agent Parser to identify details like web browser, operating system, and device type and model, or the URL Parser to identify details like protocol, domain, and query string. Simply add these parsers to a Datadog pipeline and adjust the mapping as necessary.


Back to Top