LOG IN SIGN UP
Documentation

Log streaming: Splunk

  Last updated October 08, 2018

Fastly's Real-Time Log Streaming feature can send log files to Splunk. Splunk is a web-based log analytics platform used by developers and IT teams.

Prerequisites

To use Splunk as a logging endpoint, you'll need to enable the HTTP Event Collector (HEC), create a token, and enable it. Follow the instructions on Splunk's website:

  1. Enable HEC.
  2. Create an HEC token.
  3. Enable the HEC token.

You'll need to remember the HEC token and find the URL for your collector. The URL structure depends on the type of Splunk instance you're using. Use the table below to find the URL structure for your Splunk instance.

Type URL
Self hosted https://<hostname>:8088/services/collector/event
Self-service Splunk Cloud plans https://input-<hostname>:8088/services/collector/event
All other Splunk Cloud plans https://http-inputs-<hostname>:8088/services/collector/event

While logged in to Splunk, you can find the hostname for the URL in your web browser's address bar.

Adding Splunk as a logging endpoint

After you've created a Splunk account and obtained your customer token, follow these instructions to add Splunk as a logging endpoint for Fastly services:

  1. Review the information in our Setting Up Remote Log Streaming guide.
  2. Click the Splunk logo. The Create a Splunk endpoint page appears.

    the create a Splunk endpoint page

  3. Fill out the Create a Splunk endpoint fields as follows:
    • In the Name field, type a human-readable name for the endpoint.
    • In the Log format field, type an Apache-style string or VCL variables to use for log formatting. You can use our recommended log format.
    • In the URL field, type the URL to send data to (e.g., https://<splunk_host>:8088/services/collector/event/1.0).
    • In the TLS hostname field, type the hostname used to verify the server's certificate. If you're using Splunk Enterprise, type SplunkServerDefaultCert.
    • In the TLS CA certificate field, type the CA certificate used to verify that the origin's certificate is valid. It must be in PEM format. This is not required if your origin-side TLS certificate is signed by a well-known CA. See the using TLS CA certificates section for more information.
    • In the Token field, type the token for the HEC.
  4. Click the Create button to create the new logging endpoint.
  5. Click the Activate button to deploy your configuration changes.

We recommend using the following log format to send data to Splunk.

{
  "time":%{time.start.sec}V,
  "event":  {
    "service_id":"%{req.service_id}V",
    "time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
    "time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
    "time_elapsed":%D,
    "client_ip":"%h",
    "client_as_name":"%{client.as.name}V",
    "client_as_number":"%{client.as.number}V",
    "client_connection_speed":"%{client.geo.conn_speed}V",
    "request":"%m",
    "protocol":"%H",
    "host":"%{Fastly-Orig-Host}i",
    "origin_host":"%v",
    "url":"%{cstr_escape(req.url)}V",
    "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
    "is_tls":%{if(req.is_ssl, "true", "false")}V,
    "tls_client_protocol":"%{cstr_escape(tls.client.protocol)}V",
    "tls_client_servername":"%{cstr_escape(tls.client.servername)}V",
    "tls_client_cipher":"%{cstr_escape(tls.client.cipher)}V",
    "tls_client_cipher_sha":"%{cstr_escape(tls.client.ciphers_sha )}V",
    "tls_client_tlsexts_sha":"%{cstr_escape(tls.client.tlsexts_sha)}V",
    "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
    "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
    "h2_stream_id":"%{fastly_info.h2.stream_id}V",
    "request_referer":"%{Referer}i",
    "request_user_agent":"%{User-Agent}i",
    "request_accept_content":"%{Accept}i",
    "request_accept_language":"%{Accept-Language}i",
    "request_accept_encoding":"%{Accept-Encoding}i",
    "request_accept_charset":"%{Accept-Charset}i",
    "request_connection":"%{Connection}i",
    "request_dnt":"%{DNT}i",
    "request_forwarded":"%{Forwarded}i",
    "request_via":"%{Via}i",
    "request_cache_control":"%{Cache-Control}i",
    "request_x_requested_with":"%{X-Requested-With}i",
    "request_x_att_device_id":"%{X-ATT-Device-Id}i",
    "request_x_forwarded_for":"%{X-Forwarded-For}i",
    "status":"%s",
    "content_type":"%{Content-Type}o",
    "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
    "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
    "response_age":"%{Age}o",
    "response_cache_control":"%{Cache-Control}o",
    "response_expires":"%{Expires}o",
    "response_last_modified":"%{Last-Modified}o",
    "response_tsv":"%{TSV}o",
    "server_datacenter":"%{server.datacenter}V",
    "server_ip":"%A",
    "geo_city":"%{client.geo.city.utf8}V",
    "geo_country_code":"%{client.geo.country_code}V",
    "geo_continent_code":"%{client.geo.continent_code}V",
    "geo_region":"%{client.geo.region}V",
    "req_header_size":%{req.header_bytes_read}V,
    "req_body_size":%{req.body_bytes_read}V,
    "resp_header_size":%{resp.header_bytes_written}V,
    "resp_body_size":%B,
    "socket_cwnd":%{client.socket.cwnd}V,
    "socket_nexthop":"%{client.socket.nexthop}V",
    "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
    "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
    "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,
    "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
    "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
    "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
    "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
    "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
    "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
    "socket_ploss":%{client.socket.ploss}V
  }
}

Using TLS CA certificates

If you've installed your own TLS certificate in Splunk Enterprise or Splunk Cloud, you'll need to provide the corresponding CA certificate.

Splunk Cloud

For Splunk Cloud, the default set up has the following CA certificate:

-----BEGIN CERTIFICATE-----
MIIB/DCCAaGgAwIBAgIBADAKBggqhkjOPQQDAjB+MSswKQYDVQQDEyJTcGx1bmsg
Q2xvdWQgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MRYwFAYDVQQHEw1TYW4gRnJhbmNp
c2NvMRMwEQYDVQQKEwpTcGx1bmsgSW5jMQswCQYDVQQIEwJDQTEVMBMGA1UECxMM
U3BsdW5rIENsb3VkMB4XDTE0MTExMDA3MDAxOFoXDTM0MTEwNTA3MDAxOFowfjEr
MCkGA1UEAxMiU3BsdW5rIENsb3VkIENlcnRpZmljYXRlIEF1dGhvcml0eTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzETMBEGA1UEChMKU3BsdW5rIEluYzELMAkGA1UE
CBMCQ0ExFTATBgNVBAsTDFNwbHVuayBDbG91ZDBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABPRRy9i3yQcxgMpvCSsI7Qe6YZMimUHOecPZWaGz5jEfB4+p5wT7dF3e
QrgjDWshVJZvK6KGO7nDh97GnbVXrTCjEDAOMAwGA1UdEwQFMAMBAf8wCgYIKoZI
zj0EAwIDSQAwRgIhALMUgLYPtICN9ci/ZOoXeZxUhn3i4wIo2mPKEWX0IcfpAiEA
8Jid6bzwUqAdDZPSOtaEBXV9uRIrNua0Qxl1S55TlWY=
-----END CERTIFICATE-----

Splunk Enterprise

In the Fastly web interface, type SplunkServerDefaultCert in the TLS hostname field.

For Splunk Enterprise, the default set up has the following CA certificate.

-----BEGIN CERTIFICATE-----
MIIDejCCAmICCQCNHBN8tj/FwzANBgkqhkiG9w0BAQsFADB/MQswCQYDVQQGEwJV
UzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDzANBgNVBAoM
BlNwbHVuazEXMBUGA1UEAwwOU3BsdW5rQ29tbW9uQ0ExITAfBgkqhkiG9w0BCQEW
EnN1cHBvcnRAc3BsdW5rLmNvbTAeFw0xNzAxMzAyMDI2NTRaFw0yNzAxMjgyMDI2
NTRaMH8xCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZy
YW5jaXNjbzEPMA0GA1UECgwGU3BsdW5rMRcwFQYDVQQDDA5TcGx1bmtDb21tb25D
QTEhMB8GCSqGSIb3DQEJARYSc3VwcG9ydEBzcGx1bmsuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzB9ltVEGk73QvPlxXtA0qMW/SLDQlQMFJ/C/
tXRVJdQsmcW4WsaETteeWZh8AgozO1LqOa3I6UmrWLcv4LmUAh/T3iZWXzHLIqFN
WLSVU+2g0Xkn43xSgQEPSvEK1NqZRZv1SWvx3+oGHgu03AZrqTj0HyLujqUDARFX
sRvBPW/VfDkomHj9b8IuK3qOUwQtIOUr+oKx1tM1J7VNN5NflLw9NdHtlfblw0Ys
5xI5Qxu3rcCxkKQuwz9KRe4iijOIRMAKX28pbakxU9Nk38Ac3PNadgIk0s7R829k
980sqGWkd06+C17OxgjpQbvLOR20FtmQybttUsXGR7Bp07YStwIDAQABMA0GCSqG
SIb3DQEBCwUAA4IBAQCxhQd6KXP2VzK2cwAqdK74bGwl5WnvsyqdPWkdANiKksr4
ZybJZNfdfRso3fA2oK1R8i5Ca8LK3V/UuAsXvG6/ikJtWsJ9jf+eYLou8lS6NVJO
xDN/gxPcHrhToGqi1wfPwDQrNVofZcuQNklcdgZ1+XVuotfTCOXHrRoNmZX+HgkY
gEtPG+r1VwSFowfYqyFXQ5CUeRa3JB7/ObF15WfGUYplbd3wQz/M3PLNKLvz5a1z
LMNXDwN5Pvyb2epyO8LPJu4dGTB4jOGpYLUjG1UUqJo9Oa6D99rv6sId+8qjERtl
ZZc1oaC0PKSzBmq+TpbR27B8Zra3gpoA+gavdRZj
-----END CERTIFICATE-----

Back to Top