Log streaming: Splunk

  Last updated 2024-11-12

Fastly's Real-Time Log Streaming feature can send log files to Splunk. Splunk is a web-based log analytics platform used by developers and IT teams.

NOTE

Fastly does not provide direct support for third-party services. Read Fastly's Terms of Service for more information.

Prerequisites

To use Splunk as a logging endpoint, you'll need to enable the HTTP Event Collector (HEC), create a token, and enable it. Follow the instructions on Splunk's website:

  1. Enable HEC.
  2. Create an HEC token.
  3. Enable the HEC token.
  4. Disable indexer acknowledgment for tokens used by Fastly to stream logs.

You'll need to remember the HEC token and find the URL for your collector. The URL structure depends on the type of Splunk instance you're using. Use the table below to find the URL structure for your Splunk instance.

TypeURL
Self hostedhttps://<hostname>:8088/services/collector/event
Self-service Splunk Cloud planshttps://input-<hostname>:8088/services/collector/event
All other Splunk Cloud planshttps://http-inputs-<hostname>:8088/services/collector/event

While logged in to Splunk, you can find the hostname for the URL in your web browser's address bar.

Adding Splunk as a logging endpoint

After you've created a Splunk account and obtained your customer token, follow these instructions to add Splunk as a logging endpoint for Fastly services:

  1. Deliver services
  2. Compute services
  1. Review the information in our guide to setting up remote log streaming.
  2. In the Splunk area, click Create endpoint.
  3. Fill out the Create a Splunk endpoint fields as follows:
    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. Read our guide on changing log placement for more information.
    • In the Log format field, enter an Apache-style string or VCL variables to use for log formatting. You can use our recommended log format.
    • In the URL field, enter the URL to send data to (e.g., https://<splunk_host>:8088/services/collector/event/1.0).
    • In the Token field, enter the token for the HEC.
    • (Optional) From the Use TLS controls, select whether or not to enable TLS. When you select Yes, additional TLS fields appear.
    • In the TLS hostname field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported. If you are using Splunk Enterprise see the Splunk Enterprise section below for more information.
    • In the TLS CA certificate field, enter the CA certificate used to verify that the origin's certificate is valid. It must be in PEM format. This is not required if your origin-side TLS certificate is signed by a well-known CA. See the using TLS CA certificates section for more information.
    • (Optional) In the TLS client certificate field, copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection. This field only appears when you select Yes from the Use TLS menu.
    • (Optional) In the TLS client key field, copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
    • (Optional) In the Maximum logs field, enter the maximum number of logs to append to a batch, if non-zero.
    • (Optional) In the Maximum bytes field, enter the maximum size of the log batch, if non-zero.
  4. Click Create to create the new logging endpoint.
  5. Click Activate to deploy your configuration changes.

We recommend using the following log format to send data to Splunk.

NOTE

All JSON sent to the Splunk HEC must have an event field. The event field can be text or nested JSON. There can also be other meta data in the payload. See the Splunk documentation for more information.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
{
  "time":%{time.start.sec}V,
  "host":"%{Fastly-Orig-Host}i",
  "event":  {
    "service_id":"%{req.service_id}V",
    "time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
    "time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
    "time_elapsed":%D,
    "client_ip":"%h",
    "client_as_name":"%{client.as.name}V",
    "client_as_number":"%{client.as.number}V",
    "client_connection_speed":"%{client.geo.conn_speed}V",
    "request":"%m",
    "protocol":"%H",
    "origin_host":"%v",
    "url":"%{json.escape(req.url)}V",
    "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
    "is_tls":%{if(req.is_ssl, "true", "false")}V,
    "tls_client_protocol":"%{json.escape(tls.client.protocol)}V",
    "tls_client_servername":"%{json.escape(tls.client.servername)}V",
    "tls_client_cipher":"%{json.escape(tls.client.cipher)}V",
    "tls_client_cipher_sha":"%{json.escape(tls.client.ciphers_sha )}V",
    "tls_client_tlsexts_sha":"%{json.escape(tls.client.tlsexts_sha)}V",
    "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
    "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
    "h2_stream_id":"%{fastly_info.h2.stream_id}V",
    "request_referer":"%{Referer}i",
    "request_user_agent":"%{User-Agent}i",
    "request_accept_content":"%{Accept}i",
    "request_accept_language":"%{Accept-Language}i",
    "request_accept_encoding":"%{Accept-Encoding}i",
    "request_accept_charset":"%{Accept-Charset}i",
    "request_connection":"%{Connection}i",
    "request_dnt":"%{DNT}i",
    "request_forwarded":"%{Forwarded}i",
    "request_via":"%{Via}i",
    "request_cache_control":"%{Cache-Control}i",
    "request_x_requested_with":"%{X-Requested-With}i",
    "request_x_att_device_id":"%{X-ATT-Device-Id}i",
    "request_x_forwarded_for":"%{X-Forwarded-For}i",
    "status":"%s",
    "content_type":"%{Content-Type}o",
    "response_state":"%{fastly_info.state}V",
    "response_age":"%{Age}o",
    "response_cache_control":"%{Cache-Control}o",
    "response_expires":"%{Expires}o",
    "response_last_modified":"%{Last-Modified}o",
    "response_tsv":"%{TSV}o",
    "server_datacenter":"%{server.datacenter}V",
    "server_ip":"%A",
    "geo_city":"%{client.geo.city.utf8}V",
    "geo_country_code":"%{client.geo.country_code}V",
    "geo_continent_code":"%{client.geo.continent_code}V",
    "geo_region":"%{client.geo.region}V",
    "req_header_size":%{req.header_bytes_read}V,
    "req_body_size":%{req.body_bytes_read}V,
    "resp_header_size":%{resp.header_bytes_written}V,
    "resp_body_size":%B,
    "socket_cwnd":%{client.socket.cwnd}V,
    "socket_nexthop":"%{client.socket.nexthop}V",
    "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
    "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
    "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,
    "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
    "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
    "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
    "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
    "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
    "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
    "socket_ploss":%{client.socket.ploss}V
  }
}

Using TLS CA certificates

If you've installed your own TLS certificate in Splunk Enterprise or Splunk Cloud, you'll need to provide the corresponding CA certificate.

Splunk Cloud

For Splunk Cloud, the default set up has the following CA certificate:

1
2
3
4
5
6
7
8
9
10
11
12
13
-----BEGIN CERTIFICATE-----
MIIB/DCCAaGgAwIBAgIBADAKBggqhkjOPQQDAjB+MSswKQYDVQQDEyJTcGx1bmsg
Q2xvdWQgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MRYwFAYDVQQHEw1TYW4gRnJhbmNp
c2NvMRMwEQYDVQQKEwpTcGx1bmsgSW5jMQswCQYDVQQIEwJDQTEVMBMGA1UECxMM
U3BsdW5rIENsb3VkMB4XDTE0MTExMDA3MDAxOFoXDTM0MTEwNTA3MDAxOFowfjEr
MCkGA1UEAxMiU3BsdW5rIENsb3VkIENlcnRpZmljYXRlIEF1dGhvcml0eTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzETMBEGA1UEChMKU3BsdW5rIEluYzELMAkGA1UE
CBMCQ0ExFTATBgNVBAsTDFNwbHVuayBDbG91ZDBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABPRRy9i3yQcxgMpvCSsI7Qe6YZMimUHOecPZWaGz5jEfB4+p5wT7dF3e
QrgjDWshVJZvK6KGO7nDh97GnbVXrTCjEDAOMAwGA1UdEwQFMAMBAf8wCgYIKoZI
zj0EAwIDSQAwRgIhALMUgLYPtICN9ci/ZOoXeZxUhn3i4wIo2mPKEWX0IcfpAiEA
8Jid6bzwUqAdDZPSOtaEBXV9uRIrNua0Qxl1S55TlWY=
-----END CERTIFICATE-----

Splunk Enterprise

Splunk Enterprise provides a set of default certificates, but we strongly recommend you configure your own certificates for your Fastly logging endpoint rather than relying on the default certificates. The certificates provided by Splunk Enterprise only specify a Common Name (CN), which cannot be used to properly verify the identity of the Splunk host presenting the certificate. Additionally, these certificates are less secure because the same root certificate is available in every Splunk Enterprise download. We encourage you to maintain the best possible security posture by configuring your own certificates rather than relying on the default certificates. The Splunk documentation provides a guide for configuring your own certificates.

Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Fastly
© Fastly 2024