Log streaming: Elasticsearch

1. Review the information in our guide to setting up remote log streaming.
2. In the Elasticsearch area, click Create endpoint.
3. Fill out the Create an Elasticsearch endpoint fields as follows:
   • In the Name field, enter a human-readable name for the endpoint.
   • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. Read our guide on changing log placement for more information.
   • In the Log format field, enter the data to send to Elasticsearch. See the example format section for details.
   • In the URL field, enter the Elasticsearch endpoint URL that includes a port to which logs should be sent. The URL must be sent using HTTPS on a port that can receive incoming TCP traffic from Fastly.
   • In the Index field, enter the name of the Elasticsearch index to send logs to. The index must follow the Elasticsearch index format rules. We support strftime interpolated variables inside braces prefixed with a pound symbol. For example, #{%F} will interpolate as YYYY-MM-DD with today's date.
   • (Optional) In the Pipeline field, enter the ID of the Elasticsearch ingest pipeline to apply pre-process transformations to before indexing (for example, my_pipeline_id).
   • (Optional) In the Maximum logs field, enter the maximum number of logs to append to a batch, if non-zero.
   • (Optional) In the Maximum bytes field, enter the maximum size of the log batch.
   • (Optional) In the BasicAuth user field, enter your basic authentication username.
   • (Optional) In the BasicAuth password field, enter your basic authentication password.
   • In the TLS hostname field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported.
   • (Optional) In the TLS CA certificate field, copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority.
   • (Optional) In the TLS client certificate field, copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection.
   • (Optional) In the TLS client key field, copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
4. Click Create to create the new logging endpoint.
5. Click Activate to deploy your configuration changes.

Example format

Data sent to Elasticsearch must be serialized as a JSON object. Here's an example format string for sending data to Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

{
  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
  "client_ip": "%{req.http.Fastly-Client-IP}V",
  "geo_country": "%{client.geo.country_name}V",
  "geo_city": "%{client.geo.city}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
  "url": "%{json.escape(req.url)}V",
  "request_method": "%{json.escape(req.method)}V",
  "request_protocol": "%{json.escape(req.proto)}V",
  "request_referer": "%{json.escape(req.http.referer)}V",
  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
  "response_state": "%{json.escape(fastly_info.state)}V",
  "response_status": %{resp.status}V,
  "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
  "response_body_size": %{resp.body_bytes_written}V,
  "fastly_server": "%{json.escape(server.identity)}V",
  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
}