Log streaming: Elasticsearch

Fastly's Real-Time Log Streaming feature can send log files to Elasticsearch. Elasticsearch is a distributed, RESTful search and analytics engine.

NOTE

Fastly does not provide direct support for third-party services. Read Fastly's Terms of Service for more information.

Prerequisites

Before adding Elasticsearch as a logging endpoint for Fastly services, ensure Elasticsearch is running on a remote server. You’ll need to know the endpoint URL that includes a port to which logs should be sent (make sure it can receive traffic from Fastly) and also the name of the index to send logs to. For more information on setting up Elasticsearch, check out the Elasticsearch setup documentation.

This logging endpoint works with all actively supported versions of Elasticsearch as well as some versions that have already reached their end-of-life. We also work with OpenSearch server integration. Other distributions that are API-compatible with Elasticsearch may also work but have not been explicitly tested and are not guaranteed.

Required privileges

We send data using the Bulk API via the index action. When using basic authentication, ensure that the required index privileges to use the index action are granted to the user role.

We also require access to the root path API of the Elasticsearch server. This API returns metadata about the server, such as the version number, that allows our integration to make the best choice about which bulk data API to use for each customer's server. Access to this API allows us to properly work with the wide range of Elasticsearch versions used by our customers as well as other Elasticsearch-compatible distributions.

Adding Elasticsearch as a logging endpoint

Follow these instructions to add Elasticsearch as a logging endpoint:

  1. Deliver services
  2. Compute services
  1. Review the information in our guide to setting up remote log streaming.
  2. In the Elasticsearch area, click Create endpoint.
  3. Fill out the Create an Elasticsearch endpoint fields as follows:
    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. Read our guide on changing log placement for more information.
    • In the Log format field, enter the data to send to Elasticsearch. See the example format section for details.
    • In the URL field, enter the Elasticsearch endpoint URL that includes a port to which logs should be sent. The URL must be sent using HTTPS on a port that can receive incoming TCP traffic from Fastly.
    • In the Index field, enter the name of the Elasticsearch index to send logs to. The index must follow the Elasticsearch index format rules. We support strftime interpolated variables inside braces prefixed with a pound symbol. For example, #{%F} will interpolate as YYYY-MM-DD with today's date.
    • (Optional) In the Pipeline field, enter the ID of the Elasticsearch ingest pipeline to apply pre-process transformations to before indexing (for example, my_pipeline_id).
    • (Optional) In the Maximum logs field, enter the maximum number of logs to append to a batch, if non-zero.
    • (Optional) In the Maximum bytes field, enter the maximum size of the log batch.
    • (Optional) In the BasicAuth user field, enter your basic authentication username.
    • (Optional) In the BasicAuth password field, enter your basic authentication password.
    • In the TLS hostname field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported.
    • (Optional) In the TLS CA certificate field, copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority.
    • (Optional) In the TLS client certificate field, copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection.
    • (Optional) In the TLS client key field, copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
  4. Click Create to create the new logging endpoint.
  5. Click Activate to deploy your configuration changes.

Example format

Data sent to Elasticsearch must be serialized as a JSON object. Here's an example format string for sending data to Elasticsearch:

1{
2 "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
3 "client_ip": "%{req.http.Fastly-Client-IP}V",
4 "geo_country": "%{client.geo.country_name}V",
5 "geo_city": "%{client.geo.city}V",
6 "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
7 "url": "%{json.escape(req.url)}V",
8 "request_method": "%{json.escape(req.method)}V",
9 "request_protocol": "%{json.escape(req.proto)}V",
10 "request_referer": "%{json.escape(req.http.referer)}V",
11 "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
12 "response_state": "%{json.escape(fastly_info.state)}V",
13 "response_status": %{resp.status}V,
14 "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
15 "response_body_size": %{resp.body_bytes_written}V,
16 "fastly_server": "%{json.escape(server.identity)}V",
17 "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
18}
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.