Getting started
Basics
Domains & Origins
Performance

Configuration
Basics
Conditions
Dictionaries
Domains & Origins
Request settings
Cache settings
Headers
Responses
Performance
Purging
Custom VCL
Image optimization
Video

Security
Access Control Lists
Monitoring and testing
Securing communications
Security measures
TLS
Web Application Firewall

Integrations
Logging endpoints
Non-Fastly services

Diagnostics
Streaming logs
Debugging techniques
Common errors

Account info
Account management
Billing
User access and control

Reference

    Log streaming: Elasticsearch

      Last updated October 07, 2019

    Fastly's Real-Time Log Streaming feature can send log files to Elasticsearch. Elasticsearch is a distributed, RESTful search and analytics engine.

    Prerequisites

    Before adding Elasticsearch as a logging endpoint for Fastly services, ensure Elasticsearch is running on a remote server. You’ll need to know the endpoint URL that includes a port to which logs should be sent (make sure it can receive traffic from Fastly) and also the name of the index to send logs to. For more information on setting up Elasticsearch, see the Elasticsearch setup documentation.

    Adding Elasticsearch as a logging endpoint

    Follow these instructions to add Elasticsearch as a logging endpoint:

    1. Review the information in our Setting Up Remote Log Streaming guide.
    2. Click the Elasticsearch logo. The Create an Elasticsearch endpoint page appears.

      the create an Elasticsearch endpoint page

    3. Fill out the Create an Elasticsearch endpoint fields as follows:
      • In the Name field, type a human-readable name for the endpoint.
      • In the Log format field, enter the data to send to Elasticsearch. See the example format section for details.
      • In the URL field, type the Elasticsearch endpoint URL that includes a port to which logs should be sent. Be sure this port can receive incoming TCP traffic from Fastly.
      • In the Index field, enter the name of the Elasticsearch index to send logs to. The index must follow the Elasticsearch index format rules. We support strftime interpolated variables inside braces prefixed with a pound symbol. For example, #{%F} will interpolate as YYYY-MM-DD with today's date.
      • In the Maximum logs field, optionally enter the maximum number of logs to append to a batch, if non-zero.
      • In the Maximum bytes field, optionally enter the maximum size of log batch.
      • In the BasicAuth user field, optionally enter your basic authentication username.
      • In the BasicAuth password field, optionally enter your basic authentication password.
      • In the TLS Hostname field, optionally type the hostname used to verify the server's certificate. This can be either the Common Name (CN) or Subject Alternate Name (SAN).
      • In the TLS CA certificate field, optionally copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority.
      • In the TLS client certificate field, optionally copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection.
      • In the TLS client key field, optionally copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
    4. Click the Advanced options link of the Create an Elasticsearch endpoint page. The Advanced options appear.

      the advanced options on the create an Elasticsearch endpoint page

    5. In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about waf_debug_log.
    6. Click the Create button to create the new logging endpoint.
    7. Click the Activate button to deploy your configuration changes.

    Example format

    Data sent to Elasticsearch must be serialized as a JSON object. Here's an example format string for sending data to Elasticsearch:

    {
      "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
      "time_elapsed":%{time.elapsed.usec}V,
      "is_tls":%{if(req.is_ssl, "true", "false")}V,
      "client_ip":"%{req.http.Fastly-Client-IP}V",
      "geo_city":"%{client.geo.city}V",
      "geo_country_code":"%{client.geo.country_code}V",
      "request":"%{req.method}V",
      "host":"%{req.http.Fastly-Orig-Host}V",
      "url":"%{json.escape(req.url)}V",
      "request_referer":"%{json.escape(req.http.Referer)}V",
      "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
      "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
      "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
      "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V"
    }
    
    Back to Top