Log streaming: Coralogix

      Last updated October 26, 2020

    Fastly's Real-Time Log Streaming feature can send log files to Coralogix. Coralogix provides an analytics platform that allows you to detect abnormal behavior via dynamic alerts, ratio alerts, flow anomaly detection, and threat discovery.

    Prerequisites

    If you don't already have a Coralogix account, you'll need to register for one by following the signup instructions on the Coralogix website. Once you've signed up, navigate to the Send Your Logs area in the Settings section of your Coralogix dashboard and make note of your unique private key. Coralogix uses this to associate data you send them with your account. You'll need it when you set up your endpoint with Fastly.

    If you're adding the Coralogix endpoint via the command line, instead of the web interface, you should also have your Fastly API token, the Fastly service ID, and version number of the Fastly service for which you'll be enabling Coralogix logging.

    Adding Coralogix as a logging endpoint

    Follow these instructions to add Coralogix as a logging endpoint:

    1. Review the information in our Setting Up Remote Log Streaming guide.
    2. Click the HTTPS Create endpoint button. The Create an HTTPS endpoint page appears.

      the create an HTTPS endpoint page

    3. Fill out the Create an HTTPS endpoint fields as follows:
      • In the Name field, enter a human-readable name for the endpoint.
      • In the Log format field, replace the placeholder log format and make the appropriate changes as shown in our log format and recommendations section below.
      • In the URL field, enter https://api.coralogix.com/logs/rest/singles.
      • In the Maximum logs field, leave as 0 (the default).
      • In the Maximum bytes field, type 2000000.
    4. Click the Advanced options link of the Create an HTTPS endpoint page. The Advanced options appear.

      the advanced options on the create an HTTPS endpoint page

    5. Fill out the Advanced options of the Create an HTTPS endpoint page as follows:
      • In the Content type field, enter application/json.
      • In the Custom header name field, enter private_key.
      • In the Custom header value field, enter your Coralogix private key.
      • From the Method controls, select POST.
      • From the JSON log entry format controls, select Array of JSON.
      • Leave the Select a log line format and Placement controls set to the defaults.
      • In the TLS hostname field, optionally type the hostname used to verify the origin server's certificate. This can be either the Common Name (CN) or Subject Alternate Name (SAN).
      • In the TLS CA certificate field, optionally copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority.
      • In the TLS client certificate field, optionally copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection.
      • In the TLS client key field, optionally copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection.
    6. Click the Create button to create the new logging endpoint.
    7. Click the Activate button to deploy your configuration changes.

    Log format and field setting recommendations

    Use the following log format:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    
    {
      "timestamp":%{time.start.msec}V,
      "applicationName":"fastly",
      "subsystemName":"%{req.service_id}V",
      "severity": 3,
      "json": {
        "time": {
            "start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
            "end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
            "elapsed":%D
        },
        "cdn_server": {
            "ip_ipaddr":"%A",
            "code":"%{server.datacenter}V",
            "hostname":"%{server.hostname}V",
            "region_code":"%{server.region}V",
            "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
            "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
            "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
            "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
            "h2_stream_id":"%{fastly_info.h2.stream_id}V"
        },
        "client": {
            "city_name":"%{client.geo.city.utf8}V",
            "country_code":"%{client.geo.country_code}V",
            "country_name":"%{client.geo.country_name}V",
            "continent_code":"%{client.geo.continent_code}V",
            "region":"%{client.geo.region}V",
            "ip_ipaddr":"%h",
            "name":"%{client.as.name}V",
            "number":"%{client.as.number}V",
            "connection_speed":"%{client.geo.conn_speed}V",
            "location_geopoint": {
                "lat":%{client.geo.latitude}V,
                "lon":%{client.geo.longitude}V
            }
        },
        "response": {
            "status":%>s,
            "content_type":"%{Content-Type}o",
            "age":"%{Age}o",
            "cache_control":"%{Cache-Control}o",
            "expires":"%{Expires}o",
            "last_modified":"%{Last-Modified}o",
            "tsv":"%{TSV}o",
            "header_size":%{resp.header_bytes_written}V,
            "body_size":%B
        },
        "request": {
            "host":"%{req.http.host}V",
            "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
            "backend":"%{req.backend}V",
            "service_id":"%{req.service_id}V",
            "url":"%{cstr_escape(req.url)}V",
            "url_ext":"%{req.url.ext}V",
            "header_size":%{req.header_bytes_read}V,
            "body_size":%{req.body_bytes_read}V,
            "method":"%m",
            "protocol":"%H",
            "referer":"%{Referer}i",
            "user_agent":"%{User-Agent}i",
            "accept_content":"%{Accept}i",
            "accept_language":"%{Accept-Language}i",
            "accept_encoding":"%{Accept-Encoding}i",
            "accept_charset":"%{Accept-Charset}i",
            "connection":"%{Connection}i",
            "dnt":"%{DNT}i",
            "forwarded":"%{Forwarded}i",
            "via":"%{Via}i",
            "cache_control":"%{Cache-Control}i",
            "x_requested_with":"%{X-Requested-With}i",
            "x_att_device_id":"%{X-ATT-Device-Id}i",
            "x_forwarded_for":"%{X-Forwarded-For}i"
        },
        "socket": {
            "cwnd":%{client.socket.cwnd}V,
            "pace":%{client.socket.pace}V,
            "nexthop":"%{client.socket.nexthop}V",
            "tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
            "tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
            "tcpi_rtt":%{client.socket.tcpi_rtt}V,
            "tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
            "tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
            "tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
            "tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
            "tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
            "tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
            "ploss":%{client.socket.ploss}V
        }
      }
    }
    

    The first five fields of the recommended format are required:

    The response.status field sends the request status. This is a recommended field. Using the Coralogix parsing rules, you can set a JSON Extract rule to extract the status code value from the request into a Coralogix severity, allowing you to define the severity to automatically determine the importance of the type of log. Specifically, you can automatically map HTTP status codes into a severity tag as appropriate. For example, status code 2xx will set the Coralogix severity as “INFO” and status code 4xx will set Coralogix severity as “ERROR”.

    In the Coralogix web interface, it will look like this:

    Creating a new Coralogix Rule

    Configuring Coralogix dashboards and alerting

    Coralogix provides tutorials for integrating their service with Fastly via dashboards and alerting. This includes examples of data dashboards created using Fastly data, including one for a general service overview, a visitor breakdown, and quality of service.

    Their tutorials also describe how to set up user-defined alerts for situations like no logs being received from Fastly, outages at your origin, elevated error ratios and cache misses, unusual or suspicious requests of various types, as well as potential website defacement attempts.

    Back to Top