Log streaming: Coralogix

Fastly's Real-Time Log Streaming feature can send log files to Coralogix. Coralogix provides an analytics platform that allows you to detect abnormal behavior via dynamic alerts, ratio alerts, flow anomaly detection, and threat discovery.

Prerequisites

If you don't already have a Coralogix account, you'll need to register for one by following the signup instructions on the Coralogix website. Once you've signed up, navigate to the Send Your Logs area in the Settings section of your Coralogix dashboard and make note of your unique private key. Coralogix uses this to associate data you send them with your account. You'll need it when you set up your endpoint with Fastly.

Adding Coralogix as a logging endpoint

Follow these instructions to add Coralogix as a logging endpoint:

  1. Review the information in our Setting Up Remote Log Streaming guide.
  2. Click the HTTPS Create endpoint button. The Create an HTTPS endpoint page appears.
  3. Fill out the Create an HTTPS endpoint fields as follows:
    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. See our guide on changing log placement for more information.
    • In the Log format field, replace the placeholder log format and make the appropriate changes as shown in our log format and recommendations section below.
    • In the URL field, enter the Coralogix regional FluentD API URL. Refer to the Coralogix documentation for more detail. For example, the regional FluentD API URL for Coralogix accounts in the United States is https://api.coralogix.us/logs/rest/singles.
    • In the Maximum logs field, leave as 0 (the default).
    • In the Maximum bytes field, enter 2000000.
  4. Click the Advanced options link of the Create an HTTPS endpoint page. The Advanced options appear.
  5. Fill out the Advanced options of the Create an HTTPS endpoint page as follows:
    • In the Content type field, enter application/json.
    • In the Custom header name field, enter private_key.
    • In the Custom header value field, enter your Coralogix private key.
    • From the Method controls, select POST.
    • From the JSON log entry format controls, select Array of JSON.
    • Leave the Select a log line format controls set to the defaults.
    • Leave the remaining fields blank.
  6. Click the Create button to create the new logging endpoint.
  7. Click the Activate button to deploy your configuration changes.

Log format and field setting recommendations

Use the following log format:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
{
  "timestamp":%{time.start.msec}V,
  "applicationName":"fastly",
  "subsystemName":"%{req.service_id}V",
  "severity": 3,
  "json": {
    "time": {
        "start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
        "end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
        "elapsed":%D
    },
    "cdn_server": {
        "ip_ipaddr":"%A",
        "code":"%{server.datacenter}V",
        "hostname":"%{server.hostname}V",
        "region_code":"%{server.region}V",
        "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
        "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
        "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
        "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
        "h2_stream_id":"%{fastly_info.h2.stream_id}V"
    },
    "client": {
        "city_name":"%{client.geo.city.utf8}V",
        "country_code":"%{client.geo.country_code}V",
        "country_name":"%{client.geo.country_name}V",
        "continent_code":"%{client.geo.continent_code}V",
        "region":"%{client.geo.region}V",
        "ip_ipaddr":"%h",
        "name":"%{client.as.name}V",
        "number":"%{client.as.number}V",
        "connection_speed":"%{client.geo.conn_speed}V",
        "location_geopoint": {
            "lat":%{client.geo.latitude}V,
            "lon":%{client.geo.longitude}V
        }
    },
    "response": {
        "status":%>s,
        "content_type":"%{Content-Type}o",
        "age":"%{Age}o",
        "cache_control":"%{Cache-Control}o",
        "expires":"%{Expires}o",
        "last_modified":"%{Last-Modified}o",
        "tsv":"%{TSV}o",
        "header_size":%{resp.header_bytes_written}V,
        "body_size":%B
    },
    "request": {
        "host":"%{req.http.host}V",
        "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
        "backend":"%{req.backend}V",
        "service_id":"%{req.service_id}V",
        "url":"%{cstr_escape(req.url)}V",
        "url_ext":"%{req.url.ext}V",
        "header_size":%{req.header_bytes_read}V,
        "body_size":%{req.body_bytes_read}V,
        "method":"%m",
        "protocol":"%H",
        "referer":"%{Referer}i",
        "user_agent":"%{User-Agent}i",
        "accept_content":"%{Accept}i",
        "accept_language":"%{Accept-Language}i",
        "accept_encoding":"%{Accept-Encoding}i",
        "accept_charset":"%{Accept-Charset}i",
        "connection":"%{Connection}i",
        "dnt":"%{DNT}i",
        "forwarded":"%{Forwarded}i",
        "via":"%{Via}i",
        "cache_control":"%{Cache-Control}i",
        "x_requested_with":"%{X-Requested-With}i",
        "x_att_device_id":"%{X-ATT-Device-Id}i",
        "x_forwarded_for":"%{X-Forwarded-For}i"
    },
    "socket": {
        "cwnd":%{client.socket.cwnd}V,
        "pace":%{client.socket.pace}V,
        "nexthop":"%{client.socket.nexthop}V",
        "tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
        "tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
        "tcpi_rtt":%{client.socket.tcpi_rtt}V,
        "tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
        "tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
        "tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
        "tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
        "tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
        "tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
        "ploss":%{client.socket.ploss}V
    }
  }
}

The first five fields of the recommended format are required:

  • timestamp: Leave the format of this field unchanged.
  • applicationName: Enter the name of the application in this field.
  • subsystemName: Enter the name of the subsystem in this field. This is used to separate components. We use req.service_id in the example, which isn't particularly human readable. Use whatever subsystem name makes sense that helps you identify the subsystem.
  • severity: Specify the severity and apply it to all logs using the following choices: 1 (debug), 2 (verbose), 3 (info), 4 (warning), 5 (error), 6 (critical). This can be changed later using an extract rule as described below.
  • json (object): Add or remove fields as necessary. Static fields can be added. Nested JSON formats are supported including any fields described in the Fastly VCL reference.

    Specifying a nested response.status field is a useful way to identify the status for servicing the request. Using the Coralogix parsing rules, you can set a JSON Extract rule to use the status code value from the log to populate the severity field in the Coralogix interface. Specifically, you can automatically map an HTTP status code to a severity value. For example, status code 2xx will set the Coralogix severity as “INFO” and status code 4xx will set Coralogix severity as “ERROR”.

    In the Coralogix web interface, it will look like this:

    Creating a new Coralogix Rule

Configuring Coralogix dashboards and alerting

Coralogix provides tutorials for integrating their service with Fastly via dashboards and alerting. This includes examples of data dashboards created using Fastly data, including one for a general service overview, a visitor breakdown, and quality of service.

Their tutorials also describe how to set up user-defined alerts for situations like no logs being received from Fastly, outages at your origin, elevated error ratios and cache misses, unusual or suspicious requests of various types, as well as potential website defacement attempts.

Back to Top