Log streaming: Hydrolix

Fastly's Real-Time Log Streaming feature can send log files to Hydrolix, a cloud-based time-series data platform. Hydrolix provides a native integration for Fastly log storage and analysis through Fastly's HTTPS logging endpoint. Hydrolix lets you ingest and query those logs in real-time.

Prerequisites

If you don't already have a Hydrolix account, you'll need to sign up on the Hydrolix website. You'll also need to know the following about the target Hydrolix environment:

Adding Hydrolix as a logging endpoint

Follow these instructions to add Hydrolix as a logging endpoint:

  1. Review the information in our Setting Up Remote Log Streaming guide.
  2. Click the HTTPS Create endpoint button. The Create an HTTPS endpoint page appears.

    the create an HTTPS endpoint page

  3. Fill out the Create an HTTPS endpoint fields as follows:
    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. See our guide on changing log placement for more information.
    • In the Log format field, replace the placeholder log format and make the appropriate changes as shown in our log format and recommendations section below.
    • In the URL field, enter https://<hydrolix-instance>.hydrolix.live/ingest/event, replacing <hydrolix-instance> with the name of your Hydrolix instance.
    • In the Maximum logs field, leave as 0 (the default).
    • In the Maximum bytes field, enter 0.
  4. Click the Advanced options link of the Create an HTTPS endpoint page. The Advanced options appear.
  5. Fill out the Advanced options of the Create an HTTPS endpoint page as follows:
    • In the Content type field, enter application/json.
    • In the Custom header name field, enter x-hdx-table.
    • In the Custom header value field, enter <hydrolix-project-name>.<hydrolix-table-name>, substituting the values for your Hydrolix project and table names.
    • From the Method controls, select POST.
    • From the JSON log entry format controls, select Newline delimited.
    • Leave the Select a log line format and Placement controls set to the defaults.
    • In the TLS hostname field, optionally enter a hostname to verify the server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported.
    • Leave TLS CA certificate field, TLS client certificate field and TLS client key field all empty.
  6. Click the Create button to create the new logging endpoint.
  7. Click the Activate button to deploy your configuration changes.

Log format recommendations

For this example we use the log format shown below. You can customize this format with any values you want as long as you also modify your Transform and View configurations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
  "time_elapsed":%{time.elapsed.usec}V,
  "is_tls":%{if(req.is_ssl, "true", "false")}V,
  "client_ip":"%{req.http.Fastly-Client-IP}V",
  "geo_city":"%{client.geo.city}V",
  "geo_country_code":"%{client.geo.country_code}V",
  "request":"%{req.request}V",
  "host":"%{req.http.Fastly-Orig-Host}V",
  "url":"%{json.escape(req.url)}V",
  "request_referer":"%{json.escape(req.http.Referer)}V",
  "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
  "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
  "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
  "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V"
}

Configuring Hydrolix Streaming Intake

Once you have your project and table setup and Fastly is configured to send logs to your Hydrolix instance, you can focus on setting up the Fastly Log streaming ingest pipeline by defining an ingest transform schema.

Creating a transform schema

Below is the suggested transform schema to use with the recommended log format described above. Be sure to replace <table uuid> with the UUID of your target Hydrolix table. You need to have this transform setup as the default, so "is_default" is set to true.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
{
    "name": "fastly_transform",
    "type": "json",
    "table": "<table uuid>",
    "description": "fastly https logs",
    "settings": {
        "is_default": true,
        "output_columns": [
                            {
                                "position": 0,
                                "name": "timestamp",
                                "type": "datetime",
                                "format": "2006-01-02T15:04:05",
                                "treatment": "primary"
                            },
                            {
                                "position": 1,
                                "name": "time_elapsed",
                                "type": "uint64",
                                "treatment": "tag"
                            },
                            {
                                "position": 2,
                                "name": "is_tls",
                                "type": "bool",
                                "treatment": "tag"
                            },
                            {
                                "position": 3,
                                "name": "client_ip",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 4,
                                "name": "geo_city",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 5,
                                "name": "geo_country_code",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 6,
                                "name": "request",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 7,
                                "name": "host",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 8,
                                "name": "url",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 9,
                                "name": "request_referer",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 10,
                                "name": "request_user_agent",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 11,
                                "name": "request_accept_language",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 12,
                                "name": "request_accept_charset",
                                "type": "string",
                                "treatment": "tag"
                            },
                            {
                                "position": 13,
                                "name": "cache_status",
                                "type": "string",
                                "treatment": "tag"
                            }
                        ]
        }
    }

Once you define the transform schema, Hydrolix is configured to accept the incoming Fastly log data.

Leveraging views

Hydrolix supports having many different query formats for a single dataset. The query data structure, or view, for a given dataset allows you to customized the queried data and restrict a user’s access to a specific set of columns.

Hydrolix automatically generates a default view upon transform creation that can be used to immediately query the dataset - no additional configuration is required. However, you are encouraged to familiarize yourself with the view concept and the benefits that the feature can provide. More detailed information can be found on the Hydrolix site.

Further reading

Hydrolix provides a tutorial for querying and analyzing Fastly data from withing their system.

Back to Top