Log streaming: Hydrolix

  Last updated 2023-06-06

Fastly's Real-Time Log Streaming feature can send log files to Hydrolix, a cloud-based time-series data platform. Hydrolix provides a native integration for Fastly log storage and analysis through Fastly's HTTPS logging endpoint. Hydrolix lets you ingest and query those logs in real-time.

NOTE

Fastly does not provide direct support for third-party services. Read Fastly's Terms of Service for more information.

Prerequisites

If you don't already have a Hydrolix account, you'll need to sign up on the Hydrolix website. You'll also need to know the following about the target Hydrolix environment:

TIP

Consider reading Hydrolix's documentation on integrating with Fastly.

Adding Hydrolix as a logging endpoint

NOTE

This logging endpoint is only available for Fastly's Deliver (VCL-based) services, not for Compute services.

Follow these instructions to add Hydrolix as a logging endpoint:

  1. Deliver services
  1. Review the information in our guide to setting up remote log streaming.
  2. In the HTTPS area, click Create endpoint button.
  3. Fill out the Create an HTTPS endpoint fields as follows:
    • In the Name field, enter a human-readable name for the endpoint.
    • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. Read our guide on changing log placement for more information.
    • In the Log format field, replace the placeholder log format and make the appropriate changes as shown in our log format and recommendations section below.
    • In the URL field, enter https://<hydrolix-instance>.hydrolix.live/ingest/event, replacing <hydrolix-instance> with the name of your Hydrolix instance.
    • In the Maximum logs field, leave as 0 (the default).
    • In the Maximum bytes field, enter 0.
  4. Click the Advanced options link of the Create an HTTPS endpoint page. The Advanced options appear.
  5. Fill out the Advanced options of the Create an HTTPS endpoint page as follows:
    • In the Content type field, enter application/json.
    • In the Custom header name field, enter x-hdx-table.
    • In the Custom header value field, enter <hydrolix-project-name>.<hydrolix-table-name>, substituting the values for your Hydrolix project and table names.
    • From the Method controls, select POST.
    • From the JSON log entry format controls, select Newline delimited.
    • Leave the Select a log line format and Placement controls set to the defaults.
    • In the TLS hostname field, optionally enter a hostname to verify the logging destination server's certificate. This should be one of the Subject Alternative Name (SAN) fields for the certificate. Common Names (CN) are not supported.
    • Leave TLS CA certificate field, TLS client certificate field and TLS client key field all empty.
  6. Click Create to create the new logging endpoint.
  7. Click Activate to deploy your configuration changes.

Log format recommendations

For this example we use the log format shown below. You can customize this format with any values you want as long as you also modify your Transform and View configurations.

1{
2  "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
3  "time_elapsed":%{time.elapsed.usec}V,
4  "is_tls":%{if(req.is_ssl, "true", "false")}V,
5  "client_ip":"%{req.http.Fastly-Client-IP}V",
6  "geo_city":"%{client.geo.city}V",
7  "geo_country_code":"%{client.geo.country_code}V",
8  "request":"%{req.request}V",
9  "host":"%{req.http.Fastly-Orig-Host}V",
10  "url":"%{json.escape(req.url)}V",
11  "request_referer":"%{json.escape(req.http.Referer)}V",
12  "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
13  "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
14  "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
15  "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V"
16}

Configuring Hydrolix Streaming Intake

Once you have your project and table setup and Fastly is configured to send logs to your Hydrolix instance, you can focus on setting up the Fastly Log streaming ingest pipeline by defining an ingest transform schema.

Creating a transform schema

Below is the suggested transform schema to use with the recommended log format described above. Be sure to replace <table uuid> with the UUID of your target Hydrolix table. You need to have this transform setup as the default, so "is_default" is set to true.

1{
2    "name": "fastly_transform",
3    "type": "json",
4    "table": "<table uuid>",
5    "description": "fastly https logs",
6    "settings": {
7        "is_default": true,
8        "output_columns": [
9                            {
10                                "name": "timestamp",
11                                "datatype": {
12                                    "type": "datetime",
13                                    "primary": true,
14                                    "format": "2006-01-02T15:04:05"
15                                }
16                            },
17                            {
18                                "name": "time_elapsed",
19                                "datatype": {
20                                    "type": "uint64"
21                                }
22                            },
23                            {
24                                "name": "is_tls",
25                                "datatype": {
26                                    "type": "bool"
27                                }
28                            },
29                            {
30                                "name": "client_ip",
31                                "datatype": {
32                                    "type": "string"
33                                }
34                            },
35                            {
36                                "name": "geo_city",
37                                "datatype": {
38                                    "type": "string"
39                                }
40                            },
41                            {
42                                "name": "geo_country_code",
43                                "datatype": {
44                                    "type": "string"
45                                }
46                            },
47                            {
48                                "name": "request",
49                                "datatype": {
50                                    "type": "string"
51                                }
52                            },
53                            {
54                                "name": "host",
55                                "datatype": {
56                                    "type": "string"
57                                }
58                            },
59                            {
60                                "name": "url",
61                                "datatype": {
62                                    "type": "string"
63                                }
64                            },
65                            {
66                                "name": "request_referer",
67                                "datatype": {
68                                    "type": "string"
69                                }
70                            },
71                            {
72                                "name": "request_user_agent",
73                                "datatype": {
74                                    "type": "string"
75                                }
76                            },
77                            {
78                                "name": "request_accept_language",
79                                "datatype": {
80                                    "type": "string"
81                                }
82                            },
83                            {
84                                "name": "request_accept_charset",
85                                "datatype": {
86                                    "type": "string"
87                                }
88                            },
89                            {
90                                "name": "cache_status",
91                                "datatype": {
92                                    "type": "string"
93                                }
94                            }
95                        ]
96        }
97    }

Once you define the transform schema, Hydrolix is configured to accept the incoming Fastly log data.

Leveraging views

Hydrolix supports having many different query formats for a single dataset. The query data structure, or view, for a given dataset allows you to customized the queried data and restrict a user’s access to a specific set of columns.

Hydrolix automatically generates a default view upon transform creation that can be used to immediately query the dataset - no additional configuration is required. However, you are encouraged to familiarize yourself with the view concept and the benefits that the feature can provide. More detailed information can be found on the Hydrolix site.

Further reading

Hydrolix provides a tutorial for querying and analyzing Fastly data from within their system.

Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Fastly
© Fastly 2024