Log streaming: New Relic Logs

Fastly's Real-Time Log Streaming feature can send log files to New Relic Logs.

Prerequisites

Before adding New Relic Logs as a logging endpoint for Fastly services, you will need to:

• Register for a New Relic account.
• Obtain your license key or optionally create an Insert API key.

Adding New Relic Logs as a logging endpoint

Follow these instructions to add New Relic Logs as a logging endpoint:

1. Review the information in our Setting Up Remote Log Streaming guide.
2. Click the New Relic Logs Create endpoint button. The Create a New Relic Logs endpoint page appears.
3. Fill out the Create a New Relic Logs endpoint fields as follows:
   • In the Name field, enter a human-readable name for the endpoint.
   • In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, waf_debug (waf_debug_log), and None. See our guide on changing log placement for more information.
   • In the Log format field, optionally enter an Apache-style string or VCL variables to use for log formatting. See the example format section for details.
   • In the License key / Insert key field, enter your New Relic license key or Insert API key.
   • From the Region controls, select the region to stream logs to.
4. Click the Create button to create the new logging endpoint.
5. Click the Activate button to deploy your configuration changes.

Example format

Data sent to New Relic Logs must be serialized as a JSON object. Here's an example format string for sending data to New Relic Logs:

1{
2  "timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
3  "client_ip": "%{req.http.Fastly-Client-IP}V",
4  "geo_country": "%{client.geo.country_name}V",
5  "geo_city": "%{client.geo.city}V",
6  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
7  "url": "%{json.escape(req.url)}V",
8  "request_method": "%{json.escape(req.method)}V",
9  "request_protocol": "%{json.escape(req.proto)}V",
10  "request_referer": "%{json.escape(req.http.referer)}V",
11  "request_user_agent": "%{json.escape(req.http.User-Agent)}V",
12  "response_state": "%{json.escape(fastly_info.state)}V",
13  "response_status": %{resp.status}V,
14  "response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
15  "response_body_size": %{resp.body_bytes_written}V,
16  "fastly_server": "%{json.escape(server.identity)}V",
17  "fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
18}

Logging a timestamp

If a timestamp field is present in the Fastly log, it must be specified as millisecond or seconds since Epoch to override the New Relic timestamp. If not included, Fastly will generate a timestamp.

Using New Relic Instant Observability's prebuilt Fastly dashboard

New Relic I/O is an open source ecosystem of community-contributed quickstarts for hundreds of tools and technologies. We've worked with New Relic to develop a prebuilt dashboard that highlights certain key metrics we think are important and useful. Setting up the dashboard is easy and the code is open source in case you want to customize it.

Configure your service to use the recommended expanded logging format

Since the prebuilt dashboard expects certain fields to be present, we recommend using the following logging format. That said, there is nothing to stop you from adding fields for your own purposes or to maintain backward compatibility with existing dashboards you've built. The dashboard won't break if you don't send some fields, but certain charts won't have data.

1{
2  "timestamp": %{time.start.msec}V,
3  "logtype": "accesslogs",
4  "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V",
5  "client_ip":"%h",
6  "client_device_type":"%{client.platform.hwtype}V",
7  "client_os_name":"%{client.os.name}V",
8  "client_os_version":"%{client.os.version}V",
9  "client_browser_name":"%{client.browser.name}V",
10  "client_browser_version":"%{client.browser.version}V",
11  "client_as_name":"%{client.as.name}V",
12  "client_as_number":"%{client.as.number}V",
13  "client_connection_speed": "%{client.geo.conn_speed}V",
14  "client_port": %{client.port}V,
15  "client_rate_bps":%{client.socket.tcpi_delivery_rate}V,
16  "client_recv_bytes":%{client.socket.tcpi_bytes_received}V,
17  "client_requests_count":%{client.requests}V,
18  "client_resp_body_size_write": %{resp.body_bytes_written}V,
19  "client_resp_header_size_write": %{resp.header_bytes_written}V,
20  "client_resp_ttfb": %{time.to_first_byte}V,
21  "client_rtt_us":%{client.socket.tcpi_rtt}V,
22  "content_type":"%{Content-Type}o",
23  "domain": "%{Fastly-Orig-Host}i",
24  "fastly_datacenter": "%{server.datacenter}V",
25  "fastly_host": "%{server.hostname}V",
26  "fastly_region": "%{server.region}V",
27  "host": "%v",
28  "origin_host":"%v",
29  "origin_name":"%{req.backend.name}V",
30  "request":"%{req.request}V",
31  "request_method":"%m",
32  "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
33  "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
34  "request_referer":"%{json.escape(req.http.Referer)}V",
35  "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
36  "resp_status":"%s",
37  "response": "%{resp.response}V",
38  "service_id":"%{req.service_id}V",
39  "service_version": "%{req.vcl.version}V",
40  "status":"%s",
41  "time_start":"%{begin:%Y-%m-%dT%H:%M:%SZ}t",
42  "time_end":"%{end:%Y-%m-%dT%H:%M:%SZ}t",
43  "time_elapsed":%D,
44  "tls_cipher":"%{json.escape(tls.client.cipher)}V",
45  "tls_version":"%{json.escape(tls.client.protocol)}V",
46  "url":"%{json.escape(req.url)}V",
47  "user_agent":"%{json.escape(req.http.User-Agent)}V",
48  "user_city":"%{client.geo.city.utf8}V",
49  "user_country_code":"%{client.geo.country_code}V",
50  "user_continent_code":"%{client.geo.continent_code}V",
51  "user_region":"%{client.geo.region}V"
52}

Install the Fastly dashboard

Follow these instructions to install the Fastly dashboard quickstart:

1. Select the Fastly dashboard quickstart from the New Relic marketplace.

   

   Or navigate directly to the Fastly dashboard page.

2. Click the Install quickstart button.

   

3. Click the Begin installation button.

   

4. Click the Done button or the Skip this step link.

   

5. Click the See your data button.

   

The Fastly dashboard will appear.