Setting up remote log streaming

      Last updated March 17, 2021

    Fastly's Real-Time Log Streaming feature allows you to automatically save logs to a third-party service for storage and analysis. Logs provide an important resource for troubleshooting connectivity problems, pinpointing configuration areas that could use performance tuning, and identifying the causes of service disruptions. We recommend setting up remote log streaming when you start using Fastly services.

    Configuring logging endpoints

    You can configure one or more logging endpoints for Fastly services. Follow these instructions to access the logging settings:

    1. Log in to the Fastly web interface.
    2. From the All services page, select the appropriate service. You can use the search box to search by ID, name, or domain.
    3. Click the Edit configuration button and then select the option to clone the active version. The Domains page appears.
    4. Click the Logging link. The logging endpoints page appears. If you've already added a logging endpoint, click the Create Endpoint button. The list of available logging endpoints appears.

      the logging endpoints page

    5. Follow the instructions in one of our logging endpoint guides to complete the set up process and deploy your changes.

    Once you've clicked Activate to deploy your changes, events will begin being logged immediately. The logs may take a few moments to appear on your log server.

    How, when, and where logs are streamed

    To control log streaming, Fastly provides two versions of custom log formats, each of which uses Apache-style logging directives. The logging format strings in each of these versions are based on the Common Log Format (CLF).

    Logs are streamed over TCP, not UDP, optionally using TLS for security with supported endpoints. Additionally, if you are using custom VCL be sure to include the #FASTLY log macro in your vcl_log handler.

    By default, logs are placed in your root directory every hour using the file naming format YYYY-mm-ddThh:mm:ss-<uid>. You can change both the frequency and path of these files. Our guide on changing where log files are written provides more information.

    If you've configured multiple logging endpoints for your service, the logs will be sent to all of the logging endpoints.

    Fastly uses several different log-server aggregation points and each will send logs files, none of which contain duplicate entries. These log files are created as soon as streaming starts and they're written to over the entire time period you specify (or the default). Once that time has passed, the files aren't touched any more and the logging process creates a new batch of files.

    Escaping characters in logs

    Logs respond to VCL like any other object. For example, the following code can escape quotes from User-Agent your log stream:

    log {"syslog serviceid endpointname :: "} {"""} cstr_escape(req.http.user-agent);

    Preventing duplicate log entries when using custom VCL

    If you use custom VCL commands for logging, you may notice duplicate entries in your logs. This happens because logs are being generated by both Fastly and the custom VCL logging commands. You can eliminate the duplicate entries by adding a condition that prevents Fastly from generating log entries. Follow these instructions to add the condition:

    1. On the Logging endpoints page, click the Attach a condition link next to the appropriate logging service. The Add a condition window appears.
    2. Click Create a new response condition. The Create a new response condition window appears.
    3. Fill out the Create a new response condition window as follows:
      • In the Name field, type a human-readable name for the condition.
      • In the Apply if field, type false.
      • Leave the default value set in the Priority field.
    4. Click Save and apply to.
    5. Click the Activate button to deploy your configuration changes.

    Fastly will stop generating log entries, and your logs will only contain entries generated by the custom VCL logging commands.

    Troubleshooting common logging errors

    If an error in the Fastly web interface suggests that your logging configuration appears to be broken for the currently activated service version but you're still receiving some logs, not all of Fastly's log aggregators may be able to connect to your endpoint's server. It's likely the maximum number of concurrent connections has been reached. Try configuring your logging endpoint's server to allow a higher maximum number of inbound connections and then see if the error clears up after a couple of hours.

    Back to Top