Changing log compression formats

      Last updated February 03, 2021

    Fastly's Real-Time Log Streaming feature allows you to specify compression format and options for file-based logging endpoints. These include the Azure Blob, FTP, Google Cloud Storage, Kafka, OpenStack, Amazon S3, SFTP, Digital Ocean, and Cloud Files logging endpoints.

    Available log compression formats

    Although the default is to use no compression, we allow you to choose one of several compression mechanisms:

    Using the web interface to change log compression formats

    The web interface lets you select a compression codec to apply to log files.

    Keep in mind that if you're using Gzip compression the web interface defaults to a Gzip compression level of 3, and can only be changed using the Logging API. If the Gzip compression level has been set to a value other than 3 via an API call, then that level is displayed as a read-only value.

    Follow these instructions to update a file-based logging endpoint's compression format using the web interface:

    1. Log in to the Fastly web interface.
    2. From the All services page, select the appropriate service. You can use the search box to search by ID, name, or domain.
    3. Click the Edit configuration button and then select the option to clone the active version. The Domains page appears.
    4. Click the Logging link. The Logging endpoints page appears.
    5. Click the name of a file-based logging endpoint you want to edit. The Edit this endpoint page appears.
    6. Click the Advanced options link near the bottom of the page. The Advanced options appear.
    7. In the Compression section, select a compression format for the logging endpoint.

      Logging compression options

    8. Click the Update button.
    9. Click the Activate button to deploy your configuration changes.

    Using the API to change log compression formats

    You can use the Logging API for any file-based logging endpoint to update the compression format.

    The API provides two fields for specifying log file compression options: compression_codec and gzip_level.

    Keep in mind the following with respect to the request payload:

    For example, to update the compression format for a service's SFTP logging endpoint to use the Snappy compression codec, the cURL command would look something like this:

    curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN"  -H "Accept: application/json" -d "compression_codec=snappy"

    To apply gzip compression to an Azure Blob Storage logging endpoint using the compression_codec field, the cURL command would look like this:

    curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN"  -H "Accept: application/json" -d "compression_codec=gzip"

    To update a Google Cloud Storage logging endpoint's compression format to 1 using the gzip_level field, the cURL command would look like this:

    curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN"  -H "Accept: application/json" -d "gzip_level=1"
    Back to Top