Changing log compression formats
Last updated 2021-02-03
Fastly's Real-Time Log Streaming feature allows you to specify compression format and options for file-based logging endpoints. These include the Azure Blob, FTP, Google Cloud Storage, Kafka, OpenStack, Amazon S3, SFTP, Digital Ocean, and Cloud Files logging endpoints.
Available log compression formats
Although the default is to use no compression, we allow you to choose one of several compression mechanisms:
- Zstandard, a compression algorithm defined by RFC 8478
- Snappy, a compression and decompression library used by many Google products as referenced in the Snappy compressed format description
- Gzip, a compression utility as defined in RFC 1952 and RFC 1951
Using the web interface to change log compression formats
The web interface lets you select a compression codec to apply to log files.
Keep in mind that if you're using Gzip compression the web interface defaults to a Gzip compression level of 3, and can only be changed using the Logging API. If the Gzip compression level has been set to a value other than
3 via an API call, then that level is displayed as a read-only value.
Follow these instructions to update a file-based logging endpoint's compression format using the web interface:
- Log in to the Fastly web interface.
- From the All services page, select the appropriate service. You can use the search box to search by ID, name, or domain.
- Click the Edit configuration button and then select the option to clone the active version. The Domains page appears.
- Click the Logging link. The Logging endpoints page appears.
- Click the name of a file-based logging endpoint you want to edit. The Edit this endpoint page appears.
- Click the Advanced options link near the bottom of the page. The Advanced options appear.
In the Compression section, select a compression format for the logging endpoint.
- Click the Update button.
- Click the Activate button to deploy your configuration changes.
Using the API to change log compression formats
You can use the Logging API for any file-based logging endpoint to update the compression format.
The API provides two fields for specifying log file compression options:
compression_codecis a string that specifies the codec used to compress your logs. One of the following:
gzip_levelis an integer in the range of 0, signifying no compression, to 9, signifying maximum compression.
compression_codec is recommended over use of
gzip_level unless you need to set the gzip compression level to a value other than 3.
Keep in mind the following with respect to the request payload:
- Only one of
gzip_levelshould be included in a request payload. Specifying both will result in an error.
compression_codecis set to "gzip",
gzip_levelwill default to 3.
compression_codecis not specified in the request payload and a non-zero value for
gzip_levelis set, then
compression_codecwill default to "gzip".
- If neither
gzip_levelis specified in the request, no compression is applied to log files and
compression_codecwill default to
For example, to update the compression format for a service's SFTP logging endpoint to use the Snappy compression codec, the cURL command would look something like this:
1 curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" -H "Accept: application/json" -d "compression_codec=snappy" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/5/logging/sftp/sftp-log
To apply gzip compression to an Azure Blob Storage logging endpoint using the
compression_codec field, the cURL command would look like this:
1 curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" -H "Accept: application/json" -d "compression_codec=gzip" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/7/logging/azureblob/azure-log
To update a Google Cloud Storage logging endpoint's compression format to 1 using the
gzip_level field, the cURL command would look like this:
1 curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" -H "Accept: application/json" -d "gzip_level=1" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/4/logging/gcs/gcs-log