Changing log compression formats
Last updated 2021-02-03
Fastly's Real-Time Log Streaming feature allows you to specify compression format and options for file-based logging endpoints. These include the Azure Blob, FTP, Google Cloud Storage, Kafka, OpenStack, Amazon S3, SFTP, Digital Ocean, and Cloud Files logging endpoints.
Available log compression formats
Although the default is to use no compression, we allow you to choose one of several compression mechanisms:
- Zstandard, a compression algorithm defined by RFC 8478
- Snappy, a compression and decompression library used by many Google products as referenced in the Snappy compressed format description
- Gzip, a compression utility as defined in RFC 1952 and RFC 1951
Using the web interface to change log compression formats
The web interface lets you select a compression codec to apply to log files.
Keep in mind that if you're using Gzip compression the web interface defaults to a Gzip compression level of 3, and can only be changed using the Logging API. If the Gzip compression level has been set to a value other than 3
via an API call, then that level is displayed as a read-only value.
Follow these instructions to update a file-based logging endpoint's compression format using the web interface:
- Log in to the Fastly web interface.
- From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
- Click the Edit configuration button and then select the option to clone the active version. The Domains page appears.
Click the Logging link. The Logging endpoints page appears.
Click the name of a file-based logging endpoint you want to edit. The Edit this endpoint page appears.
Click the Advanced options link near the bottom of the page. The Advanced options appear.
In the Compression section, select a compression format for the logging endpoint.
Click the Update button.
- Click the Activate button to deploy your configuration changes.
Using the API to change log compression formats
You can use the Logging API for any file-based logging endpoint to update the compression format.
The API provides two fields for specifying log file compression options: compression_codec
and gzip_level
.
compression_codec
is a string that specifies the codec used to compress your logs. One of the following:gzip_level
is an integer in the range of 0, signifying no compression, to 9, signifying maximum compression.
NOTE
Use of compression_codec
is recommended over use of gzip_level
unless you need to set the gzip compression level to a value other than 3.
Keep in mind the following with respect to the request payload:
- Only one of
compression_codec
orgzip_level
should be included in a request payload. Specifying both will result in an error. - If
compression_codec
is set to "gzip",gzip_level
will default to 3. - If
compression_codec
is not specified in the request payload and a non-zero value forgzip_level
is set, thencompression_codec
will default to "gzip". - If neither
compression_codec
norgzip_level
is specified in the request, no compression is applied to log files andcompression_codec
will default tonull
.
For example, to update the compression format for a service's SFTP logging endpoint to use the Snappy compression codec, the curl command would look something like this:
$ curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" -H "Accept: application/json" -d "compression_codec=snappy" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/5/logging/sftp/sftp-log
To apply gzip compression to an Azure Blob Storage logging endpoint using the compression_codec
field, the curl command would look like this:
$ curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" -H "Accept: application/json" -d "compression_codec=gzip" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/7/logging/azureblob/azure-log
To update a Google Cloud Storage logging endpoint's compression format to 1 using the gzip_level
field, the curl command would look like this:
$ curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN" -H "Accept: application/json" -d "gzip_level=1" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/4/logging/gcs/gcs-log
Do not use this form to send sensitive information. If you need assistance, contact support@fastly.com.