Changing log compression formats

Fastly's Real-Time Log Streaming feature allows you to specify compression format and options for file-based logging endpoints. These include the Azure Blob, FTP, Google Cloud Storage, Kafka, OpenStack, Amazon S3, SFTP, Digital Ocean, and Cloud Files logging endpoints.

Available log compression formats

Although the default is to use no compression, we allow you to choose one of several compression mechanisms:

• Zstandard, a compression algorithm defined by RFC 8478
• Snappy, a compression and decompression library used by many Google products as referenced in the Snappy compressed format description
• Gzip, a compression utility as defined in RFC 1952 and RFC 1951

Using the web interface to change log compression formats

The web interface lets you select a compression codec to apply to log files.

Keep in mind that if you're using Gzip compression the web interface defaults to a Gzip compression level of 3, and can only be changed using the Logging API. If the Gzip compression level has been set to a value other than 3 via an API call, then that level is displayed as a read-only value.

Follow these instructions to update a file-based logging endpoint's compression format using the web interface:

1. Log in to the Fastly web interface.
2. From the All services page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click the Edit configuration button and then select the option to clone the active version. The Domains page appears.
4. Click the Logging link. The Logging endpoints page appears.
5. Click the name of a file-based logging endpoint you want to edit. The Edit this endpoint page appears.
6. Click the Advanced options link near the bottom of the page. The Advanced options appear.

7. In the Compression section, select a compression format for the logging endpoint.

   

8. Click the Update button.
9. Click the Activate button to deploy your configuration changes.

Using the API to change log compression formats

You can use the Logging API for any file-based logging endpoint to update the compression format.

The API provides two fields for specifying log file compression options: compression_codec and gzip_level.

• compression_codec is a string that specifies the codec used to compress your logs. One of the following:
  • zstd Use Zstandard compression.
  • snappy Use Snappy compression.
  • gzip Use Gzip. Defaults to gzip compression level 3.
• gzip_level is an integer in the range of 0, signifying no compression, to 9, signifying maximum compression.

Keep in mind the following with respect to the request payload:

• Only one of compression_codec or gzip_level should be included in a request payload. Specifying both will result in an error.
• If compression_codec is set to "gzip", gzip_level will default to 3.
• If compression_codec is not specified in the request payload and a non-zero value for gzip_level is set, then compression_codec will default to "gzip".
• If neither compression_codec nor gzip_level is specified in the request, no compression is applied to log files and compression_codec will default to null.

For example, to update the compression format for a service's SFTP logging endpoint to use the Snappy compression codec, the cURL command would look something like this:

1
curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN"  -H "Accept: application/json" -d "compression_codec=snappy" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/5/logging/sftp/sftp-log

To apply gzip compression to an Azure Blob Storage logging endpoint using the compression_codec field, the cURL command would look like this:

1
curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN"  -H "Accept: application/json" -d "compression_codec=gzip" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/7/logging/azureblob/azure-log

To update a Google Cloud Storage logging endpoint's compression format to 1 using the gzip_level field, the cURL command would look like this:

1
curl -X PUT -H "Fastly-Key: YOUR_FASTLY_TOKEN"  -H "Accept: application/json" -d "gzip_level=1" https://api.fastly.com/service/SU1Z0isxPaozGVKXdv0eY/version/4/logging/gcs/gcs-log