Log streaming: Tardigrade
Last updated January 14, 2021
Fastly's Real-Time Log Streaming feature can send log files to Tardigrade's S3 Compatible Distributed Cloud Service. Built on the Storj Network, Tardigrade is a decentralized object storage service that is S3 compatible and end-to-end encrypted by default.
NOTE: Fastly does not provide direct support for third-party services. See Fastly's Terms of Service for more information.
Before adding Tardigrade as a logging endpoint for Fastly services, you will need to:
- Create a Tardigrade account.
- Create an API key for the project.
- Set up a Tardigrade Uplink CLI and create an access grant for the project.
- Create a Tardigrade bucket where Fastly will store the log output.
Setting up a Tardigrade S3 Gateway
After you've registered for a Tardigrade account and generated an API key, you'll need to set up an S3 Compatible Gateway endpoint using the platform of your choice.
To support virtual host-style, add
export MINIO_DOMAIN=example.com to the startup for the Tardigrade gateway.
IMPORTANT: You must enable TLS on your Tardigrade S3 Gateway.
Adding Tardigrade as a logging endpoint
Follow these instructions to add Tardigrade as a logging endpoint:
- Review the information in our Setting Up Remote Log Streaming guide.
Click the Amazon Web Services S3 logo. The Create an Amazon S3 endpoint page appears.
- Fill out the Create an Amazon S3 endpoint fields as follows:
- In the Name field, enter a human-readable name for the endpoint.
- In the Log format field, optionally enter an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. Our discussion of format strings provides more information.
- In the Timestamp format field, optionally enter a timestamp format for log files. The default is an
strftimecompatible string. Our guide on changing where log files are written provides more information.
- In the Bucket name field, enter the name of the Tardigrade bucket in which to store the logs.
- In the Access key field, enter the access key associated with the Tardigrade bucket. This should be outputted by the Gateway Process. See Tardigrades' S3 Gateway documentation for more information.
- In the Secret key field, enter the secret key associated with the Tardigrade bucket. See Tardigrades' S3 Gateway documentation for more information.
- In the Period field, optionally enter an interval (in seconds) to control how frequently your log files are rotated. This value defaults to
Click the Advanced options link of the Create a new S3 endpoint page.
- Fill out the rest of the Advanced options of the Create an Amazon S3 endpoint page as follows:
- In the Path field, optionally enter the path within the bucket to store the files. The path ends with a trailing slash. If this field is left empty, the files will be saved in the bucket's root path. Our guide on changing where log files are written provides more information.
- In the Domain field, enter the fully qualified hostname of your Tardigrade S3 Gateway.
- In the PGP public key field, optionally enter a PGP public key that Fastly will use to encrypt your log files before writing them to disk. You will only be able to read the contents by decrypting them with your private key. The PGP key should be in PEM (Privacy-Enhanced Mail) format. See our guide on log encryption for more information.
- In the Select a log line format area, select the log line format for your log messages. Our guide on changing log line formats provides more information.
- In the Compression field, optionally select the compression format you want applied to the log files. Our guide on changing log compression options provides more information.
- From the Redundancy level menu, select a setting. This value defaults to Standard.
- In the Server side encryption area, optionally select an encryption method to protect files that Fastly writes to your Tardigrade bucket. Valid values are None and AES-256.
- Click the Create button to create the new logging endpoint.
- Click the Activate button to deploy your configuration changes.