Log streaming: Amazon S3
Fastly's Real-Time Log Streaming feature can send log files to Amazon Simple Storage Service (Amazon S3). Amazon S3 is a static file storage service used by developers and IT teams. You can also use the instructions in this guide to configure log streaming to another S3-compatible service.
Before adding Amazon S3 as a logging endpoint in the Fastly application, we recommend creating an Identity and Access Management (IAM) user in Amazon S3 specifically for Fastly. Grant the user
PutObject permissions for the directory in which you want to store logs. For more information, see Amazon's Getting Your Access Key ID and Secret Access Key page.
Adding Amazon S3 as a logging endpoint
After you've registered for an Amazon S3 account and created an IAM user in Amazon S3, follow these instructions to add Amazon S3 as a logging endpoint:
- Review the information in our Setting Up Remote Log Streaming guide.
On the Logging page, find the Amazon S3 endpoints area and click New.
The New S3 endpoint window appears.
Fill out the New S3 endpoint fields as follows:
- In the Name field, type a human-readable name for the endpoint.
- In the Bucket Name field, type the name of the Amazon S3 bucket in which to store the logs.
- In the Access Key field, type the access key associated with the Amazon S3 bucket. See Amazon's S3 Getting Started Guide for more information.
- In the Secret Key field, type the secret key associated with the Amazon S3 bucket. See Amazon's S3 Getting Started Guide for more information.
- In the Path field, optionally type the path within the bucket to store the files. The path ends with a trailing slash. If this field is left empty, the files will be saved in the bucket's root path.
- In the Domain field, optionally type the domain of the Amazon S3 endpoint. If your Amazon S3 bucket was not created in the US Standard region, you must set the domain to match the appropriate endpoint URL. Use the table in the S3 section of the Regions and Endpoints Amazon S3 documentation page. If you want to use an S3-compatible storage system (such as Dreamhost's DreamObjects), set the domain to match the domain name for that service (for example, in the case of DreamObjects, the domain name would be
- In the Period field, optionally type an interval (in seconds) to control how frequently your log files are rotated. This value defaults to
- In the Gzip Level field, optionally type the level of gzip compression you want applied to the log files. You can specify any whole number from
1(fastest and least compressed) to
9(slowest and most compressed). This value defaults to
- From the Redundancy Level menu, select a setting. This value defaults to Standard. See Amazon's Using Reduced Redundancy Storage Guide for more information on using reduced redundancy storage.
- In the Format String field, optionally type an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. See our guidance on format strings for more information.
Click Create to create the new logging endpoint.
Keep in mind that although Fastly continuously streams logs into Amazon S3, the Amazon S3 website and API do not make files available for access until after the upload is complete.