LOG IN SIGN UP
Documentation

Amazon S3

Amazon S3 can be used as an origin or as a private bucket.

Using Amazon S3 as an origin

To make your S3 data buckets available through Fastly, follow the steps below.

  1. Log in to the Fastly web interface and click the Configure link.
  2. Click the Options button to open the Manage service menu and select Create. The Create a new service page appears.

    the New Service window

  3. Fill out the Create a new service fields as follows:

    • In the Server address and port field, type s3.amazonaws.com. If you are using a non-standard S3 region (anything other than us-east), you must include that region in the server address field (e.g., s3.us-west-2.amazonaws.com). In the port field, you can type either 80 for HTTP or 443 for HTTPS.
    • In the Domain name field, type the hostname you want to use as the URL (e.g., cdn.example.com).
    • In the Description field, type any descriptive name for your service.
  4. Click the Create button. A new service appears in the list of services available.

Now that the service is created, you will need to set the default host to <yourbucket>.s3.amazonaws.com by following the steps below:

  1. From the service menu, select the new service you just created.
  2. Click the Edit configuration button and then select Clone active. The service version page appears.
  3. Click the Settings tab. The Settings page appears.

    the Settings page

  4. In the Override host area, click the Specify an override host button. The Add an override host header window appears.

  5. Type the hostname of your S3 bucket. For example, <yourbucket>.s3.amazonaws.com.

    Set Override host in Service Configuration

  6. Click the Save button.

  7. Click the Activate button to deploy your configuration changes.

Testing your results

By default, we create DNS mapping called yourdomain.global.prod.fastly.net. In the example above, it would be cdn.example.com.global.prod.fastly.net. Create a DNS alias for the domain name you specified (e.g., CNAME cdn.example.com to global-nossl.fastly.net).

Fastly will cache any content without an explicit Cache-Control header for 1 hour. You can verify whether you are sending any cache headers using cURL. For example:

$ curl -I opscode-full-stack.s3.amazonaws.com

HTTP/1.1 200 OK
x-amz-id-2: ZpzRp7IWc6MJ8NtDEFGH12QBdk2CM1+RzVOngQbhMp2f2ZyalkFsZd4qPaLMkSlh
x-amz-request-id: ABV5032583242618
Date: Fri, 18 Mar 2012 17:15:38 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

In this example, no cache control headers are set so default TTL will be applied.

Enhanced cache control

If you need more control over how different types of assets are cached (e.g., Javascript files, images) media check out our Amazon S3 configuration in our Cache Control tutorial.

Using an Amazon S3 private bucket

To use an Amazon S3 private bucket with Fastly, follow the instructions below.

Before you begin

Be sure you've already made your S3 data buckets available to Fastly by pointing to the right S3 bucket and setting your origin to port 443. This needs to be done before authenticating.

Be sure you've got the AWS access key ID, AWS secret key ID, and AWS Bucket name on hand. The Amazon S3 Authorization header takes the following form:

  Authorization: AWS `_AWSAccessKeyId_`:`_Signature_`

From your developer Amazon account you will need the following information:

  1. The AWS access key ID and AWS secret access key. The AWS secret access key is issued when you register. If you do not have or remember your AWS secret access key, create a new AWS access key ID. The AWS secret access key will be displayed before disappearing again.
  2. Your AWS Bucket name.

Setting up Fastly to use an Amazon S3 private bucket

In order to use an Amazon S3 private bucket with Fastly, create two headers, a Date header (for use with the authorization Signature) and an Authorization header.

Create a Date header

  1. Log in to the Fastly web interface and click the Configure link.
  2. From the service menu, select the appropriate service.
  3. Click the Edit configuration button and then select Clone active. The service version page appears.
  4. Click the Content tab. The Content page appears.
  5. Click the Create header button. The Create a new header page appears.

    creating a date header via the new header page

  6. Fill out the Create a new header fields as follows:

    • From the Type menu, select Request, and from the Action menu, select Set.
    • In the Destination field, type http.Date.
    • In the Source field, type now.
    • From the Ignore if set menu, select No.
    • In the Priority field, type 10.
    • In the Description field, type Date.
  7. Click the Create button. A new Date header appears on the Content page. You will use this later within the Signature of the Authorization header.

Create an Authorization header

Next create the Authorization header with the specifications listed below.

  1. Click the Create header button again to create another new header. The Create a new header page appears.

    creating an authorization header via the new header page

  2. Fill out the Create a new header fields as follows:

    • From the Type menu, select Request, and from the Action menu, select Set.
    • In the Destination field, type http.Authorization.
    • From the Ignore if set menu, select No.
    • In the Priority field, type 20.
    • In the Description field, type S3 Authorization.
  3. In the Source field, type the header authorization information using the following format:

    "AWS <AWS access key ID>:" digest.hmac_sha1_base64("<AWS secret key ID>", if(req.request == "HEAD", "GET", req.request) LF LF LF req.http.Date LF "/<AWS Bucket name>" req.url.path)
    

    replacing <AWS access key ID>, <AWS secret key ID>, and <AWS Bucket name> with the information you gathered before you began. For example:

    "AWS JKCAUEFV2ONFFOFMSSLA:" digest.hmac_sha1_base64("P2WPSu68Bfl89j72vT+bXYZB7SjlOwhT4whqt27", if(req.request == "HEAD", "GET", req.request) LF LF LF req.http.Date LF "/test123" req.url.path)
    
  4. Click the Create button. The new Authorization header appears on the Content page.

A detailed look at the Source field

So what's going on in the Source field of the Authorization header? Here's the basic format:

AWS<Access Key><Signature Function><key><message>

It tells us the following:

Element Description
AWS A constant placed before the access key. It's always AWS.
access key The access key ID from your Amazon developer's account. We used JKCAUEFV2ONFFOFMSSLA in this example.
signature function The algorithm used to validate the key and message of the signature. We used digest.hmac_sha1_base64(<key>, <message>) in this example.
key The secret key ID from your Amazon developer's account. We used P2WPSu68BfI89j72vT+bXYZB7SjIOwhT4whqt27 in this example.
message The UTF-8 encoding of the StringToSign. See the table below for a break down of each portion of the message.

The message that's part of the Source field in the Authorization header takes on this basic format:

<HTTP-verb></n><Content-MD5>/n<Content-Type></n><Date></n><CanonicalizedAmzHeader></n><CanonicalizedResource>

It tells us the following:

Element Description
HTTP-verb The REST verb. We use req.request in this example. We rewrite HEAD to GET because Varnish does this internally before sending requests to origin.
/n A newline indicator constant. It's always /n.
Content-MD5 The content-md5 header value, used as a message integrity check. It's often left blank. We use LF (line feed) in this example.
Content-Type The content-type header value, used to specify the MIME-type. It's often left blank. We use LFin this example.
Date The date and time stamp. We use req.http.Date (which we created first as a separate header in the steps above).
CanonicalizedAmzHeader The x-amz headers, which customize your S3 implementation. It's often left blank. We use LF in this example.
CanonicalizedResource Your Amazon private bucket name. We use "/test123" in this example.

Following redirects to S3 objects and caching S3 responses

With custom VCL, Fastly can follow redirects to S3 objects and cache the s3 response as well as the 301 or 302 response separately.

Once the ability to upload custom VCL has been enabled, be sure to read our "How do I mix and match Fastly VCL with custom VCL?" instructions. It's important to include the entire VCL boilerplate if you do not intend to override the Fastly default settings.

To configure Fastly to follow redirects to S3 objects, insert the following VCL snippets in your custom VCL:

Within vcl_recv

sub vcl_recv {

  if (req.http.redir != "true") {
    set req.backend = Main_Origin;
  } else {
    set req.backend = s3_backend;
    set req.http.host = "s3.amazonaws.com";
  }

#FASTLY recv

  if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
    return(pass);
  }

  return(lookup);

}

Within vcl_deliver

sub vcl_deliver {

  if (resp.status == 302 || resp.status == 301) {
    set req.http.redir = "true";
    set req.url = regsub(resp.http.Location, "http://s3.amazonaws.com/(.*)$", "/\1");
    set req.http.Fastly-Force-Shield = "yes";
    restart;
  }

#FASTLY deliver

  return(deliver);
}

Be sure to set the Main_Origin and s3_backend to the actual name of your backends in the service to which you're applying these redirects. You can find the exact names by reviewing your VCL; simply click on the VCL button at the top of the page while viewing the service.

Once you added these VCL snippets to your custom VCL, upload the VCL file and then activate the new version of your service to apply the changes.


Back to Top