Working with surrogate keys

Surrogate keys are unique identifiers that you assign to groups of content for processing. While there are many use cases for surrogate keys, one of the primary way for using them with Fastly is to make purging more efficient.

Surrogate keys allow you to selectively purge related content. Using the Surrogate-Key header, you can "tag" content with a key term, a string of any characters you want. When you want to purge content associated with that key, you then issue a key purge request and all of the objects associated with that key will be purged. This process makes it easier to cache and purge content that changes rapidly and unpredictably without having to purge your entire cache.

Before you begin

This guide assumes you're already familiar with the way content delivery networks (CDNs) work in general, and the way Fastly's CDN works in particular. We recommend reviewing our guide on adding headers on HTTP requests and responses.

Understanding the Surrogate-Key header

To understand how surrogate keys work, especially in the context of content purging, you need to understand what a Surrogate-Key header is and how it behaves.

About the Surrogate-Key header

Any time your origin responds to a request for content, the response it sends will include bits of code called headers in front of the actual body of the response. Those headers are used to give additional detail and provide context for the body of the response itself. HTTP headers sent as part of a response typically look something like this:

HTTP/1.1 200 OK
Content-Type: text/html
Connection: keep-alive
...

You can control that additional information and therefore control how content is served by adding to or modifying the headers your server responds with. The Surrogate-Key header is one of them. It's what categorizes or "tags" the content as part of a specific group. The tag you choose is called a key and it describes the common trait each element of a content group shares.

For example, suppose you have a website where you publish taco recipes. You know that entire categories of recipes will need to be purged, so you thoughtfully include these recipe categories as keys in the Surrogate-Key header. The response your server sends might look more like this:

HTTP/1.1 200 OK
Surrogate-Key: veggie seasonal central-mexico
Content-Type: text/html
...

The addition of the Surrogate-Key header there tells you that the content is tagged with specific identifiers that categorize it for future use. This response contains three surrogate keys: veggie, seasonal, and central-mexico. When Fastly receives a response like this, we use the surrogate keys to create a mapping from each key to the cached content, then we strip out the Surrogate-Key header so it's not included in the response to your readers.

Creating relationships between keys and objects

One of the major advantages of surrogate keys is that they allow for a many-to-many relationship between keys and objects. An origin server's response can associate multiple keys with the object and the same key can be provided in different responses. Take a look at these two requests and responses:

GET /blog/healthy-taco-recipes HTTP/1.1
Host: www.tacolabs.com

HTTP/1.1 200 OK Content-Type: text/html
Content-Length: 1234
Surrogate-Key: mainpage low-carb

GET /recipes/low-carb-cheese-taco-shell HTTP/1.1
Host: www.tacolabs.com

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 2345
Surrogate-Key: low-carb cheese

In this example, there are two objects (/blog/healthy-taco-recipes and /recipes/low-carb-cheese-taco-shell) with three keys (mainpage, low-carb, and cheese). Two of the keys (mainpage and cheese) are associated with a single object and a third key (low-carb) is associated with both objects.

By using the Surrogate-Key header to associate keys with one or more objects, you can precisely control which objects are removed from cache during a purge. Consider the example presented above. Purging the mainpage key would remove only the /blog/healthy-taco-recipes object from the cache. On the other hand, purging the low-carb key would remove both the /blog/healthy-taco-recipes and /recipes/low-carb-cheese-taco-shell objects from the cache.

Wherever there's a relation between two different pieces of content, there might be a good reason to keep them categorized by using a surrogate key.

Generating and setting surrogate keys

There are two ways to set the Surrogate-Key header: by adding the header in the Fastly control panel, or by generating the keys with your own application. You can make your surrogate key associations on your own application server and include them in the HTTP response that you send to Fastly. This is the more useful method because you can assign exactly the keys that you want on every response.

However, if you want to set the Surrogate-Key header in the Fastly control panel you can do so following the steps below.

1. Log in to the Fastly control panel.
2. From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain.
3. Click Edit configuration and then select the option to clone the active version.
4. Click Content.

5. Click Create header.

   

6. Fill out the Create a header fields as follows:

   • In the Name field, enter a human-readable name for the header. This name is displayed in the Fastly web interface.
   • From the Type menu, select Cache, and from the Action menu, select Set.
   • In the Destination field, enter http.Surrogate-Key.
   • In the Source field, enter where the content for the header comes from. For example, if you're using Amazon S3 as an origin server, you could set the field to beresp.http.x-amz-meta-surrogate-key to specify x-amz-meta-surrogate-key as the response header received from the backend.
   • From the Ignore if set menu, select No.
   • In the Priority field, enter 10.

7. Click the Create button to create your header.

8. Click Activate to deploy your configuration changes.

Troubleshooting

You can check the surrogate keys for a URL by using the Fastly-Debug: 1 header. See the instructions on using a Fastly-Debug header with curl for more information.

Limitations

The surrogate keys sent by your origin server can be as simple or complex as you need, subject to format limitations. Surrogate keys must be formatted as a single string without spaces. Spaces separate keys.

Surrogate keys are subject to size limitations. Individual surrogate keys may not exceed 1,024 bytes in length and Surrogate-Key header values (comprising one or more space-separated keys) may not exceed 16,384 bytes in length. If either of the key or key header value limits are reached while parsing a Surrogate-Key header, the key currently being parsed and all keys following it within the same header will be ignored.

What's next

Now that you understand how the Surrogate-Key header works and how to set it, learn how to purge objects with surrogate keys.