Working with surrogate keys
Last updated 2023-01-08
Surrogate keys are unique identifiers that you assign to groups of content for processing. While there are many use cases for surrogate keys, one of the primary way for using them with Fastly is to make purging more efficient.
Surrogate keys allow you to selectively purge related content. Using the
Surrogate-Key header, you can "tag" content with a key term, a string of any characters you want. When you want to purge content associated with that key, you then issue a key purge request and all of the objects associated with that key will be purged. This process makes it easier to cache and purge content that changes rapidly and unpredictably without having to purge your entire cache.
This guide assumes you're already familiar with the way content delivery networks (CDNs) work in general, and the way Fastly's CDN works in particular. We also recommend reviewing our guide on adding headers on HTTP requests and responses.
To understand how surrogate keys work, especially in the context of content purging, you need to understand what a Surrogate-Key header is and how it behaves.
Any time your origin responds to a request for content, the response it sends will include bits of code called headers in front of the actual body of the response. Those headers are used to give additional detail and provide context for the body of the response itself. HTTP headers sent as part of a response typically look something like this:
HTTP/1.1 200 OKContent-Type: text/htmlConnection: keep-alive...
You can control that additional information and therefore control how content is served by adding to or modifying the headers your server responds with. The
Surrogate-Key header is one of them. It's what categorizes or "tags" the content as part of a specific group. The tag you choose is called a key and it describes the common trait each element of a content group shares.
For example, suppose you have a website where you publish taco recipes. You know that entire categories of recipes will need to be purged, so you thoughtfully include these recipe categories as keys in the
Surrogate-Key header. The response your server sends might look more like this:
HTTP/1.1 200 OKSurrogate-Key: veggie seasonal central-mexicoContent-Type: text/html...
The addition of the Surrogate-Key header there tells you that the content is tagged with specific identifiers that categorize it for future use. This response contains three surrogate keys:
central-mexico. When Fastly receives a response like this, we use the surrogate keys to create a mapping from each key to the cached content, then we strip out the
Surrogate-Key header so it's not included in the response to your readers.
You can't use duplicate surrogate keys. For example, if you tried to use
foo, the two
foos would be collapsed into a single instance of the term.
One of the major advantages of surrogate keys is that they allow for a many-to-many relationship between keys and objects. An origin server's response can associate multiple keys with the object and the same key can be provided in different responses. Take a look at these two requests and responses:
GET /blog/healthy-taco-recipes HTTP/1.1Host: www.tacolabs.com
HTTP/1.1 200 OK Content-Type: text/htmlContent-Length: 1234Surrogate-Key: mainpage low-carb
GET /recipes/low-carb-cheese-taco-shell HTTP/1.1Host: www.tacolabs.com
HTTP/1.1 200 OKContent-Type: text/htmlContent-Length: 2345Surrogate-Key: low-carb cheese
In this example, there are two objects (
/recipes/low-carb-cheese-taco-shell) with three keys (
cheese). Two of the keys (
cheese) are associated with a single object and a third key (
low-carb) is associated with both objects.
By using the
Surrogate-Key header to associate keys with one or more objects, you can precisely control which objects are removed from cache during a purge. Consider the example presented above. Purging the
mainpage key would remove only the
/blog/healthy-taco-recipes object from the cache. On the other hand, purging the
low-carb key would remove both the
/recipes/low-carb-cheese-taco-shell objects from the cache.
Wherever there's a relation between two different pieces of content, there might be a good reason to keep them categorized by using a surrogate key.
There are two ways to set the
Surrogate-Key header: by adding the header in the Fastly web interface, or by generating the keys with your own application. You can make your surrogate key associations on your own application server and include them in the HTTP response that you send to Fastly. This is the more useful method because you can assign exactly the keys that you want on every response.
However, if you want to set the
Surrogate-Key header in the Fastly web interface you can do so following the steps below.
- Log in to the Fastly web interface.
- From the Home page, select the appropriate service. You can use the search box to search by ID, name, or domain. You can also click Compute services or CDN services to access a list of services by type.
- Click Edit configuration and then select the option to clone the active version.
- Click Content.
Click Create header.
Fill out the Create a header fields as follows:
- In the Name field, enter a human-readable name for the header. This name is displayed in the Fastly web interface.
- From the Type menu, select Cache, and from the Action menu, select Set.
- In the Destination field, enter
- In the Source field, enter where the content for the header comes from. For example, if you're using Amazon S3 as an origin server, you could set the field to
x-amz-meta-surrogate-keyas the response header received from the backend.
- From the Ignore if set menu, select No.
- In the Priority field, enter
Click the Create button to create your header.
- Click Activate to deploy your configuration changes.
You can check the surrogate keys for a URL by using the
Fastly-Debug: 1 header. See the instructions on using a Fastly-Debug header with curl for more information.
The surrogate keys sent by your origin server can be as simple or complex as you need, subject to format limitations. Surrogate keys must be formatted as a single string without spaces. Spaces separate keys.
Surrogate keys are subject to size limitations. Individual surrogate keys may not exceed 1,024 bytes in length and
Surrogate-Key header values (comprising one or more space-separated keys) may not exceed 16,384 bytes in length. If either of the key or key header value limits are reached while parsing a
Surrogate-Key header, the key currently being parsed and all keys following it within the same header will be ignored.
Now that you understand how the
Surrogate-Key header works and how to set it, learn how to purge objects with surrogate keys.