Purging API cache with surrogate keys

Fastly makes it possible for you to cache your API so you can accelerate the performance of your service-oriented architecture. Of course, caching your API is one thing - efficiently invalidating the API cache is another matter entirely. If you've already enabled API caching and implemented API cache control, you've probably run into this problem, which was aptly described by Phil Karlton:

There are only two hard things in computer science: cache invalidation and naming things.

This guide explains how to use the Fastly API to purge your API cache with surrogate keys. Surrogate keys allow you to reduce the complexity of caching an API by combining multiple cache purges into a single key-based purge.

What's a surrogate key?

Surrogate keys allow you to selectively purge related content. Using the Surrogate-Key header, you can "tag" an object, such as an image or a blog post, with one or more keys. When Fastly fetches an object from your origin server, we check to see if you've specified a Surrogate-Key header. If you have, we add the response to a list we've set aside for each of the keys.

When you want to purge all of the responses associated with a key, issue a key purge request and all of the objects associated with that key will be purged. This makes it possible to combine many purges into a single request. Ultimately, it makes it easier to manage categorically related data.

To learn more about surrogate keys and to see how you can integrate them into your application, see our guide on getting started with surrogate keys.

Example: Purging categories

To see how surrogate keys work in conjunction with an API endpoint, imagine you have an online store and an API endpoint that returns the details of a product. When a user wants to get information about a specific product, like a keyboard, the request might look like this:

GET /product/12345

If your API is using Fastly and the response is not already cached, Fastly will make a request to your API's origin server and receive a response like this:

HTTP/1.1 200 OK
Content-Type: text/json
Cache-Control: private
Surrogate-Control: max-age=86400
Surrogate-Key: peripherals keyboards

{id: 12345, name: "Uber Keyboard", price: "$124.99"}

You knew that entire product categories would occasionally need to be purged, so you thoughtfully included the peripherals and keyboards product categories as keys in the Surrogate-Key header. When Fastly receives a response like this, we add it to an internal map, strip out the Surrogate-Key header, cache the response, and then deliver it to the end user.

Now imagine that your company decides to apply a 10% discount to all peripherals. You could issue the following key purge to invalidate all objects tagged with the peripherals surrogate key:

PURGE /service/:service_id/peripherals

When Fastly receives this request, we reference the list of content associated with the peripherals surrogate key and systematically purge every piece of content in the list.

Relational dependencies

Your API can use surrogate keys to group large numbers of items that may eventually need to be purged at the same time. Consider the example presented above. The API for your online store could have surrogate keys for product types, specific sales, or manufacturers.

From this perspective, the Surrogate-Key header provides Fastly with information about relations and possible dependencies between different API endpoints. Wherever there's a relation between two different types of resources in an API, there might be a good reason to keep them categorized by using a surrogate key.

Example: Purging product reviews and action shots

To learn how surrogate keys can help with relational dependencies, imagine that your online store wants to allow buyers to post product reviews and "action shots" depicting the products in use. To support these new features, you'll need to change your API. First, you'll need to create a new review endpoint:

GET /review/:id
POST /review

Next, you'll need to create a new action_shot endpoint:

POST /product/:id/action_shot
GET /product/:id/action_shot/:shot_id

Since both of the new endpoints refer to specific products, they'll need to be purged when relevant product information changes. Surrogate keys are a perfect fit for this use case. You can implement them by modifying the review and action_shot to return the following header:

Surrogate-Key: product/:id

This relates each of the endpoints to a specific product in the cache (where :id the product's unique identifier). When the product information changes, your API issues the following surrogate key purge:

POST https://api.fastly.com/service/:service_id/purge/product/:id

When Fastly receives this request, we purge each of the related endpoints at the same time.

Variations on a theme

You'll also want to consider using surrogate keys if your API has many different endpoints that all derive from a single source. Any time the source data changes, each of the endpoints associated with it will need to be purged from the cache. By associating each of the endpoints with a surrogate key, a single purge can be issued to purge them from the cache when the source changes.

Example: Purging product images

To understand how this works, imagine that your online store has an API endpoint for retrieving product images in various sizes:

GET /product/:id/image/:size

This endpoint returns an image of the appropriate :size (e.g., small, medium, large) for the product of the given :id. To save disk space, you opt to have the API generate each specifically sized image from a single source image using an imaging library like ImageMagick. Since the sales and marketing team uses the API to upload new product images, you set up the endpoint to include a surrogate key:

Surrogate-Key: product/:id/image

When the PUT endpoint for uploading a product image is called, the API sends the following purge request:

POST https://api.fastly.com/service/:service_id/purge/product/:id/image

When Fastly receives this request, we purge all size variations of the product image.

Back to Top