LOG IN SIGN UP
Documentation

Getting started with surrogate keys

Efficient cache invalidation is an essential part of keeping your website fast. Purging too much cache using purge all may increase your website's load time while the cache rebuilds. If you find yourself purging all cache on more than a weekly basis, consider using surrogate keys for more targeted purging.

Surrogate keys allow you to selectively purge related content. Using the Surrogate-Key header, you can tag a group of objects with a key and then use it to purge multiple pieces of content at once. This process can occur automatically within your application, making it easier to cache and purge content that changes rapidly and unpredictably.

This guide consists of four parts:

Understanding surrogate keys

After you've signed up for Fastly and added one or more services, you can start examining how your origin server responds to requests. When your origin server responds to an HTTP request for content, it's because Fastly hasn't yet cached that content or the cache has expired. Your server's response to the request will resemble the example shown below. (Note that you can use the curl command to inspect any of your server's responses.)

HTTP/1.1 200 OK
Content-Type: text/html
Connection: keep-alive
...

To control how your content is served to users and cached by Fastly, you can add to or modify the headers that are included in your origin server's response. The Surrogate-Key header is one of the headers that you can add to the response. It allows you to "tag" an object, such as an image or a blog post, with one or more keys. When the object changes, you can reference the key in a purge request to remove the object from the cache.

You can add space-delimited strings to the Surrogate-Key header, like this:

HTTP/1.1 200 OK
Surrogate-Key: key1 key2 key3
Content-Type: text/html
...

This response contains three surrogate keys: key1, key2, and key3. When Fastly receives a response like this, we use the surrogate keys to create a mapping from each key to the cached content, then we strip out the Surrogate-Key header so it's not included in the response to your readers.

Creating relationships between keys and objects

One of the major advantages of surrogate keys is that they allow for a many-to-many relationship between keys and objects. An origin server's response can associate multiple keys with the object, and the same key can be provided in different responses. Take a look at these two requests and responses:

GET /blog/ HTTP/1.1
Host: www.example.com

HTTP/1.1 200 OK Content-Type: text/html
Content-Length: 1234
Surrogate-Key: mainpage template-a
GET /blog/article/fastly-rocks HTTP/1.1
Host: www.example.com

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 2345
Surrogate-Key: template-a article-fastly-rocks

In this example, there are two objects (/blog and /blog/article/fastly-rocks) with three keys (mainpage, template-a, and article-fastly-rocks). Two of the keys (mainpage and article-fastly-rocks) are associated with a single object, and a third key (template-a) is associated with both objects.

Purging objects with surrogate keys

By using the Surrogate-Key header to associate keys with one or more objects, you can precisely control which objects are removed from cache during a purge. Consider the example presented above. Purging the mainpage key would remove only the /blog object from the cache. On the other hand, purging the template-a key would remove both the /blog and /blog/article/fastly-rocks objects from the cache.

You can use the Fastly web interface to manually purge objects via key, or you can use our Purge API to purge a collection of objects with one API call. If you're using Fastly to cache your API, check out the guide on purging API cache with surrogate keys to learn how surrogate keys can help you purge API cache.

Looking at a practical example

Let's look at a practical example to learn how surrogate keys work. Imagine you're building a picture-hosting website and you're using Fastly to speed it up. You initially decide to cache picture pages by their URL (e.g., http://www.example.com/pic/id). When a picture's information changes, you'll remove the old version by sending a purge request to the Fastly API:

PURGE /pic/{id}
Host: example.com
Accept: */*

But there's a potential problem with this solution. You display a user's information next to their pictures, so you'll need to purge all of the user's picture pages if they change their information. You could send an individual purge for each picture, but that would take too long. As an alternative, you could cache the pages for a very short amount of time, but that would waste our server resources.

Surrogate keys solve this problem. By adding a surrogate key to all of a user's picture pages (e.g., /user/542, /user/25), you can purge all of the user's pictures by sending Fastly a purge for the user's surrogate key when they update their information. Now, instead of having to purge each picture individually, you can update them all with just one request:

PURGE /service/id/purge/user/542
...

Purging multiple sites at the same time

What if you wanted to build a mobile version of the picture-hosting website to complement the desktop version? You'll need a way to purge both the desktop version and the mobile version at the same time. Surrogate keys can help in this instance. You can tag the different versions of a picture page with the same surrogate key (e.g., pic/76, pic/345) and purge them all at once. All of the related content on our sites can now be purged with one request.

Tagging templates with surrogate keys

Surrogate keys come in really handy when making changes to templates. Imagine you have to make a change to the banner of the website. Since you're caching entire page, updating the header template isn't enough. You'll also need to purge all the pages that use the template. You could purge every page on the website, but there's no reason to purge content that doesn't use the header template.

You can make things easy by using surrogate keys. By adding surrogate keys for each template on a page (e.g., /templates/pic/show, /templates/pic/header, /templates/pic/comment), you can check which templates have changed and purge only pages with modified templates.

Generating and setting surrogate keys

There are two ways to set the Surrogate-Key header: by adding the header in the Fastly web interface, or by generating the keys with your own application. We describe how to use the Fastly web interface in our guide to generating Surrogate-Key headers based on URLs (we have a separate guide for Amazon S3 origins).

Automatically generating keys with your own application is described below using Fastly's Test Blog as an example. The test blog is a Ruby on Rails application that comes preloaded with example content that can be cached and purged using Fastly via the fastly-rails Ruby gem. (We also have other API clients that support surrogate keys.)

Configuring the API client

The fastly-rails gem provides a set_surrogate_key_header method which the test blog uses to automatically generate surrogate keys for new articles. You can see how this works by examining the code in articles_controller.rb. A slightly modified excerpt of the code from the controller is shown below.

class ArticlesController < ApplicationController
  # include this before_action in controller endpoints that you wish to edge cache
  # This can be used with any custom actions. Set these headers for GETs that you want to cache
  # e.g. before_action :set_cache_control_headers, only: [:index, :show, :my_custom_action]
  before_action :set_cache_control_headers, only: [:index, :show]

  # Returns all Article objects, and sets a table_key of 'articles',
  # and a record_key for each article object: "#{table_key}/#{article_id}"
  def index
    @articles = Article.all
    set_surrogate_key_header 'articles', @articles.map(&:record_key)
  end

  # Sets a surrogate key for the current article.
  #
  # Example:
  #
  #   Article[75]
  #   Surrogate-Key:articles/75
  def show
    set_surrogate_key_header @article.record_key
  end

  ...
end

The before_action method creates Cache-Control and Surrogate-Control HTTP headers with a default TTL of 30 days. This method must be added to any controller action that you want to edge cache. The set_surrogate_key_header method sets Surrogate-Key headers for objects that you want to be able to purge. In this case, surrogate keys are set for each article and the articles index.

Examining the headers

Now that you've looked at how the surrogate keys are generated by the test blog, inspect the headers on an article page on the origin server. Here's the partial output from curl -svo /dev/null origin.example.com/articles/1:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: public, no-cache
Surrogate-Control: max-age=86400
Surrogate-Key: articles/1

Notice how the Cache-Control, Surrogate-Control, and Surrogate-Key headers are present in the response shown above. Thanks to the set_surrogate_key_header method, the test blog application automatically generates the unique articles/1 surrogate key for this article.

Next, inspect the headers on the article index page on the origin server URL. Because this page lists all of the articles, you might expect it to contain the surrogate keys for every article displayed on the page, and that is indeed the case. Here's the partial output from curl -svo /dev/null origin.example.com:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: public, no-cache
Surrogate-Control: max-age=86400
Surrogate-Key: articles articles/1 articles/2 articles/3 articles/4 articles/5 articles/6 articles/7 articles/8 articles/9 articles/10 articles/11 articles/12 articles/15 articles/16 articles/17 articles/18 articles/19 articles/20 articles/27 articles/28 articles/29 articles/30 articles/31

If you purged the articles/1 surrogate key, both http://origin.example.com/articles/1 and http://origin.example.com would be purged from the cache.

Finally, take a look at the URL that's piped through Fastly: http://fastly.example.com. The surrogate keys won't be visible in the headers, but Fastly knows what they are. Recall from understanding surrogate keys that Fastly strips out the Surrogate-Header and creates a mapping from each key to the cached content. Here's the partial output from curl -svo /dev/null fastly.example.com:

HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: public, no-cache
Content-Type: text/html; charset=utf-8
Via: 1.1 varnish
Age: 78309
X-Served-By: cache-iad2120-IAD
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1449255701.272992,VS0,VE5

With the Surrogate-Key header present on the index and article pages, you're now able to manually purge blog pages via key, or purge a collection of pages with one API call.

Limitations

The surrogate keys sent by your origin server can be as simple or complex as you need, but there are a couple limitations:


Additional resources:


Back to Top