Guide for Caching and HTTP Cache Headers for Static Content

In the context of websites and apps, caching is defined as storing content in a temporary storage, like that on the user's browser or device or on an intermediate server, to reduce the time it takes to access that file.

According to HTTP Archive, among the top 300,000 sites, the user's browser can cache nearly half of all the downloaded content. It reduces the time it takes for the user to view the images or Javascript or CSS files. This is because the user now accesses the file from his system instead of getting downloaded over the network. At the same time, caching also reduces the number of requests and data transfer from your servers. Undoubtedly this is a massive saving for repeat page views and visits.

How does Caching Work?

Suppose you open a webpage https://www.example.com, and the server returns below HTML:

...
<link type="text/css" href="https://www.example.com/app.css" rel="stylesheet">
...
<!-- Rest of the HTML -->

When the browser parses this HTML, it identifies that a CSS resource needs to load from https://www.example.com/app.css.

The browser issues a request to the server for this file, the server returns the file and also tells the browser to cache it for 30 days. Later in this guide, we cover how the server sends this instruction to the browser.

Now, let's say you open another page on the same website after a few hours. The browser again parses the HTML and come across the same CSS file on this page as well - https://www.example.com/app.css. Since the browser has this particular resource available in its local cache, it won't even go to the server. The network request is never made, the file is accessed from the local cache, and the styles are applied very quickly.

Caching with Content Delivery Networks in play

We looked at a simple example of how the browser caches a file and uses it for subsequent requests.

Now, let's take it a step further by bringing Content Delivery Networks (CDNs) in the picture. It is essential to talk about CDNs at this stage because they are used to deliver most of the static content like images, JS, and CSS.

If you have users accessing your website from different geographical locations, then CDNs help to improve the page load time by caching a copy of the content on their servers, which are closer to the user.

If you are not acquainted with the concept, you can read more about them in this detailed guide on CDNs.

Resource loading without CDN

Let's take a simple example. Users access different pages on your website, and each page loads your website's logo. Without a CDN, the request for the logo would go from the user's browser to the server before it gets cached on the user's device. Not only would it take longer to get the image from a server located far from the user, with thousands of new users, it also unnecessarily strains your web server.

Resource loading with CDN

To avoid the high load time and reduce the stress on the servers, you use a Content Delivery Network. The CDN caches the static resources on its edge servers or nodes that are physically closer to the user, reducing the time it takes to load the content when it is not there in the user's local cache.

With the context set, in the rest of the guide, we will understand both the layers of caching - one on the user's device and the second on the intermediate layer provided by the Content Delivery Networks - and how to manage them best.

Best practices for cache control

Here are a few things to help you get started while setting up your server, CDN, and the application for proper caching of content. And more importantly, how do you ensure that the user always gets the latest copy of the content.

1. Specifying how long the content can be cached

There are two primary response headers responsible for specifying how long the content stays in the cache.

a. The Expires header

It is used to define an absolute time after which the content should no longer be considered valid for caching.

Expires: Mon, 25 Dec 2020 21:31:12 GMT

b. The Cache-Control max-age directive

The Cache-Control header has a lot of other directives to control the cache behavior. To specify the time for which the content can be cached, Cache-Control has a max-age directive.  It defines a relative time in seconds for which the content can be cached. The directive below allows the content to be cached for 1 hour or 3600 seconds.

Cache-Control: max-age=3600;

If a response contains both the Expires header and the max-age directive, max-age takes precedence.

2. Who can cache the content - public vs private directives

The Cache-Control header has two other directives to specify who can cache the content.

The private directive indicates that the content in the response is meant only for the end user - the final consumer of the content. Therefore, intermediate layers, like the CDN, should not cache it.

The public directive indicates that the response can be cached by the end user and the intermediate proxies like the CDN as well. This is the directive generally used for serving static content like images.

These directives can be used with both max-age and Expires to specify a more complete cache behavior.

Cache-Control : public;
Expires : Mon, 25 Dec 2020 21:31:12 GMT

Cache-Control : public, max-age=3600;

Cache-Control : private, max-age=7200;

You can read more about the Cache-Control header and its other directives in this blog.

3. Setting Optimal Cache Lifetime

The cache lifetime depends on what kind of resource you are trying to cache.

a. HTTP cache headers for images and other static content

In almost all cases, static assets like images, JS, and CSS, do not change on a per-user basis. Thus they can be easily cached on the browser and on intermediate proxies and can be cached for a very long duration. Google generally recommends a time longer than 6 months or even a year for such content.

b. No cache header for HTML and other dynamic content

The content in the HTML or in an API response can change very frequently and on a per user basis. For example, product recommendations will change as the user browses through different products on the website and will be different for two different users. Therefore such content is generally not cached by using the no-cache and no-store directives of the Cache-Control header.

You can use the table below to set the appropriate cache lifetime based on the type of resource:

You can learn more about HTTP caching and different values for the cache-control header.

4. Cached content Validation - ETag and Last-Modified Headers

Let's say you have specified a cache lifetime of 300 seconds, and now a page issues a new request for the same resource. Assuming 300 seconds have passed since the first request, the browser can't use the cached response. Now browser can issue a new request to the server. But, it would be inefficient because if the resource on the server hasn't changed, then it doesn't make sense to download the same resource we have in the local cache.

There are two ways to solve this -

a. A validation token like ETag

The server generates this token, and it is typically a hash (or some other fingerprint) of the content, which means if the content changes, then this token will change. The ETag header is sent in the response of a request.

Cache-Control : public, max-age=3600
ETag: "1239jdsfjn39dsfnjk230dss"

In subsequent requests, the browser sends the value of this ETag in If-None-Match request header. The server checks this token against the latest resource.

If-None-Match: "1239jdsfjn39dsfnjk230dss"

b. The Last-Modified time of the content

This is similar to the ETag token, but instead of using a content hash, this header specifies the time when the content was last modified.

Cache-Control : public, max-age=3600
Last-Modified: Mon, 03 Jun 2020 11:35:28 GMT

In subsequent requests, the browser sends the value of this Last-Modified header in the If-Modified-Since header.

If-Modified-Since: Mon, 03 Jun 2020 11:35:28 GMT

If the token hasn't changed or if the content hasn't been modified since the last modified time, then a 304 - Not modified response is generated. 304 - Not modified response tells the browser that the resource in its local cache hasn't changed on the server, and its cache lifetime can be renewed for another 300 seconds. This saves time and bandwidth.

Explaining Content Validation using ETag

Subsequent requests to validate the content

You need to ensure that the server is providing the ETag tokens by making sure the necessary flags are set. Use these sample server configuration to confirm if your server is configured correctly.

5. Ensuring fresh content is served - Fingerprints in the URLs

The browser looks up the resources in its local cache based on the URL. You can force the client side to download the newer file by changing the URL of that resource. Even changing the query parameters is considered as changing the resource URL. Ever noticed file names like below?

<script src="/bundles/app.bundle.d587bbd6e38337f5accd.js" type="text/javascript"></script>
...
<div class="website-logo">
  <img src="https://www.example.com/logo-white_B43Kdsf1.png">
</div>
<div class="product-list">
  <img src="https://www.example.com/product1.jpg?v=93jdje93">
  <img src="https://www.example.com/product1.jpg?v=kdj39djd">
  ...
</div>
<!-- Rest of the HTML -->

In the above code, snippet B43Kdsf1, d587bbd6e38337f5accd, 93jdje93, and kdj39djd are essentially the fingerprint (or hash) of the content of the file. If you change the content, the fingerprint changes, hence the whole URL changes and browser's local cache for that resource is ignored.

You don't have to manually embed fingerprints in all references to static resources in your codebase. Based on your application and build tools, this can be automated.

One very popular tool WebPack can do this and much more for us. You can use long-term caching by embedding content hash in the file name using html-webpack-plugin:

plugins: [
  new HtmlWebpackPlugin({
    filename: 'index.[contenthash].html'
  })
]

Avoid embedding current timestamps in the file names because this forces the client-side to download a new file even if the content is the same. Fingerprint should be calculated based on the content of the file and should only change when the content changes.

You can also use gulp to automatic this using gulp-cache-bust module. Example setup:

var cachebust = require('gulp-cache-bust');
 
gulp.src('./dist/*/*.html')
    .pipe(cachebust({
        type: 'MD5'
    }))
    .pipe(gulp.dest('./dist'));

6. Improving the cache hit ratio on the CDN

A "cache hit" occurs when a file is requested from a CDN, and the CDN can fulfill that request from its cache. A cache miss is when the CDN cache does not contain the requested content.

Cache hit ratio is a measurement of how many requests a cache can fulfill successfully, compared to how many requests it received.

The formula for a cache hit ratio is:

When using a CDN, we should aim for a high cache hit ratio. When serving static assets using CDN, it is easy to get a cache hit ratio between 95-99% range. Anything below 80% for static assets delivery is considered bad.

Factors that can reduce cache hit ratio

1. Use of inconsistent URLs - If you serve the same content on different URLs, then that content is fetched and stored multiple times. Also, note that URLs are case sensitive.
   
   For example, the following two URL point to the same resource.
   
   https://ik.imagekit.io/demo/medium_cafe_B1iTdD0C.jpg?tr=w-100,h-100
   https://ik.imagekit.io/demo/medium_cafe_B1iTdD0C.jpg?tr=h-100,w-100
   
   But the URLs are different, and hence, two different requests are issued to the server. And two different copies are maintained on the CDN cache.
   
2. Use of timestamps in URLs - If you are embedding current timestamps in URLs, then this changes the URL and response won't be returned from the cache. For example - https://www.example.com/static/dist/js/app.bundle.dn238dj2.js?v=1578556395255
   
3. A high number of image variations - If you have implemented responsive images and have too many variations for every possible DPR value and screen size. It could lead to a low cache hit ratio. If a user gets a tailored image dimension from your servers based on their screen width and DPR, then they might be the first user to request this specific image from your CDN layer.
   
   A rule of thumb is to check popular viewport sizes and device types from your Google Analytics and optimize your application for them.
   
4. Using shorter cache lifetime - Static assets can be cached for a longer duration as long as you have a mechanism to push updates like embedding fingerprints in the URL. Setting a short cache duration increases the chances of a cache miss.

7. Use Vary Header Responsibly

By default, every CDN looks up objects in its cache based on the path and host header value. The Vary header tells any HTTP cache (intermediate proxies and CDN), which request header to take into account when trying to find the right object. This mechanism is called content negotiation and is widely used to serve WebP images in supported browsers and leveraging Brotli compression.

For example, if the client supports Brotli compression, then it adds br in the value of the Accept-Encoding request header, and the server can use Brotli compression. If the client-side doesn't support Brotli, then br won't be present in this header. In this case, the server can use gzip compression.

Since the response varies on the value of the Accept-Encoding header received from the client while sending the response, the server should add the following header to indicate the same.

Vary: Accept-Encoding

Or you can serve WebP images if the value of Accept header has a string webp. The server should add the following header in the image response:

Vary: Accept

This allows us to serve & cache different content on the same URL. However, you should use the Vary header responsibly as this can unnecessarily create multiple versions of the same resource in caches.

You should never vary responses based on User-Agent. For every unique value of User-Agent, a separate object is cached. This reduces the cache hit ratio.

Normalize request header, if possible. Most CDNs normalize the Accept-Encoding header before sending it to the origin. For example, if we are only interested in the value of the Accept-Encoding header has the string br or not, then we don't need to cache separate copy of the resource for each unique value of Accept-Encoding header.

How to Invalidate Cache - Different Methods And Caveats

If you need to update the content, then there are a couple of ways to go about it. However, you should keep in the mind that:

• Browser cache can't be purged unless you change the URL, which means you changed the resource. You will have to wait till the resource expires in the local cache.
• Purging the resource on CDN doesn't guarantee that your visitor will see updated content because intermediate proxies (and browser) could still serve a cached response.

The above limitation leaves us with the following options:

1. Change the fingerprint in the resource URL

Always embed fingerprints in the URL of static assets and do not cache HTML on the browser. This will allow you to push changes quickly and, at the same time, leverage long term cache. By changing the fingerprint in URL, we are essentially changing the URL of resource and forcing the client-side to download a new file.

2. Wait till resource expires in the local cache

If you have chosen your cache policy wisely based on how often you change the content, then you don't need to invalidate the resource from caches at all. Just wait till the resources expire in caches.

3. Use Service Workers To Manage Cache

You can cache files using service workers to manage local cache using the Cache interface.

This is useful when caching content, which often changes such as avatars or marketing banner images. However, you are responsible for implementing how your script (service worker) handles updates to the cache. All updates to items in the cache must be explicitly requested; items will not expire and must be deleted.

You can put items in the cache on network request:

self.addEventListener('fetch', function(event) {
  event.respondWith(
    caches.open('mysite-cache').then(function(cache) {
      return cache.match(event.request).then(function (response) {
        return response || fetch(event.request).then(function(response) {
          cache.put(event.request, response.clone());
          return response;
        });
      });
    })
  );
});

And then later serve it from the cache:

self.addEventListener('fetch', function(event) {
  event.respondWith(caches.match(event.request));
});

4. Purge From CDN

If you need to purge the cache from CDN and don't have any other option, most CDN providers have this option. You can integrate their cache purge API in your CMS so that if a resource is changed on the same URL, a cache purge request is submitted on the CDN.

Conclusion

Here is what you need to remember while caching static resources on CDN or local cache server:

1. Use Cache-control HTTP directive to control who can cache the response, under which conditions, and for how long.
2. Configure your server or application to send validation token Etag.
3. Do not cache HTML in the browser. Always set cache-control: no-store, no-cache before sending HTML response to the client-side.
4. Embed fingerprints in the URL of static resources like image, JS, CSS, and font files.
5. Safely cache static resources, i.e., images, JS, CSS, font files for a longer duration like six months.
6. Avoid embedding timestamps in URLs as this could quickly increase the variations of the same content on a local cache (and CDN), which will ultimately result in a lower cache hit ratio.
7. Use Vary header responsibly. Avoid using Vary: User-agent in the response header.
8. Consider implementing CDN purge API in your CMS if you need to purge content from the cache in an automated way.