A Note About 'X-Magento-Tags'

If you have ever debugged Magento, it's very likely you saw the X-Magento-Tags in response headers. What is X-Magento-Tags header used for? How is it generated?

In short, those tags are used in Varnish and CDN as identifiers. We will cover the above 2 questions on code level and after reading this blog, you should be able to understand "How It Works", especially "How Cache Purging Works".

In fact, X-Magento-Tags of ESI blocks are excluded (in main page). To make this blog and this topic more complete, we will also talk about why they need to be excluded and how ESI block works.

Recommended Extension

X-Magento-Tags Generation

Again, it's a plugin in Magento_PageCache module.

\Magento\PageCache\Model\Layout\LayoutPlugin::afterGetOutput

/**
 * Retrieve all identities from blocks for further cache invalidation.
 *
 * @param Layout $subject
 * @param mixed $result
 * @return mixed
 */
public function afterGetOutput(Layout $subject, $result)
{
    if ($subject->isCacheable() && $this->config->isEnabled()) {
        $tags = [];
        $isVarnish = $this->config->getType() === Config::VARNISH;

        foreach ($subject->getAllBlocks() as $block) {
            if ($block instanceof IdentityInterface) {
                $isEsiBlock = $block->getTtl() > 0;
                if ($isVarnish && $isEsiBlock) {
                    continue;
                }
                $tags[] = $block->getIdentities();
            }
        }
        $tags = array_unique(array_merge([], ...$tags));
        $tags = $this->pageCacheTagsPreprocessor->process($tags);
        $this->response->setHeader('X-Magento-Tags', implode(',', $tags));
    }

    return $result;
}

See this line.

$tags[] = $block->getIdentities();

Tags and Identities

Tags come from the "identities" of all blocks on the page.

Note those blocks must implement \Magento\Framework\DataObject\IdentityInterface, otherwise they don't have "identities". \Magento\Cms\Block\Page::getIdentities is a good example.

How Cache Purging in Varnish and CDN Works

Check this VCL.

if (req.method == "PURGE") {
    if (client.ip !~ purge) {
        return (synth(405, "Method not allowed"));
    }
    # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
    # has been added to the response in your backend server config. This is used, for example, by the
    # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
    if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
        return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
    }
    if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }
    if (req.http.X-Pool) {
      ban("obj.http.X-Pool ~ " + req.http.X-Pool);
    }
    return (synth(200, "Purged"));
}

Note this part.

if (req.http.X-Magento-Tags-Pattern) {
  ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}

The cache is purged by X-Magento-Tags-Pattern(regular expression).

Purge via CLI

Actually, you can purge the cache manually if you know the tags.

# Suppose we need to purge tags: cat_c_3 and cat_p_5
curl -X PURGE -H 'X-Magento-Tags-Pattern: cat_c_3,.*,cat_p_5(,|$)' https://your-store.com/product1.html

Why Magento uses Tag as identifier instead of something like URL?

We give an example to answer this question. Suppose a product with ID = 5 has its product name updated, all pages contain this product, that is, the product's page itself, category pages, if the product is on homepage then the homepage, if a blog post has the product then the blog post page, all of their cache is needed to be purged.

That's why Magento uses Tag like cat_p_5 for cache purging, otherwise if the identifier is a URL, only one page can be purged.

ESI Block

(Note ESI feature only works with Varnish and some CDNs.)

ESI stands for "Edge Side Includes". To make a block an ESI block, you need to add ttl attribute to <block /> in layout XML(The ttl must be greater than 0).

Tag Exclusion

Check the \Magento\PageCache\Model\Layout\LayoutPlugin::afterGetOutput method again.

if ($isVarnish && $isEsiBlock) {
    continue;
}

When Varnish (or CDN) is employed, identities of ESI blocks are skipped. To answer why they are skipped, you need to know how ESI block works.

ESI Mechanism

When Magento renders a page and meets an ESI block, it will render a placeholder rather the real block content. Then Varnish or CDN fetches the real block content and replace the placeholder with it before generating the final response.

So the tags of ESI blocks should be separated from the main page.

When do we need ESI block and what can we benefit from it?

The answer is to prevent purging everything if a common block is changed.

The most typical example is the menu bar, it is almost on every page. If you updated the menu, for example, changed a category's name, only the menu ESI block will be purged.

Another benefit is that the ESI block can have different lifetime than the main page. Usually the ESI block's lifetime is much shorter. For example, the main page's cache lifetime is 1 day and the ESI block's lifetime is only 1 hour.

Code

The code below shows how X-Magento-Tags are added to ESI block and the generation of block content response.

\Magento\PageCache\Controller\Block\Esi

/**
 * Returns block content as part of ESI request from Varnish
 *
 * @return void
 */
public function execute()
{
    $response = $this->getResponse();
    $blocks = $this->_getBlocks();
    $html = '';
    $ttl = 0;

    if (!empty($blocks)) {
        $blockInstance = array_shift($blocks);
        $html = $blockInstance->toHtml();
        $ttl = $blockInstance->getTtl();
        if ($blockInstance instanceof \Magento\Framework\DataObject\IdentityInterface) {
            $response->setHeader('X-Magento-Tags', implode(',', $blockInstance->getIdentities()));
        }
    }
    $this->translateInline->processResponseBody($html);
    $response->appendBody($html);
    $response->setPublicHeaders($ttl);
}

Closing Thoughts

We hope this blog explained the "How It Works" clearly regarding the X-Magento-Tags.

When designing a block, you should consider:

  • Does the block need identities?
  • Does the block need ESI mechanism?

Still have questions? Contact us.

If you found this blog post helpful, please share it!