Magento 2 Free Sitemap Based Cache Warmer Extension

A Free Magento 2 Extension designed for CDN and Varnish cache warming. Warm up URLs located in sitemaps, as well as static content, media content and dynamic catalog images. Run manually or schedule by cron expression. This extension is easily customizable with dispatched events and the CLI command is ready to be embedded in deployment scripts.

Introduction

Magento 2 comes with Full Page Cache out-of-the-box. Users can choose "Built-in Cache", "Varnish Cache" or "CDN"(requires installing the CDN's free extension).
However, no matter which caching application is employed, Magento 2 still lacks the ability to let the caching application serve "Warm Cache". In other words, a web page feels slow when it is visited the first time. The reason is at that time, the CDN didn't have the warm cache yet.
Adobe Commerce Cloud does have a Post-deploy cache warming environment configuration, but it is not a Magento extension and not user friendly. It only offers limited features. For example, it can neither warm up static content nor keep the populated cache warmed.

The goal of this extension is to become an ultimate solution to suit the needs of cache warming for a Magento store. It focuses on real-world business use cases by covering widely used content types including blog posts and keeps automation in mind.

Feature Highlights

• Support 4 Content Types

• Warm up HTML pages located in sitemaps or URL list files
  

• Warm up static and media content(i.e., files in pub/static and pub/media directories) with configurable filters
  Those contents are typically, JS, CSS and image files.
  

• Warm up Dynamic Catalog Images
  (This content type requires Magento >= 2.4.2)

• Schedule tasks by predefined or user-defined cron expression

  • Which frequency should I use and how to keep the cache warmed?

• A button to schedule a task to be run in the next minute
  It can be considered as a non-blocking "Run Manually"

• Control task duration by setting global timeout and request sending concurrency

• GUI task status display with progress information

• Custom HTTP proxy for environments with restricted outbound connections
  The user can also disable HTTP proxy entirely or just use the system one

• Internal HTTP client tweaking
  Available configurations: Timeout, Download Size Limit(Security Concern), Retries, Retry Delay
  The purpose of these configurations is to prevent getting stuck in the cache warming process
  

• CLI command with progress details
  Usage
  

• Deployment Automation

Why Sitemap Based?

Instead of retrieving URLs from the standard "3 content types" (i.e., product, category and CMS) internally, it is more practical to extract URLs from sitemaps. In this way, content types introduced by third-party extensions such as blog posts, product tags are covered as well. Any custom content which includes itself into the sitemap can be warmed up.
In addition, our design gives the user more freedom and flexibility. The user can specify multiple sitemaps as Sitemap URIs, which means the sitemap to feed the Cache Warmer can be automatically generated by Magento's built-in feature and/or managed by external tools. The user can even use gzipped sitemap.

Why URL List?

A URL list is simply a text file containing line separated URLs. For most systems, generating such a file is extremely straightforward.

A typical use case is when the store has like more than 10,000 products. Warming up the entire store will cost a huge amount of resource and time. The recommended way is to pick some products from the entire store by the user's own criteria and then put their URLs in a txt file to feed the Cache Warmer.
To be more specific, for example, the user can export products from the "Top Sellers" category to a CSV file and get their url_keys to build full URLs. Then the user saves those URLs to a top-sellers.txt file which will be used as a Sitemap URI.

Supported Sitemap URI and Sitemap Formats

Sitemap URI

Security Warning

The "Sitemap" can be on local or remote servers and acts as a "user input".
It is the user's responsibility to make sure the content of the sitemap is controlled only by the user.

Supported Schemes

• http://
• https://
• file://

Supported Sitemap Formats

• XML sitemap
• Line separated text file(URL list)
• GZIPped file of the above two
• robots.txt containing Sitemap: directives

Sitemap URI Example

# Line starting with # is treated as comment.
#
# -- Sitemap --
https://www.example.com/sitemap.xml
#
# -- Line separated URL list --
https://www.example.com/hot-urls-list.txt
#
# -- Local file --
file:///var/app/pub/sitemap.xml
#
# -- Gzipped Sitemap --
https://www.example.com/sitemap.xml.gz
#
# -- Robots --
https://www.example.com/robots.txt

URL List Example

A .txt file with content like:

https://www.example.com/product1.html
https://www.example.com/product2.html
...

Static and Media Content

The Cache Warmer scans pub/static and pub/media directories and build all URLs for those files.

File Exclusion

• Files with size larger than the "Maximum File Size" will be excluded. The "Maximum File Size" value should be smaller than the CDN's maximum cacheable file size limit
• Unix-like hidden files(file name starting with a .) and files in hidden directories are always ignored for security concerns
• Files with the following extensions are ignored because they cannot be cached by CDNs
  (There are many reasons can be found in Magento's codebase, such as here, here and here)

  zip gz gzip bz2 csv xml

Static Content

• Static content must be deployed first before warming it up
• All deployment strategies are supported by this extension
• Only store frontend static content will be warmed up

Dynamic Catalog Images

Requirements

• Magento >= 2.4.2

Magento 2.4.2 introduced a new feature -- Image optimization based on query parameters for catalog images.

Configuration Location

Stores > Settings > Configuration > Url Options > Catalog media URL format

This feature can let product use Dynamic URLs with resizing parameters.

Dynamic Product Image URL Example

https://www.example.com/media/catalog/product/p/r/product-image.jpg?width=350&height=500&store=main&image-type=small_image

The Cache Warmer is capable of "calculating" all of those URLs.

Important Notes

This content type will be automatically disabled at runtime if the Magento version does not meet the requirements or the above configuration is not set to "Image optimization based on query parameters".

Images of disabled products are excluded.

CDN Caching Behavior

Cache Lifetime

The cache warming is not a "one-time job", CDN cache has lifetime. They have a TTL(Time-To-Live) configuration, but this configuration is usually used as a fallback. In most cases, the TTL is informed by HTTP response headers which can be controlled by the web application(i.e., Magento).

However, somemany CDNs do not "respect" the specified TTL. According to their documentation, less popular content will be removed before its TTL expires and this is an expected behavior.
For example, we observed that on Fastly, if a web page is not accessed for 2 hours, it will be removed from their cache even the TTL is 1 day.

To keep the populated cache warmed, adjust the "Frequency" configuration properly.

Maximum Cacheable File Size Limit

CDNs always have a maximum cacheable file size limit. This limit can be found in their documentations and varies from CDN to CDN, plan to plan. Files larger than this limit cannot be cached by CDN, hence there is no need to warm up those files.

To exclude those large files, set the "Maximum File Size" value properly.

How Long Does The Process Take?

This extension already makes use of simultaneous requests to speed up the cache warming process.

The time it takes mainly depends on how many products the store has and the web server's performance. If the store has a lot of blog posts, CMS pages, this will become an affecting factor as well.

In our test case, the process takes around 20 ~ 40 minutes to finish for a store having around 1000 products and concurrency set to 25.

Bonus Feature

Output failed URLs collected from the cache warming process with Error Codes to a CSV file

This feature demonstrates how to customize this extension by utilizing its dispatched events.
The Error Codes can help diagnose problems, especially problems in sitemaps.

Error Codes

1. Negative error codes should not occur. Negative error code indicates there is probably a bug in this extension or its dependencies or its customization.

2. The cause can be on the HTTP proxy server side. For example:

   • The connection timeout may be caused by an incorrect HTTP proxy server address
   • The HTTP proxy server may not be able to resolve domain name

If you saw negative error codes but did not customize this extension, please report the error to improve this extension.

Command Line Usage

php bin/magento ww:cache-warmer:run [--try-number TRY-NUMBER]

This command is designed for "Run Manually".

ww:cache-warmer:run can be shorten to ww:cw:run.

--try-number is optional and for debug purpose only. It limits how many URLs will be picked out from each "Content Type". This option can be used to perform a quick test of sitemaps and configurations.

Deployment Automation

It is a good practice to place the command in deployment scripts.
Once the extension is properly configured, you can forget about it and the CDN cache will stay warm constantly.

HTTP Proxy

The user can just use the system-provided HTTP proxy server or a custom one.

This feature is also capable of handling the following situations.

• The system has a default HTTP proxy and the user wants to use a different one
• The system has a default HTTP proxy and the user wants to bypass it

HTTP Proxy Mode

Logging

Task run results are logged at

<magento_root>/var/log/wubinworks_cachewarmer.log

Free Forever

You DON'T need to worry that some features are not free or this repository will be silently deleted or even the entire extension becomes a paid one.

You can use this extension in your projects as an open dependency.

License: Open Software License (OSL 3.0)

Although this is a Free extension, it is aiming to be professional, fit most business scenarios and become a complement of Magento's Page Caching feature. If you have good ideas about this extension, do not hesitate to share them.

Installation

Requirements

• Magento 2.3 or 2.4

Important Notes: this extension is not tested on Magento 2.3, but you can give a try and any feedback is appreciated.

Install via composer

composer require wubinworks/module-cache-warmer

This extension requires dependencies that Magento 2 does not come with by default.
Installing to app/code will not work.

Configuration Location

Stores > Wubinworks > Free Cache Warmer

Report Issues

General Issues

• Bugs
• Ideas and questions
• Translation contribution and correction

Use Github Issues.

Security Issues

Use the Contact Form.
NEVER OPEN A SECURITY ISSUE PUBLICLY

Customization Issues

Please contact us directly.
The "Github Issues" is only for problems residing in the extension itself.

For Developers

Translation

Current Available Languages

• English
• Japanese

Translation Contribution

• Translation contribution or correction is welcome.

• Translation Contributors

Dispatched Events

• wubinworks_cachewarmer_cron_run_before
  Area: crontab
  By observing this event, the developer can perform custom actions such as interacting with the CDN's API before running the task. The developer can also obtain the schedule information and implement custom logic to make a final decision whether to run or skip the task.

/** @var \Magento\Framework\Event\Observer $observer */

// Get schedule model
/** @var \Magento\Cron\Model\Schedule $schedule */
$schedule = $observer->getSchedule();

// Skip entire task
$observer->getEventData()->setSkipTask(true);

// Set OPTIONAL skip task message
$observer->getEventData()->setSkipTaskMessage(
    'This message will be logged and displayed as the task progress message when task was skipped.'
);

• wubinworks_cachewarmer_cachewarmer_on_progress
  Area: global
  By observing this event, the developer can get the task progress information and its current processing result.

/** @var \Magento\Framework\Event\Observer $observer */

// Get run type
var_dump($observer->getRunType());
// 'CRON' or 'CLI'

// Get progress information
var_dump($observer->getProgress()->getData());

// Get current item
/** @var \Magento\Framework\DataObject $currentItem */
$currentItem = $observer->getCurrentItem();

var_dump($currentItem->getUrl());
var_dump($currentItem->getStatus()); // 'success' or 'error'
var_dump($currentItem->getCode()); // Check "Error Codes" section

• wubinworks_cachewarmer_cachewarmer_run_after
  Area: global
  By observing this event, the developer can get the final results of the task, such as URLs with invalid responses. Then the developer can implement custom logic to handle those results.

/** @var \Magento\Framework\Event\Observer $observer */

// Get run type
var_dump($observer->getRunType());
// 'CRON' or 'CLI'

// Get Cache Warmer report
/** @var \Wubinworks\CacheWarmer\Model\CacheWarmer\Report $report */
$report = $observer->getReport();

// Get failed sitemaps
var_dump(reset($report->getFailedSitemaps()));

// Get failed URLs
var_dump(reset($report->getFailedUrls()));

Miscellaneous

• The purpose of this extension is to warm up CDN/Varnish cache. Any other usage is considered out of development scope.
• The Cache Warmer only works for the current Magento instance. This is also a security concern. If you found any URLs are not warmed up, please check the log.
• This extension supports multiple store setup and by default the Cache Warmer runs for all stores. There is a hidden configuration that can exclude certain stores.

♥

If you like this extension or this extension helped you, please share it and ★star☆ this repository.

You may also like:
Magento 2 Disable Customer Extension
Magento 2 Disable Customer Change Email Extension
Magento 2 Price Formatter Extension