Introducing the Static Publish Queue module

Posted by Julian Seidenberg on 3 July 2013

Say your name is Awesome Developer and you’ve built a website for a client. This site runs on SilverStripe, because, let’s face it, you’re awesome and you want an easy-to-use CMS for your client, a CMS that is also packed-full with useful developer features.

Suddenly you are faced with a serious problem. Your client’s name is Popular Scrooge. He says to you one day, “Hey Awesome, you know that site you’re building for me... I want to create a section for pictures of kittens riding skateboards, and post links to all the social networks. You know I have about ten million Twitter followers, right? Will that get me lots of hits? Oh, and by the way, I don’t want to spend any more money.”

You respond, “Hey you Scrooge, both SilverStripe and I are awesome, but SilverStripe dynamically builds every page request. That many hits will totally crash a single web server. We need to buy more hardware and put in some kind of caching server in place.”

Scrooge says, “You are Awesome, I’m sure you can figure out a solution that won’t cost any extra money.”

You wonder what to do, then you (somewhat recursively) read this blog post and learn about the Static Publish Queue module. You enter the following command in your command line to install the module:

composer require silverstripe/staticpublishqueue

Next, you run the three scripts listed in the Quick Setup Instructions. Then, boom! Whenever anyone publishes a page, SilverStripe generates a static html file with the page content. The htaccess file now magically redirects requests so they access the static html file instead of hitting SilverStripe. You can tell that the content you are seeing is served from the static files, because when you look at the page source, each page has an html comment at the end telling you when the static html file was generated.

“Awesome work, Developer”, Mr Scrooge says, “I knew you could do it. It took you ten minutes and you increased the server performance by a factor of 100! Clicking around the site, pages now load virtually instantly. Fastest website I’ve ever seen! How did you do that?!”

You explain that the Static Publish Queue module creates a queue of items to be statically published. Each time you publish a page it gets added to the end of the queue. You can also place all the pages on the site into the queue by running the RebuildStaticCacheTask build task. The StaticPagesQueueReport shows you the status of the queue. The report even auto-updates as the RebuildStaticCacheTask task builds html pages from the queue, a task that runs continuously.

The htaccess rules are smart enough to direct any POST/GET requests with parameters to SilverStripe, the idea being that SilverStripe might want to generate a custom dynamic response to a page called “/search?query=kittens”, while pages without query parameters can safely be directed to the static version of the page.

The end result of all this is an html copy of the entire website structure in a “cache” directory in your web-root. If you are feeling adventurous, you might even upload this structure to Amazon S3 and use that as a static copy of your website in case your web-server goes down. Take a look at this announcement from Amazon to see the technology involved.

So, there you have it - the Static Publish Queue module makes it really easy to add static publishing to your site. The module can do a whole bunch more than described in this blog post: events, distributed cache rebuilding, custom stale pages, integration with proxy server, you name it, it’s got it! Check out the readme file and look into the source code to learn more.

Here is the module on github.

Post your comment

Note: Comments are moderated and won't show until they are approved

Comments

  • Hi Gene,
    Sorry for the delayed response.

    Yes, exclude the User Defined Form pages from the cache. The two modules are incompatible with each other.

    Posted by Julian Seidenberg, 9 months ago @candidasa4

  • This is a useful module. Static Publish Queue feature is nice and reduced the loading times. Also, the presentation is original and very funny.

    Posted by Paul, 9 months ago

  • Hi there,

    Great module!

    Im having a problem with user defined forms. I am getting the 'there was a technical message' when submitted.

    Is it best practice to ignore these page types from the cache rebuild by setting excludeFromCache in an extension of user defined forms?

    Thanks

    Posted by Gene, 9 months ago @senorgeno

  • Thanks a lot for this module.

    Posted by Research In India, 9 months ago @Researchonindia

  • Hi tv, very good point. You found a bug with the module. Thanks for that. I've fixed the bug with this commit in the master branch.

    https://github.com/silverstripe-labs/silverstripe-staticpublishqueue/commit/1121383c8fdff4faeeb7b3280a54e6020bb4cb06

    Please update your module and it should work nicely, without the need to configure anything. The point of the module is to make static publishing super-easy. Convention over configuration and all that good stuff :)

    Posted by Julian Seidenberg, 9 months ago @candidasa4

  • When I attempt to unpublish a page, I get the following error: a Uncaught Exception: Object->__call(): the method 'allpagestocache' does not exist on 'Page' error. Do those methods need to be set on Page for this module? The QuickStart guide didn't mention this, so not sure. If these methods are required, is there any other configuration related to staticpublisher that needs to be done (i.e., extending FileSystemPublisher onto Site Tree)

    Posted by tv, 9 months ago

  • Hi Tim,
    Ah, so you are just trying to bypass the static cached based on a cookie. That you can do just in by adding another rule to the htaccess file.

    RewriteCond %{HTTP_COOKIE} !^.*cookie-name.*$ [NC]

    Add that to all the rules redirecting the requests to the static cache files. It means: if not "cookie-name" set then redirect to cache, otherwise go on to next rules. Eventually the request will get redirected to SilverStripe php execution.

    What I was meaning in my previous reply was the scenario where you want to display a logged-in header and on a statically cached page, not invoking SilverStripe at all.

    Posted by Julian Seidenberg, 10 months ago @candidasa4

  • So I would create a proxy.php script in which I check for if member cookie is set, if true, route the request to main.php else check for cache file if exists or route to main.php...

    In my Page_Controller::init() i could simply check for Member::CurrentUserID() and set a simple cookie...

    The only other problematic area could be around session messages, however they would probably be triggered by post vars...

    I will have a play and let you know!

    PS. you havent approved my previous comment.

    Posted by Tim, 10 months ago @dodatNZ

  • Hi Tim,
    Glad you like the module. The module does not handle dynamic portions of the page.

    What you can do in a situation where there is high traffic but you still want e.g. a logged-in header is to set the required data in a cookie and access that cookie with client-side Javascript, then using this Javascript to change the header message on the page.

    Alternatively, for more complicated dynamic behaviour, you can follow the instructions under the "Custom .htaccess and the stale-static-main.php" section, and customise the php file to do some custom processing of the response. For example, I have used this to rewrite all image URLs to https when the page is accessed via HTTPS.

    Running a single simple php script is a lot quicker than spinning up SilverStripe fully, let alone the overhead involved in connecting to the database.

    Posted by Julian Seidenberg, 10 months ago @candidasa4

  • Love this post, Nice reading :-) and the module sounds very usefull for High traffic sites.

    Posted by Thomas B. Nielsen, 10 months ago @nobrainerweb

RSS feed for comments on this page | RSS feed for all comments

Want to know more about the company that brought you SilverStripe? Then check out SilverStripe.com

Comments on this website? Please give feedback.