magento open source full page cache

Question

I was speaking to a developer recently who said that no magento CE full page cache worked perfectly even with a default install.

my experience of this is the same.

if you use aoe_static's awesome module and phoenix's full page cache with varnish module you will get close.

with the aoe_static module caching is done on a per Action basis which seems sensible to me. hole punching is done with placeholders applied via layout xml and dynamic blocks requested via a ajax call which won't be cached. cookies are set with this call as well. he provides a default vcl which is works with varnish 2 (i never checked) and can be easily changed for varnish 3.

cache invalidation is handled quite well by the phoenix module. you can see the methods in the control folder in the module. this takes care of cache invalidation when products or categories are changed and invalidates both product pages and category pages. however the url's generated by the layered nav are likely to be cached (depending on your vcl). these are not invalidated, so watch out folks.

i'd really like to know if any other problems exist with these modules before i use them on production sites. if anyone could point me to a problem i'd be happy to try and post a solution.

but for the layered nav url's a model would be required to log the generated url's with category id. i suppose it would be simple to have a 2 column model - url and category - and then cache invalidation would need to check the model and invalidate the extra url. this should be done at the same time as the main category url gets done. doesn't sound too bad. if anyone gets my very brief explanation please advise on if i'm missing something before i waste my time.

i would rather create an open source project with some community help (or someone more experienced at the lead) that has a (deserved) reputation for reliability in a default 1.7 install at least. i think the most sensible thing would be to create one module with all the edge cases (for a default 1.7 install) covered or documented.

does anyone know how to go about such a mission? if there was more community support to do this with apc or memcached that might be more sensible for wider hosting availability. the logic or the development time wouldn't change much.

this could be expanded to cover session caching of blocks which should be kept in mind but i think focus on a reliable full page cache should be the priority.

you can also detect devices in a vcl. this could be added to shops wanting mobile version https://github.com/varnish/varnish-devicedetect/blob/master/devicedetect.vcl

i've created an email for this project so if you want to be involved it's "magefpc gmail com". anyone with experience of magento, git and debugging would be welcome.

thanks

Answer 1

1 problem what you might have with aoe_static is, that currently the extension doesnt log products so statistics in backend about viewed products are not updated. this was added recently as TODO note into the github.

problem with varnish and magento cache is, that magento cache parts of pages, blocks and varnish does cache whole pages. so even though phoenix solution have observers in place and reset varnish accordingly on product/category update, you still have blocks in magento what have to be reseted. aoe_static try's to solve this, but it does only for visible blocks on the page. but other parts of magento, as layered navigation, might be reseted as well internally on product/category update, but than are not reset in varnish. there are 2 solutions. either identify every part of magento on frontend whats not reset after product/category update and apply aoe_static solution to it. that would mean you would have to load layered nav via ajax. or the second solution is, to add more logic into phoenix extension to invalidate more varnish pages.

i have couple of custom extensions (pages what display products) and i build logic to invalidate those extensions each time product/category is updated. now im working on search, as this needs to be invalidated as well, but the complicated part is, to work around blocks and cache invalidation logic.

overall, proper cache invalidation is one of the hardest part of web development.

Answer 2

i'd like this post to be a community answer detailing approaches to particular handles and suggested caching and invalidation logic needed if it's not already included in the phoenix or aoe_static module. basically caching will be done by module/controller/action pattern laid out in the aoe_static module and most of the existing invalidation exists in the Phoenix module.

so we need to decide which blocks should be rendered via ajax and where addition cache invalidation is required. like gondo says you'll need to add a lot more invalidation logic to cache searches. i plan to start smaller with the catalog, cms and aw blog pages, much has been done here but a little more is required. if we could add search too, with the help from gondo :), that would be great.

for instance - widget instances could be a problem, as could static block saves. this can (as far as my very quick investigation has shown) be easily solved only by running a total cache invalidation using the event dispatched by the core_model_abstract method _beforeSave()

  Mage::dispatchEvent('model_save_before', array('object'=>$this));

we'd need to match the type of model before invalidating the entire cache. this pattern will also be a real time saver in other difficult areas of cache invalidation. using the instanceof method is fine but we'll need to know model names to check for. i guess community involvement is great here because the code's so easy but so is missing a model.

widgets are essentially the same from frontend point of view. the one with products should be loaded with aoe_static, as they are changing quite often. but the static ones can be treated as static (html) blocks.

we don't want to invalidate the entire cache when a product is saved and can't check if the product has been included in a widget. but the aoe_static module works with layout rendering. widgets will need to be included via layout xml and not in the admin wysiwyg as the often are. these widgets could be added to the call controller handle and place holders to what ever handle is needed.

full varnish invalidation, time to time, is not a bed thing. the whole cache invalidation is an issue if it happens too often, or if it does not happen when needed. the main goal should be to capture every invalidation scenario and trigger notice in admin panel, same type of message as you get when you save product: "One or more of the Cache Types are invalidated..." this way admin can make the decision if he/she wants to invalidate cache manually or wait. and ofc there can be some automatic invalidation logic build in based on this scenarios.

also the model must be created to store the url's created by layered nav and these should be invalidated along with the category view url's.

and with layered nav and invalidation in general in mind we must invalidate the entire cache when an attribute is saved (i think).

we should be able to put the product reviews on the product page without ajax for seo purposes.

this means when a review model is saved it must be checked using the model_save_before event and we should invalidate the product page url. this save changes the rating but i suggest this block is grabbed in the ajax call. it's not important for seo. this goes for the product view and list pages.

we could also cache and invalidate review module pages but this isn't important to the majority of users and should wait.

who wants tags blocks. these can't be loaded by ajax it defeats their purpose. at the same time they're not worth our efforts now imo.

if you cache url's with the session id appended it will cause users to get each others sessions i'm told. i think this is easily solved by piping in vcl_fetch based on a regex.

this answer is getting messy and should be redrafted sometime soon. feel free to add your 2 cents!!