Magento Sitemap and / or Sitemap index issues? - XML, PHP?

Question

Can any help recommend the best strategy for our Magento Sitemap?

We have a large Magento store with nearly 500,000 products.

Obviously we therefore have a need to split our sitemap into 50,000 chunks and have a sitemap index file. The issue is creating these sitemaps regularly and splitting them down.

When we create a sitemap using the feature on the Magento admin panel, it times out and goes to a white screen. We have tried an extension from Mageworx to split them down - but they couldn't get it working.

What is the best way of having an automatically generating XML sitemap index, point to 10 or so sitemaps down from that?

We are on Magento community, 1.5.1.0.

Answer 1

You could possibly use a crawler based sitemapper, eg http://a1-sitemap-generator.com

But half million URLs is a lot to crawler based sitemapper. However, if have 7-14 different "sections" in your website you could possibly try create a sitemap of each section (eg scheduled to be updated weekly), and then you could link all the XML sitemaps in a sitemap index file.

You could also contact the developers of the Magento sitemap solutions you have used. Maybe they can be configured to also just create a sitemap covering a part of the underlying Database. Possibly they could improve their code for handling large websites.

You could also consider server configuration. Maybe the script is just running for too long iterating through the database. There is often a server/PHP option when a script gets killed for running too long time on server.

Answer 2

I suggest you run the Magento sitemap generator from Linux Command but not from the browser, cause it's impossible to run so long time from the client.

I recently write a php script extends magento abstract.php from shell direcotry. It works, but slow.

No matter how, so many products like your site, the basic functionality of generating sitemap in Magento is impossible to run successfully. Have a try to extends abstract.php at shell directory.