PHP-处理大量数据的最佳方法是什么

Question

Thanks for reading, I'm working on magento website. in my case I want to store all products data in .json files using cron. There are 14 stores and each has more than 2000 products and it's growing in both store and product number. the over all algorithm as the following:

foreach available store
[
  define globalArray = array()
  get store products collection
  foreach product
  [
    add product data to globalArray
    add product filterable attributes to globalArray
    ... 
  ]
  save globalArray in json file
] 

The problem is the required memory is very big, I'm thinking about saving the data for each 200 product in temp json and at the end load all temp json file and merge them in one big json .

---

What could be the best way to deal with that? and is my solution good? thanks in advance

Answer 1

Use the magento core/resource_iterator model. A nice explanation can be found at: http://www.atwix.com/magento/working-with-large-collections/

EDIT: With this recipe you get a nice associative array which you can serialize or jsonify. I guess serializing is quicker.

I tend to not ask why someone wants to do this! I have no clue what you want to achieve with the product data in any non sql format.