如何设计内存和计算密集型程序以在Google App Engine上运行

Question

I have a problem with my code running on google app engine. I dont know how to modify my code to suit GAE. The following is my problem

for j in range(n):
 for d in range(j):
  for d1 in range(d):
   for d2 in range(d1):
    # block which runs in O(n^2)

Efficiently the entire code block is O(N^6) and it will run for more than 10 mins depending on n. Thus I am using task queues. I will also be needing a 4 dimensional array which is stored as a list (eg A[j][d][d1][d2]) of nxnxnxn ie needs memory space O(N^4)

Since the limitation of put() is 10 MB, I cant store the entire array. So I tried chopping into smaller chunks and store it and when retrieve combine them. I used the json function for this but it doesnt support for larger n (> 40).

Then I stored the whole matrix as individual entities of lists in datastore ie each A[j][d][d1] entity. So there is no local variable. When i access A[j][d][d1][d2] in my code I would call my own functions getitem and putitem to get and put data from datastore (used caching also). As a result, my code takes more time for computation. After few iterations, I get the error 203 raised by GAE and task fails with code 500.

I know that my code may not be best suited for GAE. But what is the best way to implement it on GAE ?

Answer 1

There may be even more efficient ways to store your data and to iterate over it.

Questions:

• What datatype are you storing, list of list ... of int ?
• What range of the nested list does your innermost loop O(n^2) portion typically operate over?
• When you do the putitem, getitem how many values are you retrieving in a single put or get?

Ideas:

• You could try compressing your json (and base64 for cut and pasting). 'myjson'.encode('zlib').encode('base64')
• Using a divide and conquer (map reduce) as @Robert suggested. You may be able to use a dictionary with tuples for keys, this may be fewer lookups then A[j][d][d1][d2] in your inner loop. It would also allow you to sparsly populate your structure. You would need to track and know your bounds of what data you loaded in another way. A[j][d][d1][d2] becomes D[(j,d,d1,d2)] or D[j,d,d1,d2]

Answer 2

You've omitted important details like the expected size of n from your question. Also, does the " # block which runs in O(n^2) " need access to the entire matrix, or are you simply populating the matrix based on the index values?

Here is a general answer: you need to find a way to break this up into smaller chunks. Maybe you can use some type of divide and conquer strategy and use tasks for parallelism. How you store your matrix depends on how you split the problem up. You might be able to store submatrices, or perhaps subvectors using the index values as key-names; again, this will depend on your problem and the strategy you use.

An alternative, if for some reason you can not figure out how to parallelize your algorithm, is to use a continuation strategy of some type. In other works, figure out about how many iterations you can typically do within the time constraints (leaving a safety margin), then once you hit that limit save your data and insert a new task to continue the processing. You'll just need to pass in the starting position, then resume running from there. You may be able to do this easily by giving a starting parameter to the outermost range, but again it depends on the specifics of your problem.

Answer 3

Sam, just give you an idea and pointer on where to start.

If what you need is somewhere between storing the whole matrix and storing the numbers one-by-one, may be you will be interested to use pickle to serialize your list, and store them in datastore for later retrieval. list is a python object, and you should be able to serialize it.

http://appengine-cookbook.appspot.com/recipe/how-to-put-any-python-object-in-a-datastore