Create a large .csv (or any other type!) file with google app engine

Question

I've been struggling to create a file with GAE for two days now, I've examined different approaches and each one seems more complex and time consuming than the previous one.

I've tried simply loading a page and writing the file in to response object with relevant headers:

self.response.headers['Content-Disposition'] = "attachment; filename=titles.csv"
q = MyObject.all()
for obj in q:
    title = json.loads(obj.data).get('title')
    self.response.out.write(title.encode('utf8')+"\n")

This tells me (in a very long error) that Full proto too large to save, cleared variables. Here's the full error .

I've also checked Cloud Storage, but it needs tons of info and tweaking in the Cloud Console just to get enabled, and Blobstorage which can save stuff only in to the DataStore.

Writing a file can't be this complicated! Please tell me that I am missing something.

Answer 1

That error doesn't have anything to do with writing a CSV, but appears to be a timeout when iterating over all MyObject entities. Remember that requests in GAE are subject to strict limits, and you are probably exceeding those. You probably want to use a cursor and the deferred API to build up your CSV in stages. But for that, you definitely will need to write to the blobstore or CS.