GAE datastore restore stops with The API call urlfetch.Fetch() took too long to respond and was cancelled

Question

I am following this guide https://cloud.google.com/appengine/docs/python/console/datastore-backing-up-restoring#restoring_data_to_another_app on how to backup data in one GAE app and restore it in another.

But every time I restore the backup on the target application I get the error:

The API call urlfetch.Fetch() took too long to respond and was cancelled.

Any ideas what I am doing wrong?

Answer 1

Your urlfetch.Fetch() is taking too long (greater than 60 seconds) to respsond and so it is timing out. Here is an article about it https://cloud.google.com/appengine/articles/deadlineexceedederrors

One solution is to use task queues. Task queues have a longer timeout or, more appropriately, let you chop the job up into smaller parts. https://cloud.google.com/appengine/docs/python/taskqueue/

Here is a simple example of how to do this with "push" task queues. I realize going from one datastore model to another might not be the redundancy you are looking for. You may want to backup the datastore entities to another app entirely or another type of database or cloud service. You also probably have multiple models you are backing up. This is just a simple example of setting up and schedule a "push" task queue using a cron job every 24 hours:

first you have to add "deferred" to the builtins in your app.yaml:

builtins:
- deferred: on

Next you need to create a second datastore model we will call "Backup" just copy paste your old model and rename it Backup - It helps to use an identical version of the same model for backups rather than the same model because you can give them the same the primary and the backup the same key:

class Backup(db.Model): # example
    prop1 = db.StringProperty()
    prop2 = db.StringListProperty()
    prop3 = db.StringProperty()

Next setup a cron job in your cron.yaml:

- description: Creates a backup of the target db every 24 hours at 10:45 GMT
url: /backup
schedule: everyday 10:45

Add /backup to your app.yaml handlers:

- url: /backup
script: mybackup.py
login: admin

Finally, create mybackup.py

from google.appengine.ext import deferred
from google.appengine.ext import db
#from google.appengine.ext import ndb

def backup_my_model(model_name):
    """
    Takes all enities in the model_name model and copies it to Backup model
    """
    logging.info("Backing up %s" % model_name)
    query = db.GqlQuery('SELECT * From %s ' % model_name)
    for primary_db in query:
        backup = Backup(key_name = primary_db.key_name)
        backup.prop1 = primary_db.prop1
        backup.prop2 = primary_db.prop2
        ...
        backup.put()



deferred.defer(backup_my_model, MyModel) #where MyModel is the model you want to backup
deferred.defer(backup_my_model, MyOtherModel)
...
deferred.defer(backup_my_model, MyFinalModel)

I hope that helps.