安排在Google App Engine中运行超过十分钟的任务的最佳方法是什么？

Question

Let's say I have a script that runs for an hour. All it does is sleep for an hour and change something in the Datastore at the end of the hour.

def task(keyname):
    time.sleep(3600)
    q = Item.all()
    item = q.filter("keyname = ", "%s" % keyname).get()
    item.value += 1
    item.put()

What would be be the best way to accomplish this?

The cron documentation says that "an HTTP request invoked by cron can run for up to 10 minutes, but is subject to the same limits as other HTTP requests." I'm assuming this means cron tasks that run over ten minutes will break.

I looked at the Task Queue documentation, but I don't want to queue these tasks one after another. I want them to happen concurrently.

I was thinking of adding a counter property and running the task every minute instead of sleeping:

def task(keyname):
    q = Item.all()
    item = q.filter("keyname = ", "%s" % keyname).get()
    if item.counter == 60:
        item.value += 1
        item.put()
    else:
        item.counter += 1
        item.put()

but that accesses the database way too many times.

So, does anyone have a smarter workaround?

Answer 1

I don't understand what you mean when you say you want "these tasks" to happen concurrently; you've only mentioned a single task.

Regardless, the task queue is the way to do this. You can use the countdown argument to the Task constructor to tell it to execute in 3600 seconds.

I usually prefer to use the deferred library , which creates tasks dynamically without needing to set up specific handlers:

from google.appengine.ext import deferred
...
deferred.defer(my_task, _countdown=3600)

Answer 2

The cron job hits a handler, and this handler starts a task. This process usually takes less than a second. After that the task can run for as long as you need it - forever, if necessary. It all depends on the target of a task . 10 minute limitation applies to front-end (F-type) instances only.

And you can run many tasks concurrently if necessary.

Answer 3

@Soricidae, you're building a cron task.

1. For this you need to have a handler that calls your method task .
2. Also, you need to specify a URL route to that handler.
3. Then you need to specify in your cron.yaml a cronjob calling for the task method.

*. Additionally I deeply suggest using taskqueues in order for the appengine to solve your load time issues during the cron task.

For example, in my case I have a welcome handler that emails users, checking every 24 hours for new users with more than 1 day that have not yet been welcomed...

So, I've defined the following routes (1 for cronjob, 1 for taskqueue):

RedirectRoute('/cronjob-welcome/', handlers.WelcomeCronjobHandler, name='cronjob-welcome', strict_slash=True),
RedirectRoute('/taskqueue-welcome/', handlers.WelcomeHandler, name='taskqueue-welcome', strict_slash=True),

Then, I've defined a cronjob handler that once called it goes for the real operations handler as a taskqueue, see:

class WelcomeCronjobHandler(BaseHandler):
    def get(self):
        welcome_url = self.uri_for('taskqueue-welcome')
        taskqueue.add(url=welcome_url, params={
            'offset': 0
        })

You can see that this handler asks for welcome_url, which calls the defined handler taskqueue-welcome in routes and does the following:

class WelcomeHandler(BaseHandler):
    """
    Core Handler for sending users welcome message
    Use with TaskQueue
    """

    @taskqueue_method
    def post(self):
        # ... MY CODE FOR EMAILING USERS ....
        # ... YOUR CODE SHOULD GO IN HERE ...

Finally, in order for the cronjob handler to be called every 24 hours, in my cron.yaml file I've added the following, notice the route for cronjob-welcome :

- description: welcome messaging
  url: /cronjob-welcome/
  schedule: every day 13:00

Summarizing, my cron.yaml calls every day at 13:00 hours for the cronjob handler , which at the same time triggers a taskqueue handler , which dispatches a google appengine taskqueue solving my scheduled task .

Hope this helps.