使用任务队列安排App Engine（Python）中多个提要的获取/解析

Question

Say I had over 10,000 feeds that I wanted to periodically fetch/parse. If the period were say 1h that would be 24x10000 = 240,000 fetches.

The current 10k limit of the labs Task Queue API would preclude one from setting up one task per fetch. How then would one do this?

Update: RE: Fetching nurls per task - Given the 30second timeout per request at some point this would hit a ceiling. Is there anyway to parallelize it so each task queue initiates a bunch of async parallel fetches each of which would take less than 30sec to finish but the lot together may take more than that.

Answer 1

Here's the asynchronous urlfetch API:

http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html

Set of a bunch of requests with a reasonable deadline (give yourself some headroom under your timeout, so that if one request times out you still have time to process the others). Then wait on each one in turn and process as they complete.

I haven't used this technique myself in GAE, so you're on your own finding any non-obvious gotchas. Sadly there doesn't seem to be a select() style call in the API to wait for the first of several requests to complete.

Answer 2

2 fetches per task? 3?

Answer 3

对提取进行分组，因此不必排队1个提取，而是将要进行10个提取的工作单元排队。