Python request.get持续超时约一分钟，然后继续正常工作

Question

I have been trying to resolve this issue for about a week now. Either I am missing something really obvious or the problem is on the server side of the API, or server is intentionally stalling me (I coded in python).

What I am trying to do:

1. I am trying to get financial data (bter market depth for all markets). The problem is the exchange service's api only supports getting data for only one market (total around 75-85, variable) So I decided that I am going to start a thread for each market
2. Every thread will handle one market, try to get data for that market, if successful return, if not add the market back to queue to be handled by a new thread later
3. Do this until all markets are covered, and repeat indefinitely to keep the data up-to-date

I coded this in python, using requests library. It works fine for a few iterations, but then server stops responding. To overcome this, I added timeout to requests.get. It times out, but the server does not respond to new queries either, for around 1 minute. Then everything works smoothly for a few iterations again, then things stall, and this repeats.

Here is the python code.

import requests, json
import thread, threading
from time import sleep, clock


#Get queue
conn = requests.get('http://data.bter.com/api/1/pairs/')
mainQueue = json.loads(conn.content)
conn.close()


#Variable globals
marketCount = 0
queue = mainQueue[:]


#Static globals
lock = threading.Lock()
completeSize = len(queue)


def getOrderData(marketid):
    global queue, marketCount

    try:
        data = requests.get(str('http://data.bter.com/api/1/depth/'
                                +marketid), timeout = 3)
    except:
        with lock:
            print "Timed out: %s" % marketid
            queue.append(marketid)
        return

    with lock:
        marketCount += 1
        data.close()
    return


while True:
    print "##################################"

    #Initialize data
    crT = clock()
    marketCount = 0
    queue = mainQueue[:]


    #Start retrieving all markets
    while marketCount != completeSize:
        while len(queue) == 0 and marketCount != completeSize:
            sleep(0.01)

        if marketCount != completeSize:
            marketid = queue.pop(0)
            thread.start_new_thread(getOrderData, (marketid,))

    #Print time spent
    print "Finished, total time:",clock()-crT
    sleep(1)

Here is the way program behaves during runtime.

Finished indicates that I got all the financial data once, and starting to update it again. As you can see, it seems that everything works fine, and then it starts stalling, timing out. All of a sudden, things start working normally again. I also noticed that after I close get connection with data.close(), a tcp connection with state TIME_WAIT remains in the tcp monitoring program for a long time. After a few iterations, there are TONS of them, just waiting with the TIME_WAIT state.

So, here is my question

1. Is it possible that, of all the TCP connections that remain in the TIME_WAIT state, are waiting for the server to send them some sort of signal to release them? If so, is it possible that server stops responding to me because I have too many (alive? active?) connections at once?
2. If not, why does that stalling period where all my get requests keep timing out happen? Is it because server might have a limit on queries per client per minute? and when I reach that, about after a minute, it magically starts working normally again,
3. I have TONS and TONS of TCP connections waiting in TIME_WAIT state and they keep accumulating. (I start around 80 connections per second. If 4 minutes is what it takes for connections to get released completely, that would be 19200 connections accumulated) How to resolve this, is it a problem at all?
4. I start so many threads. Is it a problem?
5. Getting all the data in a linear fashion, one market after another is not an option, too slow, data will be out of date. Any other ways I can keep the entire markets data up to date ?(max 3 sec age)
6. Anything else you wanna tell me?

I am aware that my code does not save the data, yet. I am just trying to get it first. Code is bad, but since I am testing a short snipped I did not bother with commenting (I changed it so many times trying to find a way)

Thanks in advance. I really hope I can overcome this.

Answer 1

You are executing requests very frequently, which is most likely not allowed by the server. The technique they are probably using is called throttling , have a look at http://www.django-rest-framework.org/api-guide/throttling , they have a nice explanation about it.