在Python中从URL检索文件返回空白

Question

I'm working on a chess related project for which I have to download a very large quantity of files from ChessTempo .

When running the following code:

import urllib.request

url = "http://chesstempo.com/requests/download_game_pgn.php?gameids="

for i in range (3,500):
    urllib.request.urlretrieve(url + str(i),'Games/Game ' + str(i) + ".pgn")
    print("Downloaded file nº " + str(i))

I get the expected list of 500~ files but they are all blank except the second and third files, which have the correct data in them.

When I open the URLs by hand, it all works perfectly. What am I missing?

Answer 1

In fact, I can only download files 2 & 3, all others are empty...

Were you logged in while accessing those files "manually"? (Which I assume to be using a web browser).

If so, FYI an http request does not only consist of the URL, lots of other information is transfered. So if you are not getting the same information, you are almost certainly not making the same request.

In chrome you can see the requests you make within a page.

From Developer Tools go to Network > Select a name form the list > Request Headers ( See picture )

The most probable thing that you may be looking for are the cookies

Hope it helps.