為什么我可以訪問包含公共文件的https網頁，卻不能使用Python腳本下載它們？

Question

我的用戶名和密碼應該是什么？

import requests
import shutil

url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx.txt"    

#Note: It's https

r = requests.get(url, auth=('', ''), verify=False,stream=True)

r.raw.decode_content = True

with open("company.20170111.idx.txt", 'wb') as f:
    shutil.copyfileobj(r.raw, f) 

Answer 1

您要加載的網址應為：

https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170103.idx

您還缺少import requests ，並且服務器端不喜歡auth參數。

import shutil
import requests

url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx"

r = requests.get(url, verify=False, stream=True)

r.raw.decode_content = True

with open("company.20170111.idx.txt", 'wb') as f:
    shutil.copyfileobj(r.raw, f)

Answer 2

這很好。 我不知道為什么：

import urllib2
url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx"
r = urllib2.urlopen(url)
for l in r:
    print l