为什么我可以访问包含公共文件的https网页，却不能使用Python脚本下载它们？

Question

我的用户名和密码应该是什么？

import requests
import shutil

url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx.txt"    

#Note: It's https

r = requests.get(url, auth=('', ''), verify=False,stream=True)

r.raw.decode_content = True

with open("company.20170111.idx.txt", 'wb') as f:
    shutil.copyfileobj(r.raw, f) 

Answer 1

您要加载的网址应为：

https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170103.idx

您还缺少import requests ，并且服务器端不喜欢auth参数。

import shutil
import requests

url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx"

r = requests.get(url, verify=False, stream=True)

r.raw.decode_content = True

with open("company.20170111.idx.txt", 'wb') as f:
    shutil.copyfileobj(r.raw, f)

Answer 2

这很好。 我不知道为什么：

import urllib2
url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx"
r = urllib2.urlopen(url)
for l in r:
    print l