Why is that I can visit https webpage containing public files but I can't download them using Python script?

Question

What should be my username and password?

import requests
import shutil

url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx.txt"    

#Note: It's https

r = requests.get(url, auth=('', ''), verify=False,stream=True)

r.raw.decode_content = True

with open("company.20170111.idx.txt", 'wb') as f:
    shutil.copyfileobj(r.raw, f) 

Answer 1

The URL you're trying to load should be:

https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170103.idx

You're also missing import requests and the server-side didn't like the auth parameter.

import shutil
import requests

url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx"

r = requests.get(url, verify=False, stream=True)

r.raw.decode_content = True

with open("company.20170111.idx.txt", 'wb') as f:
    shutil.copyfileobj(r.raw, f)

Answer 2

This worked fine. I dont know why :

import urllib2
url = "https://www.sec.gov/Archives/edgar/daily-index/2017/QTR1/company.20170111.idx"
r = urllib2.urlopen(url)
for l in r:
    print l