[英]python requests scrape with post and fill a form from drop-down menu
I am trying to scrape this page:我正在尝试抓取此页面:
http://www.barb.co.uk/viewing-data/weekly-top-30/ http://www.barb.co.uk/viewing-data/weekly-top-30/
I'm trying to fill the form, with this data:我正在尝试使用以下数据填写表格:
station=BBC2 (SD+HD)
year=2016
month=January
week=11 Jan - 17 Jan
So this is what I tried:所以这就是我尝试过的:
import requests
import json
url='http://www.barb.co.uk/viewing-data/weekly-top-30/'
payload= {
'station[]':['2'],
'period_year[]': ['2016'],
'period_month[]': ['1'],
'period_week[]': ['201601060117']
}
session=requests.session()
r=requests.post(url,data=payload)
with open ('html_res.html','w') as f:
f.write(str(r.content))
In html_res.html
I don't get the desired data.在html_res.html
我没有得到想要的数据。 I wonder what I'm doing wrong.我想知道我做错了什么。 Thanks.谢谢。
Use this code.使用此代码。
import requests
import json
headers = {
'Origin': 'http://www.barb.co.uk',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-US,en;q=0.8,he;q=0.6',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Accept': '*/*',
'Referer': 'http://www.barb.co.uk/viewing-data/weekly-top-30/',
'Connection': 'keep-alive',
}
data = [
('method', 'POST'),
('station[]', '839'),
('period[]', '201710060108'),
]
r = requests.post('http://barb-api.mediatel.co.uk//whats-new/weekly-top-30', headers=headers, data=data)
print(json.dumps(r.json(),indent=2))
You can use this website to convert curl to python code.您可以使用此网站将 curl 转换为 Python 代码。
You can achieve the same in several ways.您可以通过多种方式实现相同的目标。 Here is a concise one:这是一个简洁的:
import requests
payload={
'method':'POST','station[]':'2','period[]':'201601060117'
}
page = requests.post("http://barb-api.mediatel.co.uk//whats-new/weekly-top-30", params=payload)
for item in page.json():
data = item['data']['201601060117']
for name in data:
print(name['programme_name'],name['data_value'],name['28_days_total'])
Partial results:部分结果:
UNIVERSITY CHALLENGE (MON 2001) 3.13 3.14
TRUST ME I'M A DOCTOR (WED 2000) 3.01 3.16
ONLY CONNECT (MON 2031) 2.7 2.75
VICTORIAN BAKERS (TUE 2001) 2.69 2.73
SNOOKER (SUN 1901) 2.11 2.12
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.