有什么方法可以通过 python 中的 url 的自定义查询下载数据？

Question

I want to download the data from USDA site with custom queries. So instead of manually selecting queries in the website, I am thinking about how should I do this handier in python. To do so, I used request , http to access the url and read the content, it is not intuitive for me how should I pass the queries then make a selection and download the data as csv . Does anyone knows of doing this easily in python? Is there any workaround we could download the data from url with specific queries? Any idea?

this is my current attempt

here is the url that I am going to select data with custom queries.

import io
import requests
import pandas as pd

url="https://www.marketnews.usda.gov/mnp/ls-report-retail?&repType=summary&portal=ls&category=Retail&species=BEEF&startIndex=1"
s=requests.get(url).content
df=pd.read_csv(io.StringIO(s.decode('utf-8')))

so before reading the requested json in pandas , I need to pass following queries for correct data selection:

Category = "Retail"
Report Type = "Item"
Species = "Beef"
Region(s) = "National"
Start Dates = "2020-01-01"
End Date = "2021-02-08"

it is not intuitive for me how should I pass the queries with requested json then download the filtered data as csv . Is there any efficient way of doing this in python? Any thoughts? Thanks

Answer 1

A few details

• simplest format is text rather that HTML. Got URL from HTML page for text download
• requests(params=) is a dict . Built it up and passed, no need to deal with building complete URL string
• clearly text is space delimited, found minimum of double space

import io
import requests
import pandas as pd

url="https://www.marketnews.usda.gov/mnp/ls-report-retail"
p = {"repType":"summary","species":"BEEF","portal":"ls","category":"Retail","format":"text"}
r = requests.get(url, params=p)
df = pd.read_csv(io.StringIO(r.text), sep="\s\s+", engine="python")

	Date	Region	Feature Rate	Outlets	Special Rate	Activity Index
0	02/05/2021	NATIONAL	69.40%	29,200	20.10%	81,650
1	02/05/2021	NORTHEAST	75.00%	5,500	3.80%	17,520
2	02/05/2021	SOUTHEAST	70.10%	7,400	28.00%	23,980
3	02/05/2021	MIDWEST	75.10%	6,100	19.90%	17,430
4	02/05/2021	SOUTH CENTRAL	57.90%	4,900	26.40%	9,720
5	02/05/2021	NORTHWEST	77.50%	1,300	2.50%	3,150
6	02/05/2021	SOUTHWEST	63.20%	3,800	27.50%	9,360
7	02/05/2021	ALASKA	87.00%	200	.00%	290
8	02/05/2021	HAWAII	46.70%	100	.00%	230

Answer 2

Just format the query data in the url - it's actually a REST API:

To add more query data, as @mullinscr said, you can change the values on the left and press submit, then see the query name in the URL (for example, start date is called repDate ).

If you hover on the Download as XML link, you will also discover you can specify the download format using format=<format_name> . Parsing the tabular data in XML using pandas might be easier, so I would append format=xml at the end as well.

category = "Retail"
report_type = "Item"
species = "BEEF"
regions = "NATIONAL"
start_date = "01-01-2019"
end_date = "01-01-2021"

# the website changes "-" to "%2F"
start_date = start_date.replace("-", "%2F")
end_date = end_date.replace("-", "%2F")

url = f"https://www.marketnews.usda.gov/mnp/ls-report-retail?runReport=true&portal=ls&startIndex=1&category={category}&repType={report_type}&species={species}&region={regions}&repDate={start_date}&endDate={end_date}&compareLy=No&format=xml"

# parse with pandas, etc...