如何使用Python请求发送表单数据

Question

I'm using requests to access this webpage and subsequently parsing and inspecting the HTML with Beautiful Soup.

This page allows a user to specify the number of days in the past for which results should be returned. This is accomplished via a form on the page:

When I submit the request in the browser with my selection of 365 days and examine the response, I find this form data was sent with the request:

Of note is the form datum "dnf_class_values[procurement_notice][_posted_date]: 365" as this is the only element that corresponds with my selection of 365 days.

When this request is returned in the browser, I get n results, where n is the maximum number possible given this is the largest time period possible. n is visible in the markup as <span class="lst-cnt"> .

I can't seem to duplicate the sending of that form data with requests. Here is the relevant portion of my code:

import requests
from bs4 import BeautifulSoup as bs

formData = {'dnf_class_values[procurement_notice][_posted_date]':'365'}
r = requests.post("https://www.fbo.gov/index?s=opportunity&mode=list&tab=list&tabmode=list&pp=20&pageID=1", data = formData)
s = bs(r.content)
s.find('span',{'class':'lst-cnt'})

This is returning the same number of results as when the form is submitted with the default value for number of days.

I've tried URL encoding the key in data , as well as using requests.get , and specifying params as opposed to data . Additionally, I've attempted to append the form data field as a query string parameter:

url...?s=opportunity&mode=list&tab=list&tabmode=list&pp=20&pageID=1&dnf_class_values%5Bprocurement_notice%5D%5B_posted_date%5D=365

What is the appropriate syntax for that request?

Answer 1

You cannot send only the sections you care about, you need to send everything . Duplicate the POST request that Chrome made exactly.

Note that some of the POSTed values may be CSRF tokens. The Base64-encoded strings are particularly likely ( dnf_opt_template , dnf_opt_template_dir , dnf_opt_subform_template and dnf_class_values[procurement_notice][notice_id] ), and should probably be pulled out of the HTML for the original page using BeautifulSoup. The rest can be hardcoded.

Otherwise, your original syntax was correct.