I'm trying to scrape a website which needs login. The problem is that the url of the login form is something like this:
/customer/guest/index
The question is how may I send the login information to this post method?
PS: I'm trying to use method mentioned in this article.
UPDATE: Here is the code I'm using
from bs4 import BeautifulSoup
import requests
import csv
# Start the session
session = requests.Session()
# Create the payload
payload = {'CustomerLogin[email]':'<USERNAME>',
'CustomerLogin[password]':'<PASSWORD>',
'csrf_token': '<TOKEN>'
}
# Post the payload to the site to log in
s = session.post(" https://avangemail.net/customer/guest/index", data=payload)
print('Logging in...')
# Navigate to the next page and scrape the data
s = session.get('https://avangemail.net/customer/lists/all-subscribers')
print('Getting information...')
print(s.text)
soup = BeautifulSoup(s.text, 'html.parser')
table = soup.find('table')
tbody = table.find('tbody')
headers = [td.text for td in thead.select("tr td td")]
filename = 'avang-export'
with open(r'.'+filename+'.csv', "w") as f:
wr = csv.writer(f)
wr.writerows([[td.text for td in row.find_all("td")] for row in tbody.select("tr")])
data= {
'_username':'[YOUR_USERNAME]',
'_password':'[YOUR_PASSWORD]'
}
then pass it to post method
response = requests.post(
your_url,
data=data
)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.