I have a website and one input data which i need to fill in that website using Web Scraping.I am using post method but i am not getting expected output as i want. After running the code given below i am getting some data but that is different what i see response of the website using inspect. Input data is CP Number = 16803
Code:
import requests
from bs4 import BeautifulSoup
url = "https://www.icsi.in/student/Members/MemberSearch.aspx"
payload = {'dnn$ctr410$MemberSearch$txtCpNumber': 16803}
r = requests.post(url, params=payload)
soup = BeautifulSoup(r.content, 'html.parser')
print(soup)
Expected Output
After Post input data i will get name of user , Address etc
Change
r = requests.post(url, params=payload)
to
r = requests.post(url, data=payload)
params
are turned into query string. In your case you got url u'https://www.icsi.in/student/Members/MemberSearch.aspx?dnn%24ctr410%24MemberSearch%24txtCpNumber=16803'
Update
Here is a simple scraper written in js that uses puppeteer .
const puppeteer = require('puppeteer');
const URL = 'https://www.icsi.in/student/Members/MemberSearch.aspx';
(async () => {
const browser = await puppeteer.launch({
headless: false // for debugging
});
const page = await browser.newPage();
await page.goto(URL);
await page.focus('#dnn_ctr410_MemberSearch_txtFirstName');
page.keyboard.type('First Name');
await page.click('#dnn_ctr410_MemberSearch_btnSearch');
const html = await page.evaluate(() => document.body.innerHTML);
await browser.close();
})();
Run with the following commands:
npm i puppeteer
node scraper.js
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.