使用 Selenium 和 Python 提取表数据

Question

I am working on a project scraping a table off a web site. I am not able to give full code as this is a company specific site with a login. Basically, I am trying to fetch the records from webtable where nore than 100 records present but at a time only 12 records are displaying on screen.I tried to get records using scrollbar but it failed to do so.Kindly help me to get all records in excel sheet.Thanks in advance. I am adding my code below.

from lxml import etree
import urllib.request
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = "https//"
pd.read_html(requests.get(url,verify=False))
df = pd.read_html(url)

I am getting below error:

TypeError: cannot parse from 'Response'

Answer 1

You need to change

pd.read_html(requests.get(url,verify=False))

to

pd.read_html(requests.get(url,verify=False).content)