使用beautifulsoup从网页中的表格中提取文本信息

Question

I have a table from a webpage that I am attempting to extract the text data from.

A snippet of the HTML table looks as follows:

You can see the following table headings 'Effective Date', 'Type', 'Note' that I would like to extract text data from.

I have used the following code to attempt to extract the data:

content = driver.page_source
soup = BeautifulSoup(content)

for child in soup.find_all('table')[7].children:
    for td in child:
        print(td.text)

However, am met with a 'str' object has no attribute 'text'

Based on this HTML layout, what is the best way to find the right table - iterate through it ie the td's and select the text data appropriately? (Note that for the 'Note Headings' there are also 'br's' that may need to be iterated through). Thanks.

Answer 1

Thanks for that. After the content = driver.page_source , try and see if pandas will pull out the tables. If there are <table> tags, it will put them into a list of tables.

So:

import pandas as pd

content = driver.page_source
dfs = pd.read_html(content)

See if that returns your table in that list.