熊猫read_html缺少一些表

Question

I am using pandas read_html to find all tables in a specific webpage; however, the process seems to be missing some of the tables.

Here is the webpage: https://www.uspto.gov/web/offices/ac/ido/oeip/taf/mclsstc/mcls1.htm

and here is my simple example:

import pandas as pd

df_list = pd.read_html("https://www.uspto.gov/web/offices/ac/ido/oeip/taf/mclsstc/mcls1.htm")

print(len(df_list))

This process finds 9 of the 17 tables. How can I use this method to find all the tables?

Note: if I try this on pages for other geographical areas, I have the same problem.

Answer 1

It seems that pd.read_html function can't find all table tags. I can suggest you to use BeautifulSoup and urllib2 packages for this task. You can install it via pip install <package_name> .

import urllib2
from bs4 import BeautifulSoup

html_text = urllib2.urlopen("https://www.uspto.gov/web/offices/ac/ido/oeip/taf/mclsstc/mcls1.htm")
bs_obj = BeautifulSoup(html_text)
tables = bs_obj.findAll('table')
dfs = list()
for table in tables:
    df = pd.read_html(str(table))[0]
    dfs.append(df)

In result, you'l have all tables (in DataFrame type) in dfs list.