Web Scaping：熊猫 DataFrame.read_html(url_address) 返回一个空 DataFrame？

Question

我想 web 在此页面中刮取该表的信息，该页面还有许多其他页面。

我写了以下代码：

url = 'https://dbaasp.org/search?id.value=&name.value=&sequence.value=&sequence.option=full&sequenceLength.value=&complexity.value=&synthesisType.value=Nonribosomal&uniprot.value=&nTerminus.value=&cTerminus.value=&unusualAminoAcid.value=&intraChainBond.value=&interChainBond.value=&coordinationBond.value=&threeDStructure.value=&kingdom.value=&source.value=&hemolyticAndCytotoxicActivitie.value=on&synergy.value=&articleAuthor.value=&articleJournal.value=&articleYear.value=&articleVolume.value=&articlePages.value=&articleTitle.value='
pep_table = pd.read_html(url)

但是 output 是这样的：

pep_table
[Empty DataFrame
Columns: [ID, Name, N terminus, Sequence, C terminus, View]
Index: []]

我还尝试通过selenium webdriver 获取它：

chromedriver = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(chromedriver)  
driver.get(url)
 table = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table#DataTables_Table_0_info")))  
    tableRows = table.get_attribute("outerHTML")
    df = pd.read_html(tableRows)[0]

但它显示 selenium webdriver timeout 错误：

File "/home/es/anaconda3/envs/pyg-env/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

1. 我使用了错误的选择器吗？
2. 这个页面是搜索结果。 我需要添加更多选择器吗？
3. 如何解决这个问题？

Answer 1

你的桌子定位器是错误的。 我已经修改了。 最简单的方法，无需单击分页按钮并导航到 url。

您可以使用此 url，您必须更改偏移值。

url="https://dbaasp.org/search?id.value=&name.value=&sequence.value=&sequence.option=full&sequenceLength.value=&complexity.value=&synthesisType.value=Nonribosomal&uniprot.value=&nTerminus.value=&cTerminus.value=&unusualAminoAcid.value=&intraChainBond.value=&interChainBond.value=&coordinationBond.value=&threeDStructure.value=&kingdom.value=&source.value=&hemolyticAndCytotoxicActivitie.value=on&synergy.value=&articleAuthor.value=&articleJournal.value=&articleYear.value=&articleVolume.value=&articlePages.value=&articleTitle.value=&limit=30&offset={}"

您需要创建一个空的 dataframe 并与之连接。

使用 time.sleep() 等待，否则页面将移动得更快，无法捕获所有页面。

代码：

url="https://dbaasp.org/search?id.value=&name.value=&sequence.value=&sequence.option=full&sequenceLength.value=&complexity.value=&synthesisType.value=Nonribosomal&uniprot.value=&nTerminus.value=&cTerminus.value=&unusualAminoAcid.value=&intraChainBond.value=&interChainBond.value=&coordinationBond.value=&threeDStructure.value=&kingdom.value=&source.value=&hemolyticAndCytotoxicActivitie.value=on&synergy.value=&articleAuthor.value=&articleJournal.value=&articleYear.value=&articleVolume.value=&articlePages.value=&articleTitle.value=&limit=30&offset={}"
counter=0
df=pd.DataFrame()
while counter <150:
   driver.get(url.format(counter))
   time.sleep(2)
   table = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table#DataTables_Table_0")))  
   tableRows = table.get_attribute("outerHTML")
   df1 = pd.read_html(tableRows)[0]
   df = pd.concat([df,df1], ignore_index=True)
   counter=counter+30
print(df)

Output：

        ID              Name N terminus      Sequence C terminus  View
0     1688  Gramicidin S, GS        NaN    VXLfPVXLfP        NaN  View
1     3314      Gratisin, GR        NaN  VXLfPyVXLfPy        NaN  View
2     3316  Tyrocidine A, TA        NaN    fPFfNQYVXL        NaN  View
3     4876   Trichogin GA IV         C8   XGLXGGLXGIX        NaN  View
4     5374         Baceridin        NaN        WaXVlL        NaN  View
..     ...               ...        ...           ...        ...   ...
137  19210  Burkholdine-1215        NaN      xxGNSXXs        NaN  View
138  19212  Burkholdine-1213        NaN      xnGNSNXs        NaN  View
139  19548      Hirsutatin A        NaN        XTSXXF        NaN  View
140  19549      Hirsutatin B        NaN        XTSXXX        NaN  View
141  19554      Hirsutellide        NaN        XxIXxI        NaN  View

[142 rows x 6 columns]