[英]Python Web-scraping multiple page table to csv and DF for analysis
当我尝试对网页进行换景时,它仅将表格从第10页输出到csv文件,在此我想将每页的结果发送到该文件。 我知道我可能在这里犯了一个非常简单的错误。 任何人都可以在这里以正确的方式指导我,谢谢,我感谢您的投入。
import pandas as pd
import requests
from bs4 import BeautifulSoup
from tabulate import tabulate
#transactions over the last 17hrs
#Looping through page nimbers using url manipulation
#for i in range(1,100,1):
dfs = []
url = "https://etherscan.io/txs?p="
for index in range(1, 10, 1):
res = requests.get(url+str(index))
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0]
df = pd.read_html(str(table))
dfs.append(df)
#df[0].to_csv('Desktop/scrape.csv')
final_df[0] = pd.concat(dfs)
final_df[0].to_csv('Desktop/scrape.csv')
print( tabulate(df[0], headers='keys', tablefmt='psql'))
我收到以下类型错误。
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-10-c6a3a8b0cd1d> in <module>()
20 #df[0].to_csv('Desktop/scrape.csv')
21
---> 22 final_df[0] = pd.concat(dfs)
23 final_df[0].to_csv('Desktop/scrape.csv')
24 print( tabulate(df[0], headers='keys', tablefmt='psql'))
~/anaconda3/lib/python3.6/site-packages/pandas/core/reshape/concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
204 keys=keys, levels=levels, names=names,
205 verify_integrity=verify_integrity,
--> 206 copy=copy)
207 return op.get_result()
208
~/anaconda3/lib/python3.6/site-packages/pandas/core/reshape/concat.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy)
261 for obj in objs:
262 if not isinstance(obj, NDFrame):
--> 263 raise TypeError("cannot concatenate a non-NDFrame object")
264
265 # consolidate
TypeError: cannot concatenate a non-NDFrame object
您只是在代码中缺少一行。 pd.read_html
将返回DataFrame列表。 因此,只需在附加到dfs
之前先进行concat。
dfs = []
url = "https://etherscan.io/txs?p="
for index in range(1, 10):
res = requests.get(url+str(index), proxies=proxyDict)
soup = BeautifulSoup(res.content, 'lxml')
table = soup.find_all('table')[0]
df_list = pd.read_html(str(table))
df = pd.concat(df_list) # this line is what you're missing
dfs.append(df)
final_df = pd.concat(dfs)
final_df.to_csv('Desktop/scrape.csv')
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.