如何从熊猫数据框中的网页读取所有csv文件

Question

I would like to load all the csv files from the following webpage to a data frame

https://s3.amazonaws.com/tripdata/index.html

I tried with glob as for loading all files from a directory without success:

import glob
path ='https://s3.amazonaws.com/tripdata' # use your path
allFiles = glob.glob(path + "/*citibike-tripdata.csv.zip")
frame = pd.DataFrame()
list_ = []
for file_ in allFiles:
    df = pd.read_csv(file_, index_col=None, header=0)
    list_.append(df)
frame = pd.concat(list_)

Any suggestions?

Answer 1

The module glob is used for finding pathnames matching patterns on the same system as Python is running in, and there is no way for it to index arbitrary file hosting web servers (which isn't even possible a priori). In your case, since https://s3.amazonaws.com/tripdata/ provides the desired index, you could parse that to get the relevant files:

import re
import requests

url = 'https://s3.amazonaws.com/tripdata/'
t = requests.get(url).text
filenames = re.findall('[^>]+citibike-tripdata\.csv\.zip', t)
frame = pd.concat(pd.read_csv(url + f) for f in filenames)