如何从 pandas 数据帧中的 web 页面读取所有 csv 文件？

Question

I'm trying to read all.csv files from https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports to a data frame.

My code so far:

url = 'https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports'
x = requests.get(url).text
filenames = re.findall('[\d]{1,2}-[\d]{1,2}-[\d]{4}.csv', x)
frame = pd.concat(pd.read_csv(url + y) for y in filenames) 

Maybe somebody can help:D

Answer 1

Change the URL to

url = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/'

and it should work. This gives you access to the raw csv file and not to a page the csv is on.

Edit: Just noticed that you need your old url to get the filenames:

url_raw = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/'
url = 'https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports'
x = requests.get(url).text
filenames = re.findall('[\d]{1,2}-[\d]{1,2}-[\d]{4}.csv', x)
frame = pd.concat(pd.read_csv(url_raw + y) for y in filenames)

Answer 2

Another option is to run with the following code

frame = pd.concat(pd.read_csv(f'{url}/{y}') for y in filenames)

Answer 3

Just as an additional note, you might not get the expected behaviour from pd.concat as the csv files in the url given are inconsistent column-wise (see examples below). You might want to rename or strip some of the columns before concat.

01-27-2020.csv
Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered
03-01-2020.csv
Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered,Latitude,Longitude
04-26-2020.csv
FIPS,Admin2,Province_State,Country_Region,Last_Update,Lat,Long_,Confirmed,Deaths,Recovered,Active,Combined_Key
06-28-2020.csv
FIPS,Admin2,Province_State,Country_Region,Last_Update,Lat,Long_,Confirmed,Deaths,Recovered,Active,Combined_Key,Incidence_Rate,Case-Fatality_Ratio