基于多列在具有不同形状的两个数据帧之间减去多列

Question

I'm looking at the following three datasets from JHU

https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv

https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv

https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv

Which are on the form

 'Province/State   'Country/Region    'Lat'    'Long'   '1/22/20'    '1/23/20' ...
       NaN               Italy          x        y          0            0

I want to calculate the number of active cases per province,country and day based on formula active = confirmed - (recovered+deahts)

Before the datasets had the same shape, so I could do the following

df_active = df_confirmed.copy()
df_active.loc[4:] = df_confirmed.loc[4:]-(df_recovered.loc[4:]+df_deaths.loc[4:])

Now they do not contain data on the same countries, and do not always have the same amount of date columns.

So I need to do the following

1) Determine what date columns all 3 DF have in common,

2) Where the province and country column match, do active = confirmed - (recovered+deahts)

For point 1) I can do the following

## append all shape[1] to list
df_shape_list.append(df_confirmed.shape[1])
...  
min_common_columns = min(df_shape_list)

So I need to subtract columns 4:min_common_columns , but how do I do that where province and country column match on all 3 DF's?

Answer 1

Consider melt to transform their wide data into long format then merge on location and date. Then run needed formula:

from functools import reduce
import pandas as pd

df_confirmed = pd.read_csv("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/"
                           "csse_covid_19_time_series/time_series_covid19_confirmed_global.csv")

df_deaths = pd.read_csv("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/"
                        "csse_covid_19_time_series/time_series_covid19_deaths_global.csv")

df_recovered = pd.read_csv("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/"
                           "csse_covid_19_time_series/time_series_covid19_recovered_global.csv")


# MELT EACH DF IN LIST COMPREHENSION
df_list = [df.melt(id_vars = ['Province/State', 'Country/Region', 'Lat', 'Long'],
                   var_name = 'Date', value_name = val) 
           for df, val in zip([df_confirmed, df_deaths, df_recovered], 
                              ['confirmed', 'deaths', 'recovered'])]

# CHAIN MERGE
df_long = reduce(lambda x,y: pd.merge(x, y, on=['Province/State', 'Country/Region', 'Lat', 'Long', 'Date']),
                 df_list)

# SIMPLE ARITHMETIC
df_long['active'] = df_long['confirmed'] - (df_long['recovered'] + df_long['deaths'])

Output (sorted by active descending)

df_long.sort_values(['active'], ascending=False).head(10)

#       Province/State Country/Region      Lat     Long     Date  confirmed  deaths  recovered  active
# 15229            NaN             US  37.0902 -95.7129  3/27/20     101657    1581        869   99207
# 14998            NaN             US  37.0902 -95.7129  3/26/20      83836    1209        681   81946
# 15141            NaN          Italy  43.0000  12.0000  3/27/20      86498    9134      10950   66414
# 14767            NaN             US  37.0902 -95.7129  3/25/20      65778     942        361   64475
# 14910            NaN          Italy  43.0000  12.0000  3/26/20      80589    8215      10361   62013
# 14679            NaN          Italy  43.0000  12.0000  3/25/20      74386    7503       9362   57521
# 14448            NaN          Italy  43.0000  12.0000  3/24/20      69176    6820       8326   54030
# 14536            NaN             US  37.0902 -95.7129  3/24/20      53740     706        348   52686
# 15205            NaN          Spain  40.0000  -4.0000  3/27/20      65719    5138       9357   51224
# 14217            NaN          Italy  43.0000  12.0000  3/23/20      63927    6077       7024   50826