是否有 function 来获取 pandas dataframe 时间序列上两个值之间的差异？

Question

I am messing around in the NYT covid dataset which has total covid cases for each county, per day.

I would like to find out the difference of cases between each day, so theoretically I could get the number of new cases per day instead of total cases. Taking a rolling mean, or resampling every 2 days using a mean/sum/etc all work just fine. It's just subtracting that is giving me such a headache.

Tried methods:

• df.resample('2d').diff()

  • 'DatetimeIndexResampler' object has no attribute 'diff'

• df.resample('1d').agg(np.subtract)

  • ufunc() missing 1 of 2required positional argument(s)

• df.rolling(2).diff()

  • 'Rolling' object has no attribute 'diff'

• df.rolling('2').agg(np.subtract)

  • ufunc() missing 1 of 2required positional argument(s)

Sample data:

pd.DataFrame(data={'state':['Alabama','Alabama','Alabama','Alabama','Alabama'],
               'date':[dt.date(2020,3,13),dt.date(2020,3,14),dt.date(2020,3,15),dt.date(2020,3,16),dt.date(2020,3,17)],
               'covid_cases':[1.2,2.0,2.9,3.6,3.9]
              })

Desired sample output:

pd.DataFrame(data={'state':['Alabama','Alabama','Alabama','Alabama','Alabama'],
               'date':[dt.date(2020,3,13),dt.date(2020,3,14),dt.date(2020,3,15),dt.date(2020,3,16),dt.date(2020,3,17)],
               'new_covid_cases':[np.nan,0.8,0.9,0.7,0.3]
              })

Recreate sample data from original NYT dataset:

df = pd.read_csv('https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv',parse_dates=['date'])
df.groupby(['state','date'])[['cases']].mean().reset_index()

Any help would be greatly appreciated. Would like to learn how to do this manually/via function rather than finding a "new cases" dataset as I will be working with timeseries a lot in the very near future.

Answer 1

Let's try this bit of complete code:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv')

df['date'] = pd.to_datetime(df['date'])

df_daily_state = df.groupby(['date','state'])['cases'].sum().unstack()

daily_new_cases_AL = df_daily_state.diff()['Alabama']

ax = daily_new_cases_AL.iloc[-30:].plot.bar(title='Last 30 days Alabama New Cases')

Output:

Details:

• Download the historical case records from NYTimes github using the raw URL
• Convert the dtype of the 'date' column to datetime dtype
• Groupby 'date' and 'state' columns sum 'cases' and unstack the state level of the index to get dates of rows and states for columns.
• Take the difference by columns and select only the Alabama column
• Plot the last 30 days

Answer 2

The diff function is correct, but if you look at your error message:

'DatetimeIndexResampler' object has no attribute 'diff'

in your first tried methods, it's because diff is a function available for DataFrames, not for Resamplers, so turn it back into a DataFrame by specifying how you want to resample it.

If you have the total number of COVID cases for each day and want to resample it to 2 days, you probably only want to keep the latest update out of the two days, in which case something like df.resample('2d').last().diff() should work.