在 dataframe PANDAS 中查找每个日期的最新日期时间

Question

I have a folder on my computer that contains ~8500.csv files that are all names of various stock tickers. Within each.csv file, there is a 'timestamp' and 'users_holding' column. I have the 'timestamp' column set up as a datetime index, as the entries in that column include hourly entries for each day ex/ 2019-12-01 01:50, 2020-01-01 02:55... 2020-01-01 01:45 etc. Each one of those timestamps has a corresponding integer representing the number of users holding at that time. I want to create a for loop that iterates through all of the.csv files and tallies up the total users holding across all.csv files for the latest time every day starting on February 1st, 2020 (2020-02-01) until the last day in the.csv file. The folder updates daily, so I can't really have an end date.

This is the for loop I have set up to establish each ticker as a dataframe:

path = 'C:\\Users\\N****\\Desktop\\r******\\t**\\p*********\\'
all_files = glob.glob(path + "/*.csv")

for filename in all_files:
    df = pd.read_csv(filename, header = 0, parse_dates = ['timestamp'], index_col='timestamp')

If anyone could show me how to write the for loop that finds the latest entry for each date and tallies up that number for each day, that would be amazing.

Thank you!

Answer 1

First, create a data frame with a Datetime index (in one-hour steps):

import numpy as np
import pandas as pd

idx = pd.date_range(start='2020-01-01', end='2020-01-31', freq='H')
data = np.arange(len(idx) * 3).reshape(len(idx), 3)
columns = ['ticker-1', 'ticker-2', 'ticker-3']
df = pd.DataFrame(data=data, index=idx, columns=columns)

print(df.head())

                     ticker-1  ticker-2  ticker-3
2020-01-01 00:00:00         0         1         2
2020-01-01 01:00:00         3         4         5
2020-01-01 02:00:00         6         7         8
2020-01-01 03:00:00         9        10        11
2020-01-01 04:00:00        12        13        14

Then, groupby the index (keep year-month-day), but drop hours-minutes-seconds). The aggregation function is .last()

result = (df.groupby(by=df.index.strftime('%Y-%m-%d'))
          [['ticker-1', 'ticker-2', 'ticker-3']]
          .last()
         )

print(result.head())

            ticker-1  ticker-2  ticker-3
2020-01-01        69        70        71
2020-01-02       141       142       143
2020-01-03       213       214       215
2020-01-04       285       286       287
2020-01-05       357       358       359