在for循环中追加数据帧

Question

If I have a pd dataframe with three columns: id , start_time , end_time , and I would like to transform it into a pd.df with two columns: id , time

eg from [001, 1, 3][002, 3, 4] to [001, 1][001, 2][001, 3][002, 3][002, 4]

Currently, I am using a for loop and append the dataframe in each iteration, but it's very slow. Is there any other method I can use to save time?

Answer 1

If start_time and end_time is timedelta use:

df = pd.DataFrame([['001', 1, 3],['002', 3, 4]], 
                  columns=['id','start_time','end_time'])
print (df)
    id  start_time  end_time
0  001           1         3
1  002           3         4

#stack columns
df1 = pd.melt(df, id_vars='id', value_name='time').drop('variable', axis=1)
#convert int to timedelta 
df1['time'] = pd.to_timedelta(df1.time, unit='s')
df1.set_index('time', inplace=True)
print (df1)
           id
time         
00:00:01  001
00:00:03  002
00:00:03  001
00:00:04  002

#groupby by id and resample by one second
print (df1.groupby('id')
          .resample('1S')
          .ffill()
          .reset_index(drop=True, level=0)
          .reset_index())

      time   id
0 00:00:01  001
1 00:00:02  001
2 00:00:03  001
3 00:00:03  002
4 00:00:04  002

If start_time and end_time is datetime use:

df = pd.DataFrame([['001', '2016-01-01', '2016-01-03'],
                   ['002', '2016-01-03', '2016-01-04']], 
                  columns=['id','start_time','end_time'])
print (df)
    id  start_time    end_time
0  001  2016-01-01  2016-01-03
1  002  2016-01-03  2016-01-04

df1 = pd.melt(df, id_vars='id', value_name='time').drop('variable', axis=1)
#convert to datetime
df1['time'] = pd.to_datetime(df1.time)
df1.set_index('time', inplace=True)
print (df1)
             id
time           
2016-01-01  001
2016-01-03  002
2016-01-03  001
2016-01-04  002

#groupby by id and resample by one day
print (df1.groupby('id')
          .resample('1D')
          .ffill()
          .reset_index(drop=True, level=0)
          .reset_index())

        time   id
0 2016-01-01  001
1 2016-01-02  001
2 2016-01-03  001
3 2016-01-03  002
4 2016-01-04  002

Answer 2

Here is my take on your question:

df.set_index('id', inplace=True)

reshaped = df.apply(lambda x: pd.Series(range(x['start time'], x['end time']+1)), axis=1).\
    stack().reset_index().drop('level_1', axis=1)
reshaped.columns = ['id', 'time']
reshaped

---

Test

Input:

import pandas as pd
from io import StringIO

data = StringIO("""id,start time,end time
001, 1, 3
002, 3, 4""")

df = pd.read_csv(data, dtype={'id':'object'})
df.set_index('id', inplace=True)
print("In\n", df)

reshaped = df.apply(lambda x: pd.Series(range(x['start time'], x['end time']+1)), axis=1).\
    stack().reset_index().drop('level_1', axis=1)
reshaped.columns = ['id', 'time']
print("Out\n", reshaped)

Output:

In
    start time  end time
id      
001 1           3
002 3           4

Out
    id  time
0   001 1
1   001 2
2   001 3
3   002 3
4   002 4