Python/Pandas resampling Forex tick data for tick volume

Question

I have a bunch of tick data that I can successfully resample into time data using:

h5_file = pd.HDFStore(h5_path)
h5_file['fx_data'].groupby('Symbol')
ask = grouped['Ask'].resample('5Min', how='ohlc')
bid = grouped['Bid'].resample('5Min', how='ohlc')

But I would like to also return the tick volume. This should just be a count of how many rows make up each sample. How can this best be accomplished?

Also - when I choose to resample with smaller timeframes there are occasionally bars where the values are N/A because there was no price changes for that period. When this occurs I would like the previous close to be the value for OHLC on the current bar.

I've searched and found this code:

whatev.groupby('Symbol')closes = resampledData['close'].fillna(method='pad')
resampledData.apply(lambda x: x.fillna(closes)

I'm very new to Python and programming and don't understand lambas yet. Will this change only the close values or all the values I need changed. All help is greatly appreciated.

Answer 1

I've got a copy of part of your sample FX data (USD/EUR in May 2015) in hdf5, So I will use it here for illustration purpose.

import pandas as pd

Jian_h5 = '/media/Primary Disk/Jian_Python_Data_Storage.h5'
h5_file = pd.HDFStore(Jian_h5)  

fx_df = h5_file['fx_tick_data']
# I've only got USD/EUR in this dataset, but let's still do a groupby symbol
# and assume you have multiple symbols
grouped = fx_df.groupby('Symbol')

# calculate sub-group average bid and ask price, and also number of ticks
freq = '1min'
# an empty DataFrame
result = pd.DataFrame()
# bid/ask price: forward fill make sense
result['avg_bid'] = grouped['Bid'].resample(freq, how='mean').fillna(method='ffill')
result['avg_ask'] = grouped['Ask'].resample(freq, how='mean').fillna(method='ffill')
# tick count: NaN should be replaced by zero
result['tick_counts'] = grouped['Ask'].resample(freq, how='count').fillna(0)

Out[59]: 
                             avg_bid  avg_ask  tick_counts
Symbol  Date_time                                         
EUR/USD 2015-05-01 00:00:00   1.1210   1.1210           77
        2015-05-01 00:01:00   1.1209   1.1210          117
        2015-05-01 00:02:00   1.1209   1.1210           95
        2015-05-01 00:03:00   1.1210   1.1210           46
        2015-05-01 00:04:00   1.1211   1.1211          112
        2015-05-01 00:05:00   1.1213   1.1213          193
        2015-05-01 00:06:00   1.1214   1.1215           76
        2015-05-01 00:07:00   1.1216   1.1216          103
        2015-05-01 00:08:00   1.1216   1.1217          107
        2015-05-01 00:09:00   1.1217   1.1217           17
        2015-05-01 00:10:00   1.1216   1.1217           33
        2015-05-01 00:11:00   1.1218   1.1218           56
        2015-05-01 00:12:00   1.1217   1.1218           77
        2015-05-01 00:13:00   1.1215   1.1215           18
        2015-05-01 00:14:00   1.1215   1.1216           50
...                              ...      ...          ...
        2015-05-31 23:45:00   1.0959   1.0960           37
        2015-05-31 23:46:00   1.0959   1.0959           59
        2015-05-31 23:47:00   1.0958   1.0959           62
        2015-05-31 23:48:00   1.0956   1.0957           45
        2015-05-31 23:49:00   1.0955   1.0956           67
        2015-05-31 23:50:00   1.0955   1.0956           36
        2015-05-31 23:51:00   1.0955   1.0956           35
        2015-05-31 23:52:00   1.0956   1.0956           22
        2015-05-31 23:53:00   1.0956   1.0957           29
        2015-05-31 23:54:00   1.0957   1.0958           50
        2015-05-31 23:55:00   1.0956   1.0957           30
        2015-05-31 23:56:00   1.0957   1.0958            8
        2015-05-31 23:57:00   1.0957   1.0958           45
        2015-05-31 23:58:00   1.0957   1.0958           38
        2015-05-31 23:59:00   1.0958   1.0958           30

[44640 rows x 3 columns]