how to iterate over Pandas data frame and update based on previous rows

Question

I have some code which I got to work but it's rather slow. I need to update a table of trades and quotes. The base table is like this:

+--------+-----------+----------+----------+--------+----------+
| Symbol | Timestamp | BidPrice | AskPrice | Price  | Quantity |
+--------+-----------+----------+----------+--------+----------+
| MSFT   | 9:00      |          |          | 46.98  |      140 |
| MSFT   | 9:01      |          |          | 46.99  |      100 |
| MSFT   | 9:02      |          |          | 47     |      400 |
| MSFT   | 9:03      |          |          | 47     |      100 |
| MSFT   | 9:04      | 46.87    | 46.99    |        |          |
| MSFT   | 9:05      |          |          | 46.89  |      100 |
| MSFT   | 9:06      |          |          | 46.95  |      600 |
| MSFT   | 9:07      | 46.91    | 46.99    |        |          |
| MSFT   | 9:08      | 46.91    | 46.97    |        |          |
| MSFT   | 9:09      |          |          | 46.935 |      100 |
| MSFT   | 9:10      | 46.89    | 46.96    |        |          |
| MSFT   | 9:11      |          |          | 46.93  |      100 |
| MSFT   | 9:12      |          |          | 46.91  |      100 |
+--------+-----------+----------+----------+--------+----------+

I need to set the bid and price for each trade (there is a Price but no bid/ask). So starting with bid = 46.8 and ask = 47, set the values, and when those values change, set new values. Like this:

+--------+-----------+----------+----------+--------+----------+
| Symbol | Timestamp | BidPrice | AskPrice | Price  | Quantity |
+--------+-----------+----------+----------+--------+----------+
| MSFT   | 9:00      | 46.8     | 47       | 46.98  |      140 |
| MSFT   | 9:01      | 46.8     | 47       | 46.99  |      100 |
| MSFT   | 9:02      | 46.8     | 47       | 47     |      400 |
| MSFT   | 9:03      | 46.8     | 47       | 47     |      100 |
| MSFT   | 9:04      | 46.87    | 46.99    |        |          |
| MSFT   | 9:05      | 46.87    | 46.99    | 46.89  |      100 |
| MSFT   | 9:06      | 46.87    | 46.99    | 46.95  |      600 |
| MSFT   | 9:07      | 46.91    | 46.99    |        |          |
| MSFT   | 9:08      | 46.91    | 46.97    |        |          |
| MSFT   | 9:09      | 46.91    | 46.97    | 46.935 |      100 |
| MSFT   | 9:10      | 46.89    | 46.96    |        |          |
| MSFT   | 9:11      | 46.89    | 46.96    | 46.93  |      100 |
| MSFT   | 9:12      | 46.89    | 46.96    | 46.91  |      100 |
+--------+-----------+----------+----------+--------+----------+

I worked this out iterating over rows, but for 112k rows, it takes 35 seconds.

for i, row in qts_trd.iterrows():
    if np.isnan(row['Price']):
        bid = row['BidPrice']
        ask = row['AskPrice']        
    if np.isnan(row['BidPrice']):
        qts_trd.at[i,'BidPrice'] = bid
        qts_trd.at[i,'AskPrice'] = ask

I know the basics of lambda functions, applying the same one to every row. I think it's quicker, but as you see it changes. Is there any more efficient/quicker way to do it?

This is Python 3.7 in Spyder.

Answer 1

Try pandas fillna() function using the method='ffill'

So:

qts_trd.BidPrice.fillna(method='ffill', inplace=True)
qts_trd.AskPrice.fillna(method='ffill', inplace=True)

In my experience it's very quick

Edit:

I just realised this wont fill your first values, the below code will insert a row at the top to fill from, and then delete it.

qts_trd.loc[-1] = ['', '', 46.8, 47, '', '']
qts_trd.index += 1
qts_trd.sort_index(inplace=True)
qts_trd.BidPrice.fillna(method='ffill', inplace=True)
qts_trd.AskPrice.fillna(method='ffill', inplace=True)
qts_trd.drop(0,0,inplace=True)
qts_trd.reset_index(drop=True, inplace=True)

Edit 2.0...thanks to @no_body 's comment:

qts_trd.BidPrice.fillna(method='ffill', inplace=True).fillna(46.8)
qts_trd.AskPrice.fillna(method='ffill', inplace=True).fillna(47)