模拟新行，Python比SAS慢得多，如何加快速度？

Question

Here is the problem and it is for a model implementation task. Given I have some data. I need to simulate some new data,some of the variable values are dependent on the values of previous row and a random number r.

For example, say I have

AsOfDate  Var1  Var2    r
6/4/2018    A   0.3     0.2

Creating two rows, the output would be

AsOfDate    Var1    Var2    r
6/4/2018    A       0.3    0.2
6/5/2018    B       0.06   0.95
6/6/2018    A       0.057

The logic is, as of 6/4 r=0.2 less than Var2, then as of 6/5 Var1=B, Var2=0.3*0.2=0.06. As of 6/5, r=0.95 greater than Var2, then as of 6/6 Var1=A, Var=0.06*0.95=0.057.

I apologize if I confuse you. But I'm trying my best to describe this. I can't think of a way that I can do this without using a for loop. I ran the following simple SAS and Python code just to compare the speed. To my supprise, python/Pandas is much lower than SAS datastep. I'm no expert in Python, so I'm wondering if there is better way to do this and make it runs much faster. Thanks in advance for your help.

a=pd.DataFrame(data={'id':[1],'val':[2]})
tick=time.time()
n=0
b=pd.DataFrame()
for n in range(10000):
    a['id']=a['id']+1
    a['val']=a['val']+(n+1)
    b=pd.concat([b,a])
tock=time.time()
print(tock-tick)

time took: 7.54027533531189 sec

data test;
input id val;
datalines;
1 2
;
run;

%let _timer_start = %sysfunc(datetime());

data test(drop=i);
    set test;
    do i=1 to 10000;
        id=id+1;
        val=val+(i+1);
        output;
    end;
run;

data _null_;
  dur = datetime() - &_timer_start;
  put 30*'-' / ' TOTAL DURATION:' dur time13.2 / 30*'-';
run;

time took: 0.01 sec

Answer 1

The most straightforward answer is: because you have chosen the most inefficient way : )

Ie this code (not really optimised):

import time
import pandas as pd
tick=time.time()
n=0
a = {
    'id': 1,
    'val': 2,
}
data = []
for n in range(10000):
    a['id'] = a['id']+1
    a['val'] = a['val']+(n+1)
    data.append([a['id'], a['val']])
df = pd.DataFrame(data, columns=['id', 'val'])
tock=time.time()
print(tock-tick)

Does the same (unless i made some stupid mistake) few hundred times faster, probably not much slower than sas.

Answer 2

if you can, build your data outside pandas

from itertools import accumulate
a = list(range(2, 10002))
b = [2+i for i in accumulate(range(1,10001))]
df = pd.DataFrame(data={'id':a,'val':b})