Python / Pandas：最快的设置和检索数据的方式，无需链接测定

Question

I am doing som routines that acces scalars and vectors from a pandas dataframe, and then sets the results after some calculations.

Initially I used the form df[var][index] to do this, but encountered problems with chained assaignment ( http://pandas.pydata.org/pandas-docs/dev/indexing.html%23indexing-view-versus-copy )

So I change it to use the df.loc[index,var]. Which solved the view/copy problem but it is very slow. For arrays I convert it to a pandas series and uses the builtin df.update(). I am now searching for the fastest/best way of doing this, without having to worry about chained assaingment. In the documentation they say that for example df.at[] is the quickest way to access scalars. Does anyone have any experience with this ? Or can point at some literature that can help ?

Thanks

Edit: Code looks like this, which I think is pretty standard.

    def set_var(self,name,periode,value):
        try:
            if navn.upper() not in self.data:
                 self.data[name.upper()]=num.NaN 
            self.data.loc[periode,name.upper()]=value
        except:
            print('Fail to set'+navn])

    def get_var(self,navn,periode):
    ''' Get value '''
    try:
        value=self.data.loc[periode,navn.upper()]


    def set_series(data, index):
        outputserie=pd.Series(data,index)
        self.data.update(outputserie) 


dataframe looks like this:
  SC0.data
  <class 'pandas.core.frame.DataFrame'>
  PeriodIndex: 148 entries, 1980Q1 to 2016Q4
  Columns: 3111 entries, CAP1 to CHH_DRD
  dtypes: float64(3106), int64(2), object(3)

edit2:

a df could look like

               var     var1
      2012Q4  0.462015  0.01585
      2013Q1  0.535161  0.01577
      2013Q2  0.735432  0.01401
      2013Q3  0.845959  0.01638
      2013Q4  0.776809  0.01657
      2014Q1  0.000000  0.01517
      2014Q2  0.000000  0.01593

and I basically want to perform two operations:

1) perhaps update var1 with the same scalar over all periodes

2) solve var in 2014Q1 as var,2013Q4 = var1,2013Q3/var2013Q4*var,2013Q4

This is done as part of a bigger model setup, which is read from a txt file. Since I doing loads of these calculations, the speed og setting and reading data matter

Answer 1

The example you gave above can be vectorized.

In [3]: df = DataFrame(dict(A = np.arange(10), B = np.arange(10)),index=pd.period_range('2012',freq='Q',periods=10))

In [4]: df
Out[4]: 
        A  B
2012Q1  0  0
2012Q2  1  1
2012Q3  2  2
2012Q4  3  3
2013Q1  4  4
2013Q2  5  5
2013Q3  6  6
2013Q4  7  7
2014Q1  8  8
2014Q2  9  9

Assign a scalar

In [5]: df['A'] = 5

In [6]: df
Out[6]: 
        A  B
2012Q1  5  0
2012Q2  5  1
2012Q3  5  2
2012Q4  5  3
2013Q1  5  4
2013Q2  5  5
2013Q3  5  6
2013Q4  5  7
2014Q1  5  8
2014Q2  5  9

Perform a shifted operation

In [8]: df['C'] = df['B'].shift()/df['B'].shift(2)

In [9]: df
Out[9]: 
        A  B         C
2012Q1  5  0       NaN
2012Q2  5  1       NaN
2012Q3  5  2       inf
2012Q4  5  3  2.000000
2013Q1  5  4  1.500000
2013Q2  5  5  1.333333
2013Q3  5  6  1.250000
2013Q4  5  7  1.200000
2014Q1  5  8  1.166667
2014Q2  5  9  1.142857

Using a vectorized assignment

In [10]: df.loc[df['B']>5,'D'] = 'foo'

In [11]: df
Out[11]: 
        A  B         C    D
2012Q1  5  0       NaN  NaN
2012Q2  5  1       NaN  NaN
2012Q3  5  2       inf  NaN
2012Q4  5  3  2.000000  NaN
2013Q1  5  4  1.500000  NaN
2013Q2  5  5  1.333333  NaN
2013Q3  5  6  1.250000  foo
2013Q4  5  7  1.200000  foo
2014Q1  5  8  1.166667  foo
2014Q2  5  9  1.142857  foo