Reading in repeated blocks of data using pandas and python

Question

I have a file with the following data:

       2008   1   1     ATMOS CO2 = 382. ppm
                                                   SOIL LAYER NO
                        1           1           2           3           4         TOT
       DEPTH(m)        0.01        0.10        0.33        0.64        0.81
 BD 33kpa(t/m3)        1.48        1.48        1.48        1.50        1.53
        SAND(%)        82.2        82.2        82.2        66.9        67.4
        SILT(%)         5.3         5.3         9.8        23.1        19.6
        CLAY(%)        12.5        12.5         8.0        10.0        13.0   
    WHSC(kg/ha)        525.       4729.       4480.       6119.       1114.      16968.
    WHPC(kg/ha)       1123.      10104.       9572.      13076.       2381.      36256.
     WOC(kg/ha)       1717.      15455.      14638.      19995.       3641.         55.



         2008  12  31     ATMOS CO2 = 382. ppm
                                                   SOIL LAYER NO
                        1           1           2           3           4         TOT
       DEPTH(m)        0.01        0.10        0.33        0.64        0.81
 BD 33kpa(t/m3)        1.48        1.48        1.48        1.50        1.53
        SAND(%)        81.4        81.4        81.4        67.7        67.4
        SILT(%)         6.5         6.5        10.3        22.3        19.6
        CLAY(%)        12.1        12.1         8.2        10.0        13.0
    WHSC(kg/ha)        499.       4559.       4291.       6017.       1117.      16483.
    WHPC(kg/ha)       1123.      10109.       9576.      13081.       2382.      36271.
     WOC(kg/ha)       1633.      14757.      13993.      19316.       3601.         53.

Each block begins with the date of the year eg 2008 1 1 denotes 1st January 2008 and 2008 12 31 denotes 31st Decemeber 2008.

In each block, there are values for several parameters eg DEPTH, SAND(%), WOC etc. I want to extract values for WOC for a user given year and day and month eg 2008 12 31, and for a specific column eg TOT. I can read the dataframe but not sure what is the best way to proceed after that:

df = pandas.read_csv('data.txt')

Answer 1

You have to process file line by line and then use StringIO as input of function read_csv .

import pandas as pd
import numpy as np
from StringIO import StringIO
pathToFile = 'test/file.txt'
f = open(pathToFile)
s = StringIO()
cur_atm = np.nan

for ln in f:
    #replace multiply spaces to one ;
    ln = ';'.join(ln.split())
    if('ppm' in ln):
        cur_atm = ln.split(';')
        #items of list cur_atm
        print cur_atm
        #get 2 item from back of list cur_atm
        cur_atm = cur_atm[-2]
        continue
    if (ln.startswith('20')) | (ln.startswith('19')):
        continue;
    #remove rows start with string SOIL and 1;1, remove empty rows
    if  (ln.startswith('SOIL')) | (ln.startswith('1;1')) | (ln == ''):
        continue;
    if  ln.startswith('BD;'):
        ln = ln.replace('BD;', 'BD ') 
        continue;        

    #print ln to StringIO s
    s.write(str(cur_atm) + ";" + ln + '\n')
s.seek(0)

# create new dataframe with desired column names
df = pd.read_csv(s, sep=";", index_col=[1], names=['ATM','','1','2','3','4','5', 'TOT'])

print df
#             ATM        1        2         3         4        5    TOT
#                                                                      
#DEPTH(m)     382     0.01      0.1      0.33      0.64     0.81    NaN
#SAND(%)      382    82.20     82.2     82.20     66.90    67.40    NaN
#SILT(%)      382     5.30      5.3      9.80     23.10    19.60    NaN
#CLAY(%)      382    12.50     12.5      8.00     10.00    13.00    NaN
#WHSC(kg/ha)  382   525.00   4729.0   4480.00   6119.00  1114.00  16968
#WHPC(kg/ha)  382  1123.00  10104.0   9572.00  13076.00  2381.00  36256
#WOC(kg/ha)   382  1717.00  15455.0  14638.00  19995.00  3641.00     55
#DEPTH(m)     382     0.01      0.1      0.33      0.64     0.81    NaN
#SAND(%)      382    81.40     81.4     81.40     67.70    67.40    NaN
#SILT(%)      382     6.50      6.5     10.30     22.30    19.60    NaN
#CLAY(%)      382    12.10     12.1      8.20     10.00    13.00    NaN
#WHSC(kg/ha)  382   499.00   4559.0   4291.00   6017.00  1117.00  16483
#WHPC(kg/ha)  382  1123.00  10109.0   9576.00  13081.00  2382.00  36271
#WOC(kg/ha)   382  1633.00  14757.0  13993.00  19316.00  3601.00     53

df = df.loc['WOC(kg/ha)', ['ATM', 'TOT']]
print df
#            ATM  TOT
#                    
#WOC(kg/ha)  382   55
#WOC(kg/ha)  382   53