使用NA将数据读取到python中并按行计算均值

Question

I am reading in data from a csvfile and attempt to calculate the mean columnwise. While the number of columns is fixed, the number of rows isn't. Therefore I first read in the rows I need, make them a list and then form a numpy array of the list. But it doesn't work.

import csv
import numpy

Reading in (loops through every file and find matches, which will then be appended):

with open(input_file, mode='r') as f:
    reader = csv.reader(f, delimiter=';')
    for row in reader:
        pass
        # matchin algorithm omitted
        found_line = row
        del found_line[0] #remove first entry on name

input_file looks like

Weihnachtsmann;16;30.3125;0.00677830307346;0.000491988890358;0.2796728754;0.00371057513915;0.000667111407605;0.00177896375361
Tannenbaum;6;33.5;0.032918005099;0.00312809941211;0.308224811515;0.0124857679873;0.00644874360685;0.000667111407605
Heilier Klaus;1;NA;NA;NA;NA;NA;NA;NA

Then, I make a list out of the entries that match:

author_list.append(','.join(found_line))
author_array = numpy.array(author_list)

I am not creating the numpy array in the first place because I heard it's unpythonic and slow to append to numpy arrays.

print author_arry

yields

['1,NA,NA,NA,NA,NA,NA' '6;33.5;0.032918005099;0.00312809941211;0.308224811515;0.0124857679873;0.00644874360685;0.000667111407605' '16;30.3125;0.00677830307346;0.000491988890358;0.2796728754;0.00371057513915;0.000667111407605;0.00177896375361']

but I am not even sure if that's an array with the dimensions I want (should be exactly eight columns) or just one colum and three rows.

Afterwards, I have to convert the NA s that come from R into numpy's NaN (if I am correctly) and I don't know how to do that. I tried

[author_entry.replace('NA','nan') for author_entry in author_list]

but I get an error.

Answer 1

There are a number of different ways you could read in the data from the file using NumPy. Here's one way using np.genfromtxt . The names in the first column become NumPy nan values, as do any other non-float strings in your file:

>>> arr = np.genfromtxt(input_file, delimiter=';', dtype=np.float64)
>>> arr
array([[             nan,   1.60000000e+01,   3.03125000e+01,
          6.77830307e-03,   4.91988890e-04,   2.79672875e-01,
          3.71057514e-03,   6.67111408e-04,   1.77896375e-03],
       [             nan,   6.00000000e+00,   3.35000000e+01,
          3.29180051e-02,   3.12809941e-03,   3.08224812e-01,
          1.24857680e-02,   6.44874361e-03,   6.67111408e-04],
       [             nan,   1.00000000e+00,              nan,
                     nan,              nan,              nan,
                     nan,              nan,              nan]])

This is an array with 3 rows and 9 columns. To remove the first entry on each line, you could just slice and reassign with arr = arr[:, 1:] .

You can calculate the row-wise mean using np.nanmean (to ignore the nan values when calculating the mean):

>>> np.nanmean(arr, axis=1)
array([ 5.82569998,  4.98298407,  1.        ])