读取混合类型（浮点和字符串）的文件

Question

I have a data file composed of N columns and M lines which I need to read storing each column into an array/list. The file is usually filled with numbers (floats) and in those cases I can just do:

import numpy as np
f_data = np.loadtxt('file.dat', unpack=True)

and the result is columns stored in f_data as sublists where its elements are floats , as expected.

Other times the file can have random strings scattered around (see an example of such file here ) In those cases I need to read it in the same way (ie: unpacked with each column stored in a list/array and all elements in it stored as float type) with all the strings converted to a default float (for example 99.999 )

In the example of the data file above, the column 5 would look like this after reading it:

f_data[5]
[2.049, 0.946, 0.942, 0.889, 99.999, 0.879, 0.989, 1.142, 1.062, 0.551, 1.233, 0.503]

Notice that all elements are of type float and the string that was found was converted to 99.999 and also stored as a float.

np.genfromtxt is able to read a file with mixed types but the result is that all the floats are stored as strings, which is not what I need.

How can I do this?

Answer 1

np.genfromtxt is the answer, but it's a little bit tricky to get it working just right.

Try:

np.genfromtxt("file.txt", dtype=float, filling_values=99.99)

This forces the type to a float, in every case. When numpy finds a value that isn't a float, it declares this value invalid, and thus missing. Filling values gives a default answer for what to do when the data are missing, in your case, 99.99.

And, to edit as requested, to store column-wise, add unpack=True, making the total answer,

np.genfromtxt("file.txt", dtype=float, filling_values=99.99, unpack=True)