Python read arrays delimited by white space from columns in a file

Question

I have a file with the following structure:

1
2
3

23
33
55

1
2
4

...

and so on. So I want to extract the data to a multidimensional array, ie [[1,2,3], [23,33,55], [1,2,4]...] . By now I have tried to use numpy.loadtxt() function, but I get a one dimensional array with all the numbers, and also tried this snippet:

data_tot = []
with open('file.txt', 'r') as infile:
     for line in infile:
         if line.rstrip() != '':
            data = []
            data.append(line.rstrip())
         else:
            data_tot.append(data)

where data_tot is the array I want to have, but I get something like data_tot = [[1], [23], [1] ...]

Any idea of how to solve this problem. Thanks in advance.

Answer 1

In the snippet you provided, every time when the line is not empty, data list is cleared.

data_buf = []
data_tot = []
with open('file.txt', 'r') as infile:
     for line in infile:
         if line.rstrip() == '':
            data_tot.append(data_buf[:])
            data_buf = []
         else:
            data_buf.append(line.rstrip())
if len(data_buf) > 0:
    data_tot.append(data_buf[:])

Note that data_buf[:] copies list object to avoid its modification in the next iteration. Also you should add the last buffer to the total list, if it is not followed by empty line.

Here is the full standalone sample code with StringIO instead of the file

import io

f = io.StringIO("""1
2
3

23
33
55

1
2
4
""")
data_buf = []
data_tot = []
with f as infile:
     for line in infile:
         if line.rstrip() == '':
            data_tot.append(data_buf[:])
            data_buf = []
         else:
            data_buf.append(line.rstrip())
data_tot.append(data_buf[:])

Answer 2

You can change the shape of your numpy array with reshape

#reshape the array to 3 by n 
np.loadtxt("file.txt").reshape(-1,3)

which with you data should give:

[[  1.   2.   3.]
 [ 23.  33.  55.]
 [  1.   2.   4.]
 ...]