read a tabular dataset from a text file in python

Question

I have many text files with the following format,

%header
%header
table
.
.
.
another table
.
.
.

If I didn't have the second table, I could use a simple commnad to read the file such as :

numpy.loadtxt(file_name, skiprows=2, dtype=float, usecols={0, 1})

is there an easy way to read the first table without having to read the files line by line, something like numpy.loadtxt

Answer 1

Use numpy.genfromtxt and set max_rows according to info from the header.

As an example, I created the following data file:

# nrows=10
# nrows=15
1
2
3
4
5
6
7
8
9
10
.
.
.
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
.
.
.

The the following oversimplified code could read the two tables from the file (of course you can enhance the code to meet your needs):

f = open('filename.txt')
# read header and find the number of rows to read for each table:
p = f.tell()
l = f.readline()
tabrows = []
while l.strip().startswith('#'):
    if 'nrows' in l:
        tabrows.append(int(l.split('=')[1]))
    p = f.tell()
    l = f.readline()
f.seek(p)
# read all tables assuming each table is followed by three lines with a dot:
import numpy as np
tables = []
skipdots = 0
ndotsafter = 3
for nrows in tabrows:
    tables.append(np.genfromtxt(f, skip_header=skipdots, max_rows=nrows))
    skipdots = ndotsafter
f.close()