reading txt file field after field

Question

Hello I have a txt file that is composed of N lines by 96 values i expected it was organised N*96 but not in fact number of lines is double and first line is 50 second 46 and so on

I wrote a code to re compose orginal line with 2 others reading line after line but i 'm asking is there is a way of reading each field in succession instead. each value is separated with space caractere.

for j in range (239) :
    L=[]
    lc1 = f.readline().split()
    lc2 = f.readline().split()

    for i in range(50) :
        L.append(lc1[i])

    for i in range(46) :
        L.append(lc2[i])

    table[j][:]=L   

f.close()  

Regards

Answer 1

Hmm... reading the code made more sense than reading your description, regarding the split line part. :)

Given the input structure of your text is known. Ie, 96 words per line, you can do the following instead:

ValuesPerLine=96
ResultsTable=[]
TableIndex=0
NewLine=[]

for line in f:
    for value in line.split():
        NewLine.append(value)
        if len(NewLine) == ValuesPerLine:
             ResultsTable[TableIndex][:]=NewLine
             NewLine=[]
             TableIndex=TableIndex+1

In this manner, even if the lines are later split differently, so long as the expectation is that you have 96 values per line in the result, this will generate that from your source.

The benefit of the "for line in f:" is that it effectively handles it like "readlines()" vs "readline()", so you get some performance improvement.

Edit:

Depending on the file type you are opening, it might be helpful to strip out any "\\n" from the line read from the file to avoid having one value in the set, every now and then, having an embedded "\\n".

Answer 2

I am not sure if I understood your problem, but here is a way to gather your two loops into one statement :

for j in range (239) :
    lc1 = f.readline().split()
    lc2 = f.readline().split()

    L = lc1[:50] + lc2[:46]

    table[j][:]=L   

f.close()  

I can't help trying to recycle the nice trick proposed by @jon-clements in Copy the last three lines of a text file in python?

q = collections.deque(2) # define max size of deque 
table = []
with open("test.txt") as f:
    for line in f :
        q.append(line)
        if len(q) != 2 or len(q[0]) != 50 :
            continue
        table.append(q[0] + q[1])

Works with python 2.7 or higher

Answer 3

Assuming the file can easily be loaded into memory, I would do something like:

all_fields = f.read().split():
for i in range(0, len(all_fields), 96):
    fields = all_fields[i:i+96]
    # process fields