Get number from a line of a txt file and use it as input to extract lines

Question

the title might sound weird. So I'll do my best to explain the problem.

I have a txt file (generated by another software) that looks like:

 EVAPORATION AND TRANSIPIRATION TOTALS     PERIOD    1   STEP    1,   15 COLUMNS,   10 ROWS,  1 LAYERS       ELAPSED TIME 6.0000000E+00           DAYS

    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000        1.00000    


 EVAPORATION AND TRANSIPIRATION TOTALS     PERIOD    1   STEP    2,   15 COLUMNS,   10 ROWS,  1 LAYERS       ELAPSED TIME 1.2000000E+01           DAYS

    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    10.00000        2.00000        2.00000        2.00000        2.00000        7.00000        2.00000        6.00000        6.00000        5.00000        2.00000        3.00000        4.00000        1.00000        0.00000


 EVAPORATION AND TRANSIPIRATION TOTALS     PERIOD    1   STEP    3,   15 COLUMNS,   10 ROWS,  1 LAYERS       ELAPSED TIME 1.8000000E+01           DAYS

    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    2.00000        4.00000        1.00000        2.00000        4.00000        1.00000        3.00000        4.00000        6.00000        8.00000        0.00000        1.00000        2.00000        2.00000        1.00000    
    2.00000        3.00000        2.00000        2.00000        2.00000        5.00000        0.00000        6.00000        1.00000        3.00000        2.00000        3.00000        4.00000        1.00000        0.00000      


 EVAPORATION AND TRANSIPIRATION TOTALS     PERIOD    1   STEP    4,   15 COLUMNS,   10 ROWS,  1 LAYERS       ELAPSED TIME 2.4000000E+01           DAYS

    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000        0.00000    
    3.00000        3.00000        3.00000        3.00000        3.00000        3.00000        3.00000        4.00000        4.00000        3.00000        2.00000        1.00000        0.00000        3.00000        1.00000    

As you can see it has 2 empty lines (also for the first block, not visible here) 1 line with some text, another empty line and a kind of matrix (and so on).

What I want to do is extract are the last n lines from each matrix (from each block) of some STEP (given as numbers from the user, for example from STEP 2 to STEP 3)

The n value is given by the ROWS number above each matrix. The number is always the same for each block (so in this example I'd like to extract the last 10 rows of the 2nd and 3rd matrix).

I need to put the line-blocks in a dictionary where the key in the STEP number (the changes for each block, in this example 2 and 3) and the values are the corresponding last 10 lines (so maybe np arrays as values).

Someone has some suggestions?

Thanks

Answer 1

You could use the .split() function to make this easier. It is explained quite well here .

I am sure this is not the most efficient way to do it (and any other suggestions will be welcome), but because you know the number of spaces between each value, you can use this to get a list of the values. There seem to be currently 8 spaces between each of the values in the code provided by you. Once you have opened your file an stored as myfile , for example, if you call

file = myfile.split("        ")

a list of items is stored in file. This list is basically the whole of myfile , but each time you see 8 spaces between two items, this becomes a new item in the file. So, each of your values will be stored as a separate item of this list, since there are 8 spaces between each of them. You will probably need to delete or ignore the first few items of the list as these will include parts of your title which have been split as well. This is a pretty bad explanation but hopefully you can figure it out.

You can use this to write a function to place each block of values in its separate list.

First use this to place each block into a list of lists (a nested list), where each item in the main list is a row of your block (or a column, depending on how you want to access them). The concept is covered here if you haven't met it before. So, they will end up like this:

step1 = [[0.00000, 0.00000, 0.00000...], [0.00000, 0.00000, 0.00000...], [0.00000, 0.00000, 0.00000...], ...]

In this case step1 will contain 20 lists (each one being a row of the STEP 1 block), and each list contains 15 values - one for each column of the STEP 1 block.

If you want access, say, row 3 of your table, you can just type step1[3] , and this will give you a list with all the items of row 3 in order, from left to right on your block shown above in your question (for the sake of explaining, I am calling the first (top) row 0 , the second row 1 , etc.

If you then want to find the item in row 4, column 3 (calling the first column column 0 ), you can find it as step1[4][3] .

Note that I have assigned the rows first, and then nested a the column as lists inside them, as this is slightly easier for extracting whole rows at once, as you wanted to.

Answer 2

The following assumes that the format of the file is specific (reasonable enough since it's generated by a program). Namely, for each block you two empty lines, a string line, an empty line, and 20 rows of data (24 rows overall).

Since the files are small relative to available memory, you can load the whole file in one go. Then with simple arithmetic you can figure out how many lines to skip to start reading from a specific block. Then, you can pass the next n lines as a generator to numpy.genfromtxt() which will efficiently and painlessly load them to an array. The only thing to note is that when you read the file with readlines() , you keep the newline character with you, which you want to remove when you pass the data to numpy ( line[:-1] ).

import numpy as np

def read_data(fname, rows, first_block, last_block):
    with open(fname) as f:
        data = f.readlines()
    blocks = {}
    for block in range(first_block, last_block+1):
        start = 24 * block + 24 - rows
        blocks[block] = np.genfromtxt((line[:-1] for line in data[start:start+rows]), autostrip=True)
    return blocks

You can run it like this

data = read_data('my_data.txt', 10, 2, 3)

and it will return a dictionary of float type arrays. In this case, you'll have data[2] and data[3] .