how to modify specific lines in a text file in a loop?

Question

I am using python 2.7 (OS-centos6)

I have a text file. For example, it consists of these lines:

0     4.064  16.786   7.016    0
1     5.520  14.733   5.719    0
2     5.904  17.898   5.222    0
3     3.113  18.613  18.453    0
4     3.629  16.760   5.118    0
            :
            :
            :
398   6.369  14.623    6.624    0
399   5.761  18.084    7.212    0
400   2.436  17.021   10.641    0

Last column contains all 0's initially. It is a flag basically. I want to modify this text file ie I want to make last column entry to be 1 (ie change the flag value to 1) whenever some criterion is matched for the specific line. For example,line numbers 3,20,250,400 satisfies this criterion. Then I want to make flag value (last column entries) of these specific lines to be 1 without changing other values present on these lines. Also, I want to do this in loop since I have many criteria. Therefore I have to go to top of the file everytime (ie for every criterion) and scan it from top to bottom; whenever criterion is satisfied, mark the specific line's flag as 1.

Important: I am using same modified file then to select only those lines (for further processing) whose flag value is NOT 1. For each iteration of the loop mentioned above, I want to read this modified file. This means, in short, I want to modify file (ie set flag to 1) for one criterion --> then read the modified file --> do processing --> then take next criterion --> set the flag to 1 for this criterion --> read the modified file--> and so on.

I would like to add this: The criterion to be satisfied takes into account two different lines everytime. eg If the difference between 2nd column entries for 3rd & 398th lines is less than 2.0, then set flag of 398th line to 1. ie difference 17.898 - 18.084 is less than 2.0, so flag of 398th line will be set to 1

Any help will be highly appreciated.

Answer 1

Okay. First you'll want to open the file and read each line.

I'd recommend reading the file line by line from one file and writing it to a second file.

with open("original.dat", "r"), open("new.dat", "w") as source, destination:
    for line in source:
        # split on spaces is the default:
        line_no, v1, v2, v3, flag = line.split()
        # just an example, do whatever checks you need to
        should_set_flag = some_computation(v1, v2, v3)
        if should_set_flag: 
            flag = 1
        destination.write("{} {} {} {} {}\n".format(line_no, v1, v2, v3, flag))

Perhaps I'm not understanding your requirement of reading the whole file each time you make one change. Given that the lines seem to be independent of one another I'm not sure why that's at all necessary.

Answer 2

    f=open("filename",'r')
    data=f.readlines()
    f.close()
    #remove file by using os.rm or using subprocess
    i=0
    while i < len(data):
          #do something
          #make changes to data list
    f=open("filename",'w')
    f.write(data)

That is the only way probably.Load data,remove old file,make changes,write to a new file.

Answer 3

why do you need to write the file back? it's only 400 lines, you can keep the lines in memory and to the processing one by one:

def is_criterion_1_fulfilled(row):
    return row[1]<4 # only an example

def process_1(row):
    print row # or do anything else with the line

def filter_and_process(iterator, criterion, process):
    for row in iterator:
        if criterion(row):
            continue
        process(row)
        yield row

def main():
    with open(filename, 'r') as inp:
        dataset = [map(float, line.split()) for line in inp]
    dataset = list(filter_and_process(dataset, is_criterion_1_fulfilled, process_1))
    dataset = list(filter_and_process(dataset, is_criterion_2_fulfilled, process_2))
    ....

if __name__ == '__main__':
    main()

Answer 4

# Imports
import re

# Functions
def check_data(record, records):
    # TODO Implement check operation
    return False

# Read input data
infile = "data.txt"
with open(infile, "r") as f:
    # Make a list of lists
    records = [re.split('\s+',record) for record in f.read().splitlines()]

# Process the data
for i, record in enumerate(records):
    # enumerate so as to refer to ith record if necessary,
    # but lineno anyway available in record[0]
    if check_data(record, records):
        record[4] = '1'


# Write modified data
outfile = "out%s" % infile
with open(outfile, "w") as f:
    for record in records:
        f.write('\t'.join(record)+'\n')