将.txt文件分为Python的多个部分

Question

I'm a begginer in Python, and I have a question about file reading : I need to process info in a file to write it in another one. I know how to do that, but it's reaaally ressource-consuming for my computer, as the file is really big, but I know how it's formatted ! The file follows that format :

4 13
9 3 4 7
3 3 3 3
3 5 2 1

I won't explain what it is for, as it would take ages and would not be very useful, but the file is essentialy made of four lines like these, again and again. For now, I use this to read the file and convert it in a very long chain :

inputfile = open("input.txt", "r")
output = open("output.txt", "w")
Chain = inputfile.read()
Chain = Chain.split("\n")
Chained = ' '.join(Chain)
Chain = Chained.split(" ")
Chain = list(map(int, Chain))

Afterwards, I just treat it with "task IDs", but I feel like it's really not efficient. So do you know how I could divide the chain into multiple ones knowing how they are formatted? Thanks for reading !

Answer 1

How about:

res = []
with open('file', 'r') as f:
  for line in f:
    for num in line.split(' '):
      res.append(int(num))

Instead of reading the whole file into memory, you go line by line. Does this help?

If you need to go 4 lines at a time, just add an internal loop.

Regarding output, I'm assuming you want to do some computation on the input, so I wouldn't necessarily do this in the same loop. Either process the input once reading is done, or instead of using a list, use a queue and have another thread read from the queue while this thread is writing to it.

Perhaps the utility of a list comprehension will help a bit as well (I doubt this will make an impact):

res = []
with open('file', 'r') as f:
  for line in f:
    res.append( int(num) for num in line.split() )

Answer 2

hmm there's some method to write to a file without reading it i believe

Add text to end of line without loading file

https://docs.python.org/2.7/library/functions.html#print

from __future__ import print_function
# if you are using python2.7
i = open("input","r")
f = open("output.txt","w")
a = "awesome"
for line in i:
    #iterate lines in file input
    line.strip()
    #this will remove the \n in the end of the string
    print(line,end=" ",file=f) 
    #this will write to file output with space at the end of it

this might help, i'm a newbie too, but with better google fu XD

Answer 3

Maybe do it line by line. This way it consumes less memory.

inputfile = open("input.txt", "r")
output = open("output.txt", "a")

while True:
    line = inputfile.readline()
    numbers = words.split(" ")
    integers = list(map(int, numbers))

    if not line: 
       break

There is probably a newline character \\n in the words. You should also replace that with an empty string.

Answer 4

If you don't wanna to consume memory (you can run of it if file is very large), you need to read lien by line.

with open('input.txt', 'w') as inputfile, open('"output.txt', 'w') as output:
    for line in inputfile:
        chain = line.split(" ")
        #do some calculations or what ever you need
        #and write those numbers to new file
        numbers = list(map(int, chain))
        for number in numbers
            output.write("%d " % number)