如何开始从python的顶部读取文件？

Question

I am trying to add dependencies from a list to a requirements.txt file depending on the platform the software is going to run. So I wrote the following code:

if platform.system() == 'Windows':
    # Add windows only requirements
    platform_specific_req = [req1, req2]
elif platform.system() == 'Linux':
    # Add linux only requirements
    platform_specific_req = [req3]

with open('requirements.txt', 'a+') as file_handler:
    for requirement in platform_specific_req:
        already_in_file = False
        # Make sure the requirement is not already in the file
        for line in file_handler.readlines():
            line = line.rstrip()  # remove '\n' at end of line
            if line == requirement:
                already_in_file = True
                break
        if not already_in_file:
            file_handler.write('{0}\n'.format(requirement))
    file_handler.close()

But what is happening with this code is that when the second requirement is going to be searched in the list of requirements already in the file, the for line in file_handler.readlines(): seems to be pointing to the last element of the list in the file so the new requirement is actually only compared to the last element in the list, and if it is not the same one it gets added. Obviously this is causing several elements to be duplicated in the list, since only the first requirement is being compared against all the elements in the list. How can I tell python to start comparing from the top of the file again?

Solution: I received many great responses, I learned a lot, thanks Guys. I ended up combining two solutions; the one from Antti Haapala and the one from Matthew Franglen into one. I am showing the final code here for reference:

# Append the extra requirements to the requirements.txt file
with open('requirements.txt', 'r') as file_in:
    reqs_in_file = set([line.rstrip() for line in file_in])
    missing_reqs = set(platform_specific_reqs).difference(reqs_in_file)

with open('requirements.txt', 'a') as file_out:
    for req in missing_reqs:
        file_out.write('{0}\n'.format(req))

Answer 1

You open the file handle before iterating over the existing requirement list. You then read the entire file handle for each requirement.

The file handle will finish after the first requirement because you have not reopened it. Reopening the file for each iteration would be very wasteful - read the file into a list and then use that inside the loops. Or do a set comparison!

file_content = set([line.rstrip() for line in file_handler])
only_in_platform = set(platform_specific_req).difference(file_content)

Answer 2

Do not try to read the file again for each requirement. While appending does work for this very use case, for modifications in general it is easier to just:

1. Read the content from the file into a list (preferably skipping empty lines)
2. Modify the list
3. Open the file again for writing and save the modified data.

So for example

with open('requirements.txt', 'r') as fin:
    requirements = [ i for i in (line.strip() for line in fin) if i ]

for req in platform_specific_req:
    if req not in requirements:
        requirements.append(req)

with open('requirements.txt', 'w') as fout:
    for req in requirements:
        fout.write('{0}\n'.format(req))
        # or print(req, file=fout)

Answer 3

The answer to your explicit question: file_handler.seek(0) will seek it back to the beginning of the file.

Some neat improvements:

You can use the file handler itself as an iterator instead of calling the readlines() method.

If your file is too large to read entirely in to memory, then iterating over the lines in the file directly is fine - but you should change how you're doing it. As is, you're iterating over the entire file for each requirement, but IO is costly. You should probably iterate over the lines, and for each line check if it's one of the requirements. Like so:

with open('requirements.txt', 'a+') as file_handler:
   for line in file_handler:
      line = line.rstrip()
      if line in platform_specific_req:
         platform_specific_req.remove(line)
   for req in platform_specific_req:
      file_handler.write('{0}\n'.format(req))

Answer 4

I know I'm answering a little late, but I would suggest doing it this way, opening it once, reading and appending in the same go. Note this should work on every platform regardless of your system:

import os

def ensure_in_file(lines, file_path):
    '''
    idempotent function to append lines to a file if they're not already there
    '''
    with open(file_path, 'r+U') as f: # r+U allows append, Universal Newline mode
        # set of all lines in the file, less newlines, and trailing spaces too.
        file_lines = set(l.rstrip() for l in f)
        # write lines not in the file, add the os line separator as you go
        f.writelines(l + os.linesep for l in set(lines).difference(file_lines))

You can test this

a_file = '/temp/temp/foo/bar' # insert your own file path here.

# with open(a_file, 'w') as f:  # ensure a blank file
    # pass 
ensure_in_file(['this', 'that'], a_file)
with open(a_file, 'rU') as f: 
    print f.read()

ensure_in_file(['this', 'that'], a_file)
with open(a_file, 'rU') as f:
    print f.read()

Each print statement should demonstrate that the file has each line once.