“IndexError: list index out of range” when reading file

Question

Just started learning Python and I'm struggling with this a little.

I'm opening a txt file that will be variable in length and I need to iterate over a user definable amount of lines at a time. When I get to the end of the file I receive the error in the subject field. I've also tried the readlines() function and a couple of variations on the "if" statement that causes the problem. I just can't seem to get the code to find EOF.

Hmm, as I write this, I'm thinking ... do I need to addlist "EOF" to the array and just look for that? Is that the best solution, to find a custom EOF?

My code snippet goes something like:

### variables defined outside of scapy PacketHandler ##
x = 0
B = 0
##########

with open('dict.txt') as f:
    lines = list(f)
    global x
    global B
    B = B + int(sys.argv[3])
        while x <= B:
           while y <= int(sys.argv[2]): 
               if lines[x] != "":
                   #...do stuff...
                   # Scapy send packet Dot11Elt(ID="SSID",info"%s" %               (lines[x].strip()) 
                   # ....more code...
           x = x 1

Answer 1

Try a for in loop. You have created your list, now iterate through it.

with open('dict.txt') as f:
    lines = list(f)
    for item in lines: #each item here is an item in the list you created
        print(item)

this way you go through each line of your text file and don't have to worry about where it ends.

edit:

you can do this as well!

with open('dict.txt') as f:
    for row in f:
        print(row)

Answer 2

Let's say you need to read X lines at a time, put it in a list and process it:

with open('dict.txt') as f:

    enoughLines = True  

    while enoughLines:
        lines = []

        for i in range(X):
            l = f.readline()
            if l != '':
                lines.append( l )
            else:
                enoughLines = False
                break

        if enoughLines:
            #Do what has to be done with the list “lines”
        else:
            break

#Do what needs to be done with the list “lines” that has less than X lines in it

Answer 3

The following function will return a generator that returns the next n lines in a file:

def iter_n(obj, n):
    iterator = iter(obj)
    while True:
        result = []
        try:
            while len(result) < n:
                result.append(next(iterator))
        except StopIteration:
            if len(result) == 0:
                raise
        yield result

Here is how you can use it:

>>> with open('test.txt') as f:
...     for three_lines in iter_n(f, 3): 
...         print three_lines
...
['first line\n', 'second line\n', 'third line\n']
['fourth line\n', 'fifth line\n', 'sixth line\n']
['seventh line\n']

Contents of test.txt:

first line
second line
third line
fourth line
fifth line
sixth line
seventh line

Note that, because the file does not have a multiple of 3 lines, the last value returned is not 3 lines, but just the rest of the file.

Because this solution uses a generator, it doesn't require that the full file be read into memory (into a list), but iterates over it as needed.

In fact, the above function can iterate over any iterable object, like lists, strings, etc:

>>> for three_numbers in iter_n([1, 2, 3, 4, 5, 6, 7], 3): 
...     print three_numbers
... 
[1, 2, 3]
[4, 5, 6]
[7]
>>> for three_chars in iter_n("1234567", 3): 
...     print three_chars
... 
['1', '2', '3']
['4', '5', '6']
['7']

Answer 4

If you want to get n lines in a list use itertools.islice yielding each list:

from itertools import islice
def yield_lists(f,n):
    with open(f) as f:
        for sli in iter(lambda : list(islice(f,n)),[]):
            yield sli

If you want to use loops, you don't need a while loop at all, you can use an inner loop in range n-1 calling next on the file object with a default value of an empty string, if we get an empty string break the loop if not just append and again yield each list:

def yield_lists(f,n):
    with open(f) as f:
        for line in f:
            temp = [line]
            for i in range(n-1):
                line = next(f,"")
                if not line:
                    break
                temp.append(line)
            yield temp