loading a file python

Question

I am trying to load a file using python, but I don't want to load it all at once. I am having an error and I don't know how to fix it.

This is the code:

import numpy as np
from itertools import islice

with open('C:/Users/jack/Desktop/folder/a.txt') as f:
    while True:
        next_n_lines = np.loadtxt(islice(f, 2))
        if (next_n_lines.any()==0):
            break
        a = next_n_lines[:, 0:2]
        b = next_n_lines[:, 2:4]
        print(a)
        print(b)

and this is the output:

[[ 1.  2.]
 [ 2.  3.]]
[[ 3.  4.]
 [ 4.  5.]]
[[ 3.  4.]
 [ 4.  5.]]
[[ 5.  6.]
 [ 6.  7.]]
[[ 5.  6.]
 [ 6.  7.]]
[[ 7.  8.]
 [ 8.  9.]]
Traceback (most recent call last):
  File "C:\Users\jack\Desktop\folder\a.py", line 9, in <module>
    a = next_n_lines[:, 0:2]
IndexError: too many indices for array

this is the file i am loading:

1   2   3   4
2   3   4   5
3   4   5   6
4   5   6   7
5   6   7   8
6   7   8   9
7   8   9   10

Answer 1

As I understood it, there is one line missing to your loading file, hence it is not possible to create the last two arrays.

Here is a method to load two lines at a time:

import numpy as np
from itertools import islice

with open(r'C:\\path\\to\\file\\a.txt') as f:
    while True:
        try :
            line1 = next(f)
            line2 = next(f)
        except StopIteration:
            break

        a1 = np.loadtxt(islice(line1.split('   '), 2))
        a2 = np.loadtxt(islice(line2.split('   '), 2))
        a = np.array([a1,a2])

        b1 = np.loadtxt(islice(line1.split('   '), 2, 4))
        b2 = np.loadtxt(islice(line2.split('   '), 2, 4))
        b = np.array([b1,b2])

        print(a)
        print(b)

f is an iterator, so you can call the next method to get the next value of the iterator. You can certainely generalize the code to reading n lines at a time.

The output is:

[[ 1.  2.]
 [ 2.  3.]]
[[ 3.  4.]
 [ 4.  5.]]
[[ 3.  4.]
 [ 4.  5.]]
[[ 5.  6.]
 [ 6.  7.]]
[[ 5.  6.]
 [ 6.  7.]]
[[ 7.  8.]
 [ 8.  9.]]

Answer 2

If you need an alternate and more standard way of efficiently accessing any line of a file you may check the linecache module provided by the standard library. This has the advantage of working even when you don't have access to NumPy .

Answer 3

First make sure you understand what the slice and loadtxt produce. Then you can worry about splitting the data:

In [150]: with open('stack46580159.txt') as f:
     ...:     while True:
     ...:         data = np.loadtxt(islice(f,2))
     ...:         print(data)
     ...:         if (data.any()==0):
     ...:             break
     ...:         
[[ 1.  2.  3.  4.]
 [ 2.  3.  4.  5.]]
[[ 3.  4.  5.  6.]
 [ 4.  5.  6.  7.]]
[[ 5.  6.  7.  8.]
 [ 6.  7.  8.  9.]]
[  7.   8.   9.  10.]
/usr/local/bin/ipython3:3: UserWarning: loadtxt: Empty input file: "<itertools.islice object at 0xab79bc84>"
  # -*- coding: utf-8 -*-
[]

You have an odd number of lines. So the last slice feeds loadtxt just one line (plus a warning). loadtxt returns a 1d array, not the 2d that you were expecting.

A simple fix is to make sure data is 2d before you slice it

In [155]: with open('stack46580159.txt') as f:
     ...:     while True:
     ...:         data = np.loadtxt(islice(f,2))
     ...:         data = np.atleast_2d(data)
     ...:         print(data)
     ...:         if (data.any()==0):
     ...:             break
     ...:         data[:,:2]
     ...:         
[[ 1.  2.  3.  4.]
 [ 2.  3.  4.  5.]]
[[ 3.  4.  5.  6.]
 [ 4.  5.  6.  7.]]
[[ 5.  6.  7.  8.]
 [ 6.  7.  8.  9.]]
[[  7.   8.   9.  10.]]

There are other ways of iterating through the file, or loading it all and then split. But make sure you understand what is going on at each step. Don't just assume.