Read data from a txt file into 2D array python

Question

I'm fairly new to python and I would like to know if I could get some assistance with the problem I'm trying to solve:

I would like to design a loop to iterate over every file in the directory and place the data into a 2D array for every file. I have a large directory of .txt files that contain 22 lines of 2 numbers per line.

An example of how the files contents will be organized is:

# Start of file_1.txt
1 2
3 4
5 6
7 8

# Start of file 2.txt
6 7
8 9
3 4
5 5

I would like to read the data separated by whitespace into the first two reference locations in the array(ie array = [x0][y0] ) , and at the newline, write the following data into the next location of the array (ie array=[x1][y2] ). I see a lot of people saying to use numpy , scipy , and other methods but that is confusing me further.

The output that I am seeking is:

[[1,2],[3,4],[5,6],[7,8], ...]

I'm bit stuck on how to iterate through the files in a directory and simultaneously place them into a 2D array. The code I have so far is:

import os
trainDir = 'training/'
testDir = 'testing/'
array2D = []

for filename in os.listdir(trainDir,testDir):
    if filename.endswith('.txt'):
        array2D.append(str(filename))

print(array2D)

At the moment the above code does not work for two directories, but it does for one. Any help would be appreciated.

Answer 1

You are defining your array2D wrong at the beginning, that's not valid Python syntax. The following code should work:

import os

d = 'HERE YOU WRITE YOUR DIRECTORY'

array2D = []

for filename in os.listdir(d):
    if not filename.endswith('.pts'):
        continue

    with open(filename, 'r') as f:
        for line in f.readlines():
            array2D.append(line.split(' '))

print(array2D)

Answer 2

I would advise running the python script in the same directory as the files you wish to read, for the sake of simplicity. Otherwise you will have to define a path to the directory that contains the files.

Also, I'm not sure if only you will be using this program, but it might be a good idea to define a try-except block around the FileIO portion of the code to prevent the program from crashing if it is unable to read from a file for any reason.

The below code reads all files in the directory that houses the python script and creates a 2D list of the files' contents (also, it explicitly rebuilds the 1D lists in a manner that ensures you have a list of integers rather than strings):

import os

output_2d_list = []
current_working_directory = os.path.abspath('.')

# Iterates over all files contained within the same folder as the python script.
for filename in os.listdir(current_working_directory):
    if filename.endswith('.pts'):

        # Ensuring that if something goes wrong during read, that the program exits gracefully.
        try:
            with open(filename, 'r') as current_file:

                # Reads each line of the file, and creates a 1d list of each point: e.g. [1,2].
                for line in current_file.readlines():
                    point = line.split(' ')
                    x = int(point[0])
                    y = int(point[1])
                    point_as_array = [x, y]
                    output_2d_list.append(point_as_array)
        except IOError:
            print "Something went wrong when attempting to read file."

# For testing purposes to see the output of the script.
# In production, you would be working with output_2d_list as a variable instead.
print output_2d_list