speed up 3d array fill from file using python

Question

I'm filling a 3d array using python, each array element represent a pixel.

The values that I need to insert to the array are stored in a very large .txt file (56 millions lines, formatted as follows - x,y,z,r,g,b)

right now I:

1. init 3d array with zeros.

2. read the file line by line.

3. for each line taking only the first 3 elements (x,y,z).

4. calculate the array location[i,j] from the x and y

5. if array[i,j] equals zero --> insert line read from file

6. else skip to next file

for 56 million lines it takes me about 160 seconds

how can speed this up using python? (gpu is available)

array = np.zeros((height, width), dtype=np.float32)

with open(point_cloud_file) as pc_file:
    while True:
        line = pc_file.readline()
        if not line:
            break
        nof_read_lines += 1

        new_line = line.strip()
        try:
            x, y, z, _, _, _ = new_line.split(',')
        except:
            nof_skipped_lines += 1
            continue

        # insert to array
        pixel_x = some calculation
        pixel_y = some calculation
        if 0 < pixel_x < width and 0 < pixel_y < height:
            if array[int(pixel_y), int(pixel_x), 0] == 0:
                array[int(pixel_y), int(pixel_x), :] = x, y, z
            else:
                nof_skipped_lines += 1  # pixel already filled with values

Answer 1

maybe readlines() be helpful in this situation this example read all of the lines at once and load the whole file into memory:

with open('foo') as f:
lines = f.readlines()
for line in lines:
    pass

but you are dealing with large text file so you can limit the buffer usage in each loop

with open('foo') as f:
    while True:
        lines = f.readlines(8192)
        if not lines:
            break
        for line in lines:
            pass

file.readlines([sizehint]) sizehint is bytes count according to documentation