.append in python is very slow, is there any way to improve it?

Question

i = 2
feature_vector_set = []
while i < 2405 - 2:
    j = 2
    while j < 1200 - 2:
       block = diff_image[i-2:i+3, j-2:j+3]
       feature = block.flatten()
       feature_vector_set.append(feature)
       j = j+1
    i = i+1

diff_image is int16 with shape(2405,1200) the whole loop takes 40mins to run and mainly caused by the following line:

feature_vector_set.append(feature)

Is there any alternative way to achieve the same result?

Answer 1

If you are going to append a lot of elements, a list is not the appropriate data structure for it.

Try using a deque. It's part of the python collections, it uses a doubly linked list internally.

from collections import deque

i = 2
feature_vector_set = deque()
while i < 2405 - 2:
   j = 2
   while j < 1200 - 2:
       block = diff_image[i-2:i+3, j-2:j+3]
       feature = block.flatten()
       feature_vector_set.append(feature)
       j = j+1
   i = I+1
feature_vector_list = list(feature_vector_set)

You can find the time complexities of common operations on python data types here

Deque documentation

Answer 2

You can try it

i = 2
feature_vector_set = set()
while i < 2405 - 2:
    j = 2
    while j < 1200 - 2:
       block = diff_image[i-2:i+3, j-2:j+3]
       feature = block.flatten()
       feature_vector_set.add(feature)
       j = j+1
    i = I+1
feature_vector_list = list(feature_vector_set)