Python - 切片迭代的列表转换随着每个切片而增加

Question

I have a generator from read_sql and convert this generator to iterator using itertools.islice. So I convert this generator to iterator in slices using start and stop arguments. And this process runs in a loop to convert generator to iterator in 3 slices and convert the iterator to a list.

First time when it runs -> iterable_slice = list(it.islice(generator_df, 0, 3)) takes 2.99 seconds Second time when it runs -> iterable_slice = list(it.islice(generator_df, 4, 6)) takes 5.3 seconds and with every new loop or next set of slices, list conversion takes more time.

Why does this happen and where I am making a mistake? Thoughts please. Thank You.

#function to convert generator to slices

def gen_to_itr(generator_df,slice_start,slice_end):
    iterable_slice = list(it.islice(generator_df, slice_start,slice_end))

#main function 
slices = 3
slice_start = 0
slice_end = slices
flg_cnt = 0
while slice_end <= bcnt and flg_cnt <= 1:
    generator_df = pd.read_sql(query2, test_connection_forbankcv_connection, chunksize = 1800)
    first = time.perf_counter()
    iterable_slice = gen_to_itr(generator_df,slice_start,slice_end)
    end = time.perf_counter()
    print(f'Chunk list created in {round(end-first, 2)} second(s)')
    slice_start = slice_start+slices
    .....

Answer 1

it.islice() has to skip over the first slice_start elements of the generator when creating the new iterator. This takes time proportional to slice_start .

However, I find it hard to believe that skipping each element of a pandas series would take about 1 second. If the chunksize were smaller than the slice sizes, it might need to do another fetch from the database to get the next chunk. But as long as you're in the same chunk, I think it should have the same speed as iterating through a static series.