tqdm progress for processing sequence in chunks

Question

I am processing a sequence in chunks, where the last chunk may be shorter, and would like to show progress bar showing the number of items. The straightforward approach is

import tqdm, math
total=567
chunkSize=100
# each pass process items i0…max(i0+chunkSize,total)
for i0 in tqdm.tqdm(range(0,total,chunkSize)): pass

resulting in showing the number of chunks, not of the items, of course:

100%|█████████████████████████████████| 6/6 [00:00<00:00, 75121.86it/s]

Somewhat better options are

for i0 in tqdm.tqdm(range(0,total,chunkSize),unit_scale=chunkSize,total=total/chunkSize): pass
for i0 in tqdm.tqdm(range(0,total,chunkSize),unit_scale=float(chunkSize),total=total/chunkSize): pass
for i0 in tqdm.tqdm(range(0,total,chunkSize),unit_scale=chunkSize,total=math.ceil(total/chunkSize)): pass

which respectively give:

106%|██████████████████████████████████| 600.0/567.0 [00:00<00:00, 6006163.25it/s]
106%|██████████████████████████████████| 600/567.0 [00:00<00:00, 5264816.74it/s]
100%|██████████████████████████████████| 600/600 [00:00<00:00, 4721542.96it/s]

where those going over 100% show understandably

tqdm/std.py:533: TqdmWarning: clamping frac to range [0, 1]

So what I need is progress bar which will show the number of items (not chunks), correct percentages and will also correctly show the max value, not rounded to the chunk size. Ideas?

Answer 1

Variable chunk size? Could handle this manually with tqdm.tqdm.update :

import tqdm
total = 567
chunkSize = 100

with tqdm.tqdm(total=total) as pbar:
    # each pass process items i0…min(i0 + chunkSize, total)
    for i0 in range(0, total, chunkSize):
        end = min(i0 + chunkSize, total)
        do_something(start=i0, end=end)
        pbar.update(end - i0)