[英]Saving images faster on disk
我开发了一个 Python 多线程程序,一个生产者线程以高 fps(约 75 fps)获取帧(512 x 640、16uint),以及两个消费者线程,一个用于实时可视化,另一个用于保存为 16 -bit tiff。 对于每个消费者,我使用不同的队列。
实时可视化效果很好,但在视频停止后保存需要很长时间(2 分钟的录制甚至需要 20 秒)。 为了保存,我使用了具有类似性能的 tifffile 库或 cv2。
更新信息
图片为灰度16位numpy arrays直接放入队列,不压缩,使用tifffile.imsave保存。 可视化的第二个队列实时正常工作,因此保存必须是最慢的过程。 我需要独立保存每张图片,保存 3D 暂时不是一个选项。 使用不同的线程进行保存可能会破坏我的采集顺序。
考虑到我需要以与记录相同的顺序保存它们,有什么方法可以在 Python 和/或操作系统(windows10)中加速该过程? 我有一个 SSD 970 EVO 磁盘驱动器
class VideoGet():
def __init__(self,input_dict,folder,record):
self.handle=input_dict['handle']
self.frame_t=input_dict['frametype']
self.frameSize= input_dict['frameSize']
# self.buffer = np.zeros(shape=(513,640), dtype=np.uint16)
self.record = record
self.save = False
self.done=False
self.counter=0
self.folder = folder
def displayer(self,q2):
while self.record is True:
if q2.empty() is True:
pass
else:
framedisplay = q2.get()
cv2.namedWindow("Video", cv2.WINDOW_NORMAL)
cv2.imshow("Video", framedisplay[:-1,:])
cv2.waitKey(1)
q2.task_done()
cv2.destroyAllWindows()
def consumer(self,q):
data=[]
while True:
if q.empty():
time.sleep(0.002)
pass
else:
frame_get = q.get()
if frame_get is None:
print('gone')
break
imsave(os.path.join(self.folder,(str(self.counter).zfill(5)+'.tiff')), frame_get[:-1,:])
if self.counter==0:
TS=1.0e-3 *struct.unpack('Q',(frame_get[-1,6:10]).tobytes())[0]
entr=[str(self.counter).zfill(5),str(round(1.0e3*(1.0e-3 *struct.unpack('Q',(frame_get[-1,6:10]).tobytes())[0]-TS)))]
data.append(entr)
self.counter=self.counter+1
q.task_done()
if data:
df = pd.DataFrame(data, columns = ['Picture', 'Timestamp'])
df.to_csv(os.path.join(self.folder,'timestamps.txt'), header=False, index=False, sep=' ')
print('done')
self.done=True
def producer(self,buffer,q,q2):
while self.record is True:
buffer=np.empty_like(buffer)
if camera.properties.get_frame(self.handle,self.frame_t,4,buffer,self.frameSize)==0:
frame=buffer
q2.put(frame)
if self.save is True:
q.put(frame)
del frame
print('None')
q.put(None)
def run(self,buffer,q,q2):
prod_thread=Thread(target=self.producer,args=(buffer,q,q2,))
display_thread=Thread(target=self.displayer,args=(q2,))
con_thread= Thread(target=self.consumer, args=(q,))
prod_thread.start()
display_thread.start()
con_thread.start()
当您不显示代码时,很难说出了什么问题。 此外,我不清楚为什么如果您有多个作家,收购顺序会改变。
这是一个脚本,用于生成与图像大小相同的合成帧,并将它们按顺序保存为 TIFF 文件。 它的速度随着更多编写器线程的增加而呈线性增长:
NFRAMES NWRITERS TIME(s)
1000 1 1.48
1000 2 0.78
1000 4 0.48
#!/usr/bin/env python3
import time
import numpy as np
import threading, queue
from tifffile import imsave
def writer(q):
print('[WRITER] Started')
total = 0
while True:
(frameNum, im) = q.get()
if frameNum < 0:
break
# Save as TIFF
imsave(f'frame-{frameNum}.tif', im)
total += 1
print(f'[WRITER] Complete: wrote {total} frames')
if __name__ == "__main__":
# Edit these to suit
NFRAMES = 1000
NWRITERS= 4
# Create dummy image of correct size
h, w = 640, 512
im = np.random.randint(0, 65536, (h,w), dtype=np.uint16)
# Create a queue to pass frames to writer(s)
q = queue.Queue(16)
print('[MAIN] Started')
start = time.time()
# Create and start writer thread(s)
threads = []
for _ in range(NWRITERS):
t = threading.Thread(target=writer, args=(q,))
t.start()
threads.append(t)
# Generate a large number of frames to store
for frameNum in range(NFRAMES):
# Put a tuple of frameNum and image in queue
q.put((frameNum, im))
# Sentinel to tell each writer to exit
for _ in range(NWRITERS):
q.put((-1,-1))
# Wait for our writer thread(s) to exit
for thread in threads:
thread.join()
elapsed = time.time() - start;
print(f'[MAIN] Complete: {NFRAMES} frames, with {NWRITERS} writers in {elapsed} seconds')
样品 Output
[MAIN] Started
[WRITER] Started
[WRITER] Started
[WRITER] Started
[WRITER] Started
[WRITER] Complete: wrote 250 frames
[WRITER] Complete: wrote 250 frames
[WRITER] Complete: wrote 250 frames
[WRITER] Complete: wrote 250 frames
[MAIN] Complete: 1000 frames, with 4 writers in 0.4869719505310059 seconds
我注意到的一件事是,如果将tifffile.imsave()
替换为:
np.save(f'frame-{frameNum}.npy', im)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.