I read the data slice from a large file. A 400Mb picture took 4 seconds. It only takes 1 second for the disk to read this file. The program does very little computation. How to improve speed?
from opentile import OpenTile
import time
import traceback
import os
os.environ.setdefault('TURBOJPEG', 'C:/lib/')
try:
tiler = OpenTile.open('svs800.svs')
except:
traceback.print_exc()
s=tiler.get_level(0)
tile_size=str(s.tiled_size).split("x")
time1=time.time()
from multiprocessing.pool import ThreadPool
def get_data(s):
# This function reads a piece of binary data from a certain position of the image
# and then adds the header data
return tiler.get_tile(0,0,0, (s[0], s[1]))
pool = ThreadPool(5)
y = pool.map(get_data, [(i,j) for i in range(int(tile_size[0])) for j in range(int(tile_size[1]))])
print("tiles",len(y))
time2=time.time()
print(time2)
Simple sequential approach:
from opentile import OpenTile
import os
import time
def timer(func):
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
end = time.perf_counter()
print(f'{func.__name__} {end-start:.4f}s')
return result
return wrapper
os.environ.setdefault('TURBOJPEG', '/opt/libjpeg-turbo')
@timer
def open_svs(filename):
return OpenTile.open(filename)
@timer
def get_data(tiler, x, y):
return [tiler.get_tile(0, 0, 0, (x_, y_)) for x_ in range(x) for y_ in range(y)]
tiler = open_svs('18959.svs')
x, y = map(int, str(tiler.get_level(0).tiled_size).split('x'))
data = get_data(tiler, x, y)
assert len(data) == x * y
Output:
open_svs 0.0082s
get_data 0.5843s
Note:
x, y values for this file are 183, 114 respectively. The file size is 563,271,749 bytes
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.