[英]h5py randomly unable to open object (component not found)
I'm trying to load hdf5 datasets into a pytorch training for loop. 我正在尝试将hdf5数据集加载到pytorch training for循环中。
Regardless of num_workers in dataloader, this randomly throws "KeyError: 'Unable to open object (component not found)' " (traceback below). 无论 dataloader中的num_workers如何,这都会随机抛出“KeyError:'无法打开对象(未找到组件)'”(下面的回溯)。
I'm able to start the training loop, but not able to get through 1/4 of one epoch without this error which happens for random 'datasets' (which are 2darrays each). 我能够开始训练循环,但是没有能够通过一个时期的1/4而没有发生随机“数据集”(每个是2个数组)的错误。 I'm able to separately load these arrays in the console using the regular f['group/subroup'][()]
so it doesn't appear like the hdf file is corrupted or that there's anything wrong with the datasets/array. 我可以使用常规f['group/subroup'][()]
在控制台中单独加载这些数组,因此它看起来不像hdf文件已损坏或数据集/数组有任何问题。
I've tried: 我试过了:
X = f[ID]
then X = X[()]
先调用父组,然后初始化数组,例如: X = f[ID]
则X = X[()]
Because this recurs with num_workers=0, I figure it's not a multithreading issue although the traceback seems to point to lines from /torch/utils/data/dataloader that prep the next batch. 因为这与num_workers = 0重复,我认为它不是多线程问题,尽管回溯似乎指向/ torch / utils / data / dataloader准备下一批的行。
I just can't figure out why h5py can't see the odd individual dataset, randomly. 我无法弄清楚为什么h5py无法随机看到奇怪的个体数据集。
IDs are strings to match hdf paths eg: ID = "ID_12345//Ep_-1//AN_67891011//ABC"
ID是与hdf路径匹配的字符串,例如: ID = "ID_12345//Ep_-1//AN_67891011//ABC"
excerpt from dataloader: 摘自dataloader:
def __getitem__(self, index):
ID = self.list_IDs[index]
# Start hdf file in read mode:
f = h5py.File(self.hdf_file, 'r', libver='latest', swmr=True)
X = f[ID][()]
X = X[:, :, np.newaxis] # torchvision 0.2.1 needs (H x W x C) for transforms
y = self.y_list[index]
if self.transform:
X = self.transform(X)
return ID, X, y
` `
Expected: training for loop 预期:循环训练
Actual: IDs / datasets / examples are loaded fine initially, then after between 20 and 200 steps... 实际:ID /数据集/示例最初加载正常,然后在20到200步之间加载......
Traceback (most recent call last): Traceback(最近一次调用最后一次):
File "Documents/BSSA-loc/mamdl/models/main_v3.py", line 287, in main() File "Documents/BSSA-loc/mamdl/models/main_v3.py", line 203, in main for i, (IDs, images, labels) in enumerate(train_loader): File "/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 615, in next batch = self.collate_fn([self.dataset[i] for i in indices]) File "/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 615, in batch = self.collate_fn([self.dataset[i] for i in indices]) File "/home/james/Documents/BSSA-loc/mamdl/src/data_loading/Data_loader_v3.py", line 59, in getitem X = f[ID][()] File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/h5py/_hl/group.py", line 262, in getitem oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py. 文件“Documents / BSSA-loc / mamdl / models / main_v3.py”,第287行,在main()文件“Documents / BSSA-loc / mamdl / models / main_v3.py”,第203行,主要为i,(枚举(train_loader)中的ID,图像,标签):文件“/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/torch/utils/data/dataloader.py”,第615行,in next batch = self.collate_fn([self.dataset [i] for i in indices])文件“/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/torch/utils/data/dataloader .py“,第615行,批处理= self.collate_fn([self.dataset [i] for i in indices])文件”/home/james/Documents/BSSA-loc/mamdl/src/data_loading/Data_loader_v3.py“ ,第59行,在getitem中 X = f [ID] [()]文件“h5py / _objects.pyx”,第54行,在h5py._objects.with_phil.wrapper文件“h5py / _objects.pyx”,第55行,在h5py中._objects.with_phil.wrapper文件“/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/h5py/_hl/group.py”,第262行,在getitem oid = h5o.open(self .id,self._e(name),lapl = self._lapl)文件“h5py / _objects.pyx”,第54行,在h5py中。 _objects.with_phil.wrapper _objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5o.pyx", line 190, in h5py.h5o.open 文件“h5py / _objects.pyx”,第55行,在h5py._objects.with_phil.wrapper文件“h5py / h5o.pyx”,第190行,在h5py.h5o.open中KeyError: 'Unable to open object (component not found)' KeyError:'无法打开对象(未找到组件)'
For the record, my best guess is that this was due a bug in my code for hdf construction, which was stopped and started multiple times in append mode. 为了记录,我最好的猜测是,这是由于我的hdf构造代码中的一个错误,它在追加模式下被停止并多次启动。 Some datasets appeared as though they were complete when queried f['group/subroup'][()]
but were not able to loaded with pytorch dataloader. 当查询f['group/subroup'][()]
但是无法加载pytorch dataloader时,一些数据集看起来好像是完整的。
Haven't had this issue since rebuilding hdf differently. 自从重建hdf以来,没有遇到过这个问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.