[英]Failed to find HDF5 dataset data - Single Label Regression Using Caffe and HDF5 data
我正在使用@shai的代碼來創建我的hdf5文件,可以在這里找到:
我的數據是灰度圖像(開始時為1000張圖像),標簽是一維浮點數
所以,我將他的代碼修改為:
import h5py, os
import caffe
import numpy as np
SIZE = 224
with open( 'train.txt', 'r' ) as T :
lines = T.readlines()
X = np.zeros( (len(lines), 1, SIZE, SIZE), dtype='f4' ) #Changed 3 to 1
y = np.zeros( (len(lines)), dtype='f4' ) #Removed the "1,"
for i,l in enumerate(lines):
sp = l.split(' ')
img = caffe.io.load_image( sp[0], color=False ) #Added , color=False
img = caffe.io.resize( img, (SIZE, SIZE, 1) ) #Changed 3 to 1
# you may apply other input transformations here...
X[i] = img
y[i] = float(sp[1])
with h5py.File('train.h5','w') as H:
H.create_dataset( 'X', data=X ) # note the name X given to the dataset!
H.create_dataset( 'y', data=y ) # note the name y given to the dataset!
with open('train_h5_list.txt','w') as L:
L.write( 'train.h5' ) # list all h5 files you are going to use
ValueError Traceback (most recent call last)
<ipython-input-19-8148f7b9e03d> in <module>()
13 img = caffe.io.resize( img, (SIZE, SIZE, 1) ) #Changed 3 to 1
14 # you may apply other input transformations here...
---> 15 X[i] = img
16 y[i] = float(sp[1])
ValueError: could not broadcast input array from shape (224,224,1) into shape (1,224,224)
所以我更改了第13行:
img = caffe.io.resize( img, (SIZE, SIZE, 1) )
至:
img = caffe.io.resize( img, (1, SIZE, SIZE) )
代碼運行正常。
為了訓練,我使用了這個solver.prototxt文件:
net: "MyCaffeTrain/train_test.prototxt"
# Note: 1 iteration = 1 forward pass over all the images in one batch
# Carry out a validation test every 500 training iterations.
test_interval: 500
# test_iter specifies how many forward passes the validation test should carry out
# a good number is num_val_imgs / batch_size (see batch_size in Data layer in phase TEST in train_test.prototxt)
test_iter: 100
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
# We want to initially move fast towards the local minimum and as we approach it, we want to move slower
# To this end, there are various learning rates policies available:
# fixed: always return base_lr.
# step: return base_lr * gamma ^ (floor(iter / step))
# exp: return base_lr * gamma ^ iter
# inv: return base_lr * (1 + gamma * iter) ^ (- power)
# multistep: similar to step but it allows non uniform steps defined by stepvalue
# poly: the effective learning rate follows a polynomial decay, to be zero by the max_iter: return base_lr (1 - iter/max_iter) ^ (power)
# sigmoid: the effective learning rate follows a sigmod decay: return base_lr * ( 1/(1 + exp(-gamma * (iter - stepsize))))
lr_policy: "inv"
gamma: 0.0001
power: 0.75
#stepsize: 10000 # Drop the learning rate in steps by a factor of gamma every stepsize iterations
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results, that is, every 5000 iterations it saves a snapshot of the weights
snapshot: 5000
snapshot_prefix: "MyCaffeTrain/lenet_multistep"
# solver mode: CPU or GPU
solver_mode: CPU
我的train_test.prototxt文件是:
name: "LeNet"
layer {
name: "mnist"
type: "HDF5Data"
top: "data"
top: "label"
hdf5_data_param {
source: "MyCaffeTrain/train_h5_list.txt"
batch_size: 1000
}
include: { phase: TRAIN }
}
layer {
name: "mnist"
type: "HDF5Data"
top: "data"
top: "label"
hdf5_data_param {
source: "MyCaffeTrain/test_h5_list.txt"
batch_size: 1000
}
include: { phase: TEST }
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 20
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2"
type: "Convolution"
bottom: "pool1"
top: "conv2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 50
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "pool2"
top: "ip1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 500
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "ip1"
top: "ip1"
}
layer {
name: "ip2"
type: "InnerProduct"
bottom: "ip1"
top: "ip2"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
inner_product_param {
num_output: 10
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "ip2"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}
layer {
name: "loss"
type: "EuclideanLoss"
bottom: "ip2"
bottom: "label"
top: "loss"
}
但是當我訓練時,我得到了這個錯誤:
I0914 13:59:33.198423 8251 layer_factory.hpp:74] Creating layer mnist
I0914 13:59:33.198452 8251 net.cpp:96] Creating Layer mnist
I0914 13:59:33.198467 8251 net.cpp:415] mnist -> data
I0914 13:59:33.198510 8251 net.cpp:415] mnist -> label
I0914 13:59:33.198532 8251 net.cpp:160] Setting up mnist
I0914 13:59:33.198549 8251 hdf5_data_layer.cpp:80] Loading list of HDF5 filenames from: MyCaffeTrain/train_h5_list.txt
I0914 13:59:33.198884 8251 hdf5_data_layer.cpp:94] Number of HDF5 files: 1
F0914 13:59:33.200848 8251 io.cpp:237] Check failed: H5LTfind_dataset(file_id, dataset_name_) Failed to find HDF5 dataset data
*** Check failure stack trace: ***
@ 0x7fcfa9fb05cd google::LogMessage::Fail()
@ 0x7fcfa9fb2433 google::LogMessage::SendToLog()
@ 0x7fcfa9fb015b google::LogMessage::Flush()
@ 0x7fcfa9fb2e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7fcfaa426b13 caffe::hdf5_load_nd_dataset_helper<>()
@ 0x7fcfaa423ec5 caffe::hdf5_load_nd_dataset<>()
@ 0x7fcfaa34bd3d caffe::HDF5DataLayer<>::LoadHDF5FileData()
@ 0x7fcfaa345ae6 caffe::HDF5DataLayer<>::LayerSetUp()
@ 0x7fcfaa3fdd75 caffe::Net<>::Init()
@ 0x7fcfaa4001ff caffe::Net<>::Net()
@ 0x7fcfaa40b935 caffe::Solver<>::InitTrainNet()
@ 0x7fcfaa40cd6e caffe::Solver<>::Init()
@ 0x7fcfaa40cf36 caffe::Solver<>::Solver()
@ 0x411980 caffe::GetSolver<>()
@ 0x4093a6 train()
@ 0x406de0 main
@ 0x7fcfa9049830 __libc_start_main
@ 0x407319 _start
@ (nil) (unknown)
我盡了最大努力,但仍然找不到這個錯誤背后的原因。 我創建的數據庫格式是否正確? 我在某處讀到了數據格式應該是:
N, C, H, W (No. of Data, Channels, Height, Width)
For my case: 1000,1,224,224
on checking X.shape I get the same result : 1000,1,224,224
我沒有得到我做錯的地方。 任何幫助,將不勝感激。 提前致謝。
我解決了以下更改代碼的問題:
H.create_dataset( 'data', data=X ) # note the name X given to the dataset! Replaced X by data
H.create_dataset( 'label', data=y ) # note the name y given to the dataset! Replaced y by label
錯誤消失了。
我仍然遇到EuclideanLoss圖層的問題,雖然我會查看它並在必要時發布另一個問題。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.