[caffe]: check fails: Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000)

Question

I have the train and label data as data.mat. (I have 200 training data with 6000 features and labels are (-1, +1) that have saved in data.mat).

I am trying to convert my data in hdf5 and run Caffe using:

load data.mat
hdf5write('my_data.h5', '/new_train_x', single( reshape(new_train_x,[200, 6000, 1, 1]) ) );
hdf5write('my_data.h5', '/label_train', single( reshape(label_train,[200, 1, 1, 1]) ), 'WriteMode', 'append' );

And my layer.prototxt (just data layer) is:

layer {
  type: "HDF5Data"
  name: "data"
  top: "new_train_x"     # note: same name as in HDF5
  top: "label_train"     # 
  hdf5_data_param {
    source: "/path/to/list/file.txt"
    batch_size: 20
  }
  include { phase: TRAIN }
}

but, i have an error: ( Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000))

 I1222 17:02:48.915861 3941 layer_factory.hpp:76] Creating layer data I1222 17:02:48.915871 3941 net.cpp:110] Creating Layer data I1222 17:02:48.915877 3941 net.cpp:433] data -> new_train_x I1222 17:02:48.915890 3941 net.cpp:433] data -> label_train I1222 17:02:48.915900 3941 hdf5_data_layer.cpp:81] Loading list of HDF5 filenames from: file.txt I1222 17:02:48.915923 3941 hdf5_data_layer.cpp:95] Number of HDF5 files: 1 F1222 17:02:48.993865 3941 hdf5_data_layer.cpp:55] Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000) *** Check failure stack trace: *** @ 0x7fd2e6608ddd google::LogMessage::Fail() @ 0x7fd2e660ac90 google::LogMessage::SendToLog() @ 0x7fd2e66089a2 google::LogMessage::Flush() @ 0x7fd2e660b6ae google::LogMessageFatal::~LogMessageFatal() @ 0x7fd2e69f9eda caffe::HDF5DataLayer<>::LoadHDF5FileData() @ 0x7fd2e69f901f caffe::HDF5DataLayer<>::LayerSetUp() @ 0x7fd2e6a48030 caffe::Net<>::Init() @ 0x7fd2e6a49278 caffe::Net<>::Net() @ 0x7fd2e6a9157a caffe::Solver<>::InitTrainNet() @ 0x7fd2e6a928b1 caffe::Solver<>::Init() @ 0x7fd2e6a92c19 caffe::Solver<>::Solver() @ 0x41222d caffe::GetSolver<>() @ 0x408ed9 train() @ 0x406741 main @ 0x7fd2e533ca40 (unknown) @ 0x406f69 _start Aborted (core dumped)

Many thanks!!!! Any advice would be appreciated!

Answer 1

The problem

It seems like there is indeed a conflict with the order of elements in arrays: matlab arranges the elements from the first dimension to the last (like fortran), while caffe and hdf5 stores the arrays from last dimension to first:
Suppose we have X of shape n x c x h x w then the "second element of X " is X[2,1,1,1] in matlab but X[0,0,0,1] in C (1-based vs 0-based indexing doesn't make life easier at all).
Therefore, when you save an array of size=[200, 6000, 1, 1] in Matlab, what hdf5 and caffe are actually seeing is as array of shape=[6000,200] .

Using the h5ls command line tool can help you spot the problem.
In matlab you saved

>> hdf5write('my_data.h5', '/new_train_x', 
  single( reshape(new_train_x,[200, 6000, 1, 1]) );
>> hdf5write('my_data.h5', '/label_train', 
  single( reshape(label_train,[200, 1, 1, 1]) ),
  'WriteMode', 'append' );

Now you can inspect the resulting my_data.h5 using h5ls (in Linux terminal):

user@host:~/$ h5ls ./my_data.h5
  label_train              Dataset {200}
  new_train_x              Dataset {6000, 200}

As you can see, the arrays are written "backwards".

Solution

Taking this conflict into account when exporting data from Matlab, you should permute :

load data.mat
hdf5write('my_data.h5', '/new_train_x', 
  single( permute(reshape(new_train_x,[200, 6000, 1, 1]),[4:-1:1] ) );
hdf5write('my_data.h5', '/label_train', 
  single( permute(reshape(label_train,[200, 1, 1, 1]), [4:-1:1] ) ),
  'WriteMode', 'append' );

Inspect the resulting my_data.h5 using h5ls now results with:

user@host:~/$ h5ls ./my_data.h5
  label_train              Dataset {200, 1, 1, 1}
  new_train_x              Dataset {200, 6000, 1, 1}

Which is what you expected in the first place.