Check failed: error == cudaSuccess during training SSD

Question

I am training SSD and I have error as

I0116 13:10:31.206343  3447 net.cpp:761] Ignoring source layer drop6
I0116 13:10:31.207219  3447 net.cpp:761] Ignoring source layer drop7
I0116 13:10:31.207229  3447 net.cpp:761] Ignoring source layer fc8
I0116 13:10:31.207233  3447 net.cpp:761] Ignoring source layer prob
F0116 13:10:31.227303  3447 parallel.cpp:130] Check failed: error == cudaSuccess (10 vs. 0)  invalid device ordinal
*** Check failure stack trace: ***
    @     0x7f158382e5cd  google::LogMessage::Fail()
    @     0x7f1583830433  google::LogMessage::SendToLog()
    @     0x7f158382e15b  google::LogMessage::Flush()
    @     0x7f1583830e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f158412f7bd  caffe::DevicePair::compute()
    @     0x7f15841354e0  caffe::P2PSync<>::Prepare()
    @     0x7f1584135fee  caffe::P2PSync<>::Run()
    @           0x40af10  train()
    @           0x407608  main
    @     0x7f1581fbd830  __libc_start_main
    @           0x407ed9  _start
    @              (nil)  (unknown)
Aborted (core dumped)

My Graphic is Quadro4200. ./deviceQuery gives me

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro K4200"
  CUDA Driver Version / Runtime Version          9.0 / 8.0
  CUDA Capability Major/Minor version number:    3.0
  Total amount of global memory:                 4034 MBytes (4230479872 bytes)
  ( 7) Multiprocessors, (192) CUDA Cores/MP:     1344 CUDA Cores
  GPU Max Clock rate:                            784 MHz (0.78 GHz)
  Memory Clock rate:                             2700 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 524288 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = Quadro K4200
Result = PASS

I can successfully test SSD library, just that I have error in training.

Is that Graphic card not powerful enough to train the library?

Answer 1

I found the error.

If we run this command python examples/ssd/ssd_pascal.py in ssd, the next step of training command is as follow.

gdb --args ./build/tools/caffe train --solver="models/VGGNet/VOC0712/SSD_300x300/solver.prototxt" --weights="models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel" --gpu 0,1,2,3 2>&1 | tee jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.log

this --gpu 0,1,2,3 2>&1 is giving the issue. I changed to --gpu 0 and run from the training command directly as

./build/tools/caffe train --solver="models/VGGNet/VOC0712/SSD_300x300/solver.prototxt" --weights="models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel" --gpu 0 | tee jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.log ./build/tools/caffe train --solver="models/VGGNet/VOC0712/SSD_300x300/solver.prototxt" --weights="models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel" --gpu 0 | tee jobs/VGGNet/VOC0712/SSD_300x300/VGG_VOC0712_SSD_300x300.log ,

then it solved.