Loading custom Python Layer in Caffe-Import Error

Question

I've written my own Caffe layer in Python (maskextractor.py). When training the network from scratch, it worked well. But once I've tried finetuning from the saved network:

../caffe/build/tools/caffe train -solver solverFCN8s_MCN_newmodule.prototxt -snapshot snapshot/train8MCNs_borders_pascal_maskextractor_iter_1.solverstate

the error I got is failed import of the new layer:

I1127 09:38:40.254966  3102 layer_factory.hpp:77] Creating layer maskextractor
ImportError: No module named mask_extractor
terminate called after throwing an instance of 'boost::python::error_already_set'
*** Aborted at 1511775520 (unix time) try "date -d @1511775520" if you are using GNU date ***

Clearly Caffe can't find the new layer. I've added it to the Pythonpath via sys.path.insert and then copied to caffe/python/caffe/ an recompiled pycaffe, but it didn't help either.

EDIT: this only happens when I finetune. If I start from solver:

import numpy as np
from PIL import Image
import os, sys
caffe_dir = "/home/ICTDOMAIN/453615/Downloads/caffe/python"
sys.path.insert(0,caffe_dir)
import caffe
newmodule_dir = "../lib/mask_extractor"
sys.path.insert(0, newmodule_dir)
import mask_extractor
#
caffe.set_mode_gpu()
caffe.set_device(0)

# continue from the saved weights
weights = 'snapshot/train8MCNs_borders_pascal_adadelta_maskextractor_new_iter_1.caffemodel'
solver=caffe.get_solver('solverFCN8s_MCN_adadelta_maskextractor_new.prototxt')
solver.net.copy_from(weights)

solver.solve()

everything works fine. But I want to continue training from he snapshot. In such case I get the error above. PythonPath looks like

print sys.path

 ['/home/ICTDOMAIN/453615/Downloads/caffe/python', '../lib/mask_extractor', '/home/ICTDOMAIN/453615/Downloads/caffe/python', '/home/ICTDOMAIN/453615/Downloads/fcn.berkeleyvision.org/voc-fcn8s', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/fast_rcnn-0.0.0-py2.7-linux-x86_64.egg', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

EDIT 2: this is how the sys.path and the import looks like

from subprocess import call
import numpy as np
from PIL import Image
import os, sys
#caffe_dir = "/home/ICTDOMAIN/453615/Downloads/caffe-crfrnn/python"
#caffe_dir = "/home/ICTDOMAIN/453615/Downloads/caffe/python"
#sys.path.insert(0,caffe_dir)
#import caffe
newmodule_dir = "/home/ICTDOMAIN/453615/Downloads/fcn.berkeleyvision.org/lib/mask_extractor"
sys.path.insert(0, newmodule_dir)
import mask_extractor
#import caffe
#
caffe.set_mode_gpu()
caffe.set_device(0)
print sys.path
# continue from the saved weights

call('/home/ICTDOMAIN/453615/Downloads/fcn.berkeleyvision.org/voc-fcn8s/run_ft.sh', shell=True)'

and sys.path is now

'/home/ICTDOMAIN/453615/Downloads/caffe/python', '/home/ICTDOMAIN/453615/Downloads/fcn.berkeleyvision.org/lib/mask_extractor'

Yet the exact same problem persists. As I mentioned before, this only comes up when I invoke caffe from the tools dir. When I create caffe net with the solver rather than solverstate, no problems are reported.

Answer 1

Step 1: Make sure you can import your layer from Python

To test your python code, you should be able to open python and type from module_name import layer_name where module_name and layer_name is what you use in the prototxt definition.

As indicated, you passed this step.

Step 2: Make sure your PYTHONPATH is valid

If your PYTHONPATH is valid, you should be able to go to the directory of your caffe excutable and then import your module. What is the result when you perform this step?

The paths in PYTHONPATH should all be absolute.