如何在pyMIC中为Xeon Phi编译knn

Question

I am trying to compile a modified mlpack knn_example.cpp example for the Xeon Phi Knights Corner. It uses mlpack library and also the Armadillo c++ libraries. It seems to compile successfully however when I run the pymic code, it throws the following error:

`pymic.offload_error.OffloadError: Could not load library 'knn.so' on device 0`.

The modified c++ code is:

extern "C" {
    #include <Python.h>
    #include <numpy/arrayobject.h>
}

#include <mlpack/core.hpp>
#include <mlpack/methods/neighbor_search/neighbor_search.hpp>
#include <pymic_kernel.h>

using namespace mlpack;
using namespace mlpack::neighbor; // NeighborSearch and NearestNeighborSort
using namespace mlpack::metric; // ManhattanDistance

PYMIC_KERNEL
void calc_knn(double *arr, double *distarr)
{
    // Armadillo is a C++ linear algebra library; mlpack uses its matrix data type.
    arma::mat data(arr, 17, 4);

    /*
     * Load the data from a file. mlpack does not provide an example dataset, 
     * so I've included a tiny one.
     *
     * 'data' is a helper class in mlpack that facilitates saving and loading 
     * matrices and models.
     *
     * Pass the filename, matrix to hold the data, and set fatal = true to have
     * it throw an exception if there is an issue.
     *
     * 'Load' excepts comma-separated and tab-separated text files, and will 
     * infer the format.
     */
    //data::Load("data.csv", data, true);

    /* 
     * Create a NeighborSearch model. The parameters of the model are specified
     * with templates:
     *  - Sorting method: "NearestNeighborSort" - This class sorts by increasing
     *    distance.
     *  - Distance metric: "ManhattanDistance" - The L1 distance, sum of absolute
     *    distances.
     *
     * Pass the reference dataset (the vectors to be searched through) to the
     * constructor.
     */
     NeighborSearch<NearestNeighborSort, ManhattanDistance> nn(data);

    /*
     * Create the matrices to hold the results of the search. 
     *   neighbors [k  x  n] - Indeces of the nearest neighbor(s). 
     *                         One column per data query vector and one row per
     *                        'k' neighbors.
     *   distances [k  x  n] - Calculated distance values.
     *                         One column per data query vector and one row per
     *                        'k' neighbors.
     */
    arma::Mat<size_t> neighbors;
    arma::mat distances(distarr, 17, 17); 

    /*
     * Find the nearest neighbors. 
     *
     * If no query vectors are provided (as is the case here), then the 
     * reference vectors are searched against themselves.
     *
     * Specify the number of neighbors to find, k = 1, and provide matrices
     * to hold the result indeces and distances.
     */ 
    nn.Search(1, neighbors, distances);

    // Print out each neighbor and its distance.
    for (size_t i = 0; i < neighbors.n_elem; ++i)
    {
        std::cout << "Nearest neighbor of point " << i << " is point "
        << neighbors[i] << " and the distance is " << distances[i] << ".\n";
    }


}

Compilation argument is as follows:

icpc -std=c++11 -I/home/userxx/Downloads/pyMIC/include -I/usr/include/python2.7 -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/usr/include -fPIC -qopenmp  -g -shared -mmic -o knn.so knn.cpp

And running it using this python code in knn.py:

import pymic as mic
import numpy as np
from collections import namedtuple

Point = namedtuple("Point", "x y val distance")

# load the library with the kernel function (on the target)
device = mic.devices[0]
library = device.load_library(("knn.so",))
stream = device.get_default_stream()

#na = np.arange(1, 33)
#a = stream.bind(na)


n = 17; # Number of data points 

arr = np.array([[1, 12, 0, 0],
    [2,  5, 0, 0],
    [5, 3, 1, 0],
    [3, 2, 1, 0],
    [3, 6, 0, 0],
    [1.5, 9, 1, 0],
    [7, 2, 1, 0],
    [6, 1, 1, 0],
    [3.8, 3, 1, 0],
    [3, 10, 0, 0],
    [5.6, 4, 1, 0],
    [4,2,1, 0],
    [3.5, 8, 0, 0],
    [2, 11, 0, 0],
    [2, 5, 1, 0],
    [2, 9, 0, 0],
    [1, 7, 0, 0]])

distances = np.zeros((n,n))

distances_off = stream.bind(distances)

print "input:"
print "--------------------------------------"
print arr
print

stream.invoke(library.calc_knn, arr, distances_off)
stream.sync()

print "output:"
print "--------------------------------------"
distance_off.update_host()
stream.sync()
#np.asarray(p_off)

print "The distances are: "
print distance_off

And the output with OFFLOAD_REPORT=3 follows:

[root@localhost knn_mlpack_example]# python2.7 knn.py 
PYMIC: debug level set to 7
PYMIC: tracing is disabled
PYMIC: found 1 device(s)
PYMIC: created stream 0x169f6f0 for device 0
PYMIC: searching for liboffload_array.so in 
PYMIC:     looking for liboffload_array.so in /usr/lib64/python2.7/site-packages/pymic
PYMIC: loading '/usr/lib64/python2.7/site-packages/pymic/liboffload_array.so' on device 0
[Offload] [HOST]          [State]           Initialize logical card 0 = physical card 0
[Offload] [MIC 0] [File]                    src/pymicimpl_misc.cc
[Offload] [MIC 0] [Line]                    186
[Offload] [MIC 0] [Tag]                     Tag 0
[Offload] [HOST]  [Tag 0] [State]           Start target
[Offload] [HOST]  [Tag 0] [State]           Setup target entry: __offload_entry_pymicimpl_misc_cc_186target_lo_cb16b0e23fb7bc6a3c1ad05f43fef8d0icc06321843140yPzMH
[Offload] [HOST]  [Tag 0] [Signal]          signal : none
[Offload] [HOST]  [Tag 0] [Signal]          waits  : none
[Offload] [HOST]  [Tag 0] [State]           Gather copyin data: base=0x16af790 length=80289
[Offload] [HOST]  [Tag 0] [State]           Create target buffer: size=82225 offset=1936
[Offload] [HOST]  [Tag 0] [State]           Host->target pointer data 80289
[Offload] [HOST]  [Tag 0] [State]           Host->target copyin data 24 
[Offload] [HOST]  [Tag 0] [State]           Execute task on target
[Offload] [HOST]  [Tag 0] [State]           Target->host pointer data 320
[Offload] [MIC 0] [Tag 0] [State]           Start target entry: __offload_entry_pymicimpl_misc_cc_186target_lo_cb16b0e23fb7bc6a3c1ad05f43fef8d0icc06321843140yPzMH
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr__ZN5pymic19target_load_libraryEiRKSsRSsRm.176  NOCOPY
[Offload] [MIC 0] [Tag 0] [Var]             size_in  IN
[Offload] [MIC 0] [Tag 0] [Var]             data  IN
[Offload] [MIC 0] [Tag 0] [Var]             bufsz  IN
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr__ZN5pymic19target_load_libraryEiRKSsRSsRm.176  OUT
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr__ZN5pymic19target_load_libraryEiRKSsRSsRm.176  OUT
[Offload] [MIC 0] [Tag 0] [Var]             handle_device_ptr  OUT
[Offload] [MIC 0] [Tag 0] [Var]             tempname_cstr_sz  INOUT
[Offload] [MIC 0] [Tag 0] [State]           Target->host copyout data   16
[Offload] [HOST]  [Tag 0] [CPU Time]        0.835535(seconds)
[Offload] [MIC 0] [Tag 0] [CPU->MIC Data]   80313 (bytes)
[Offload] [MIC 0] [Tag 0] [MIC Time]        0.001485(seconds)
[Offload] [MIC 0] [Tag 0] [MIC->CPU Data]   336 (bytes)

PYMIC: successfully loaded '/usr/lib64/python2.7/site-packages/pymic/liboffload_array.so' on device 0 with handle 0x7f168c0011c0
PYMIC: starting initialization of the offload infrastructure
PYMIC: loading supporting pyMIC libraries on all devices
PYMIC: searching for knn.so in 
PYMIC:     looking for knn.so in /usr/lib64/python2.7/site-packages/pymic
PYMIC:     looking for knn.so in 
PYMIC: loading 'knn.so' on device 0
[Offload] [MIC 0] [File]                    src/pymicimpl_misc.cc
[Offload] [MIC 0] [Line]                    186
[Offload] [MIC 0] [Tag]                     Tag 1
[Offload] [HOST]  [Tag 1] [State]           Start target
[Offload] [HOST]  [Tag 1] [State]           Setup target entry: __offload_entry_pymicimpl_misc_cc_186target_lo_cb16b0e23fb7bc6a3c1ad05f43fef8d0icc06321843140yPzMH
[Offload] [HOST]  [Tag 1] [Signal]          signal : none
[Offload] [HOST]  [Tag 1] [Signal]          waits  : none
[Offload] [HOST]  [Tag 1] [State]           Gather copyin data: base=0x16dc310 length=3100981
[Offload] [HOST]  [Tag 1] [State]           Create target buffer: size=3101765 offset=784
[Offload] [HOST]  [Tag 1] [State]           Host->target pointer data 3100981
[Offload] [HOST]  [Tag 1] [State]           Host->target copyin data 32 
[Offload] [HOST]  [Tag 1] [State]           Execute task on target
[Offload] [HOST]  [Tag 1] [State]           Target->host pointer data 320
[Offload] [MIC 0] [Tag 1] [State]           Start target entry: __offload_entry_pymicimpl_misc_cc_186target_lo_cb16b0e23fb7bc6a3c1ad05f43fef8d0icc06321843140yPzMH
[Offload] [MIC 0] [Tag 1] [Var]             __offload_stack_ptr__ZN5pymic19target_load_libraryEiRKSsRSsRm.176  NOCOPY
[Offload] [MIC 0] [Tag 1] [Var]             size_in  IN
[Offload] [MIC 0] [Tag 1] [Var]             data  IN
[Offload] [MIC 0] [Tag 1] [Var]             bufsz  IN
[Offload] [MIC 0] [Tag 1] [Var]             __offload_stack_ptr__ZN5pymic19target_load_libraryEiRKSsRSsRm.176  OUT
[Offload] [MIC 0] [Tag 1] [Var]             __offload_stack_ptr__ZN5pymic19target_load_libraryEiRKSsRSsRm.176  OUT
[Offload] [MIC 0] [Tag 1] [Var]             handle_device_ptr  OUT
[Offload] [MIC 0] [Tag 1] [Var]             tempname_cstr_sz  INOUT
[Offload] [MIC 0] [Tag 1] [State]           Target->host copyout data   16
[Offload] [HOST]  [Tag 1] [CPU Time]        0.020388(seconds)
[Offload] [MIC 0] [Tag 1] [CPU->MIC Data]   3101013 (bytes)
[Offload] [MIC 0] [Tag 1] [MIC Time]        0.014470(seconds)
[Offload] [MIC 0] [Tag 1] [MIC->CPU Data]   336 (bytes)

caught!
Traceback (most recent call last):
  File "knn.py", line 9, in <module>
    library = device.load_library(("knn.so",))
  File "/usr/lib64/python2.7/site-packages/pymic/_tracing.py", line 128, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/pymic/offload_device.py", line 168, in load_library
    return OffloadLibrary(libraries[0], device=self)
  File "/usr/lib64/python2.7/site-packages/pymic/offload_library.py", line 125, in __init__
    filename)
  File "src/pymic_libxstream.pyx", line 238, in pymic.pymic_libxstream.pymic_library_load
  File "src/pymic_libxstream.pyx", line 231, in pymic.pymic_libxstream._c_pymic_library_load
pymic.offload_error.OffloadError: Could not load library 'knn.so' on device 0
[Offload] [MIC 0] [File]                    src/pymicimpl_misc.cc
[Offload] [MIC 0] [Line]                    258
[Offload] [MIC 0] [Tag]                     Tag 2
[Offload] [HOST]  [Tag 2] [State]           Start target
[Offload] [HOST]  [Tag 2] [State]           Setup target entry: __offload_entry_pymicimpl_misc_cc_258target_un_0cc318b72500803c42a213bcdc3aa259icc06321843140yPzMH
[Offload] [HOST]  [Tag 2] [Signal]          signal : none
[Offload] [HOST]  [Tag 2] [Signal]          waits  : none
[Offload] [HOST]  [Tag 2] [State]           Gather copyin data: base=0x1667c78 length=22
[Offload] [HOST]  [Tag 2] [State]           Create target buffer: size=3214 offset=3192
[Offload] [HOST]  [Tag 2] [State]           Host->target pointer data 22
[Offload] [HOST]  [Tag 2] [State]           Host->target copyin data 24 
[Offload] [HOST]  [Tag 2] [State]           Execute task on target
[Offload] [HOST]  [Tag 2] [State]           Target->host pointer data 256
[Offload] [MIC 0] [Tag 2] [State]           Start target entry: __offload_entry_pymicimpl_misc_cc_258target_un_0cc318b72500803c42a213bcdc3aa259icc06321843140yPzMH
[Offload] [MIC 0] [Tag 2] [Var]             __offload_stack_ptr__ZN5pymic21target_unload_libraryEiRKSsm.30  NOCOPY
[Offload] [MIC 0] [Tag 2] [Var]             handle  IN
[Offload] [MIC 0] [Tag 2] [Var]             bufsz  IN
[Offload] [MIC 0] [Tag 2] [Var]             tempname_cstr  IN
[Offload] [MIC 0] [Tag 2] [Var]             __offload_stack_ptr__ZN5pymic21target_unload_libraryEiRKSsm.30  OUT
[Offload] [MIC 0] [Tag 2] [Var]             errorcode  OUT
[Offload] [MIC 0] [Tag 2] [State]           Target->host copyout data   4
[Offload] [HOST]  [Tag 2] [CPU Time]        0.001950(seconds)
[Offload] [MIC 0] [Tag 2] [CPU->MIC Data]   46 (bytes)
[Offload] [MIC 0] [Tag 2] [MIC Time]        0.000585(seconds)
[Offload] [MIC 0] [Tag 2] [MIC->CPU Data]   260 (bytes)

offload error: cannot unload library from the device 0 (error code 14)

Will be grateful if anyone can put me on the right path.

Answer 1

There's an open issue at the pymic Github site that investigates if there's a bug with pymic or if it's an issue with the environment settings. Please see https://github.com/intel/pyMIC/issues/22 . I will post here, once the issue has been resolved and closed.

Answer 2

This issue has been closed at the Github project site. Workaround:

The problem is that dependences of shared-object libraries are not copied automatically. For MKL, one can manually copy the MKL shared-object libraries through the scp command (for mic0, similar for other coprocessors in the system):

scp $MKLROOT/lib/mic/*.so mic0:/lib64

If home-brew libraries have dependencies, then they can be copied with similar scp commands targeting /lib64 on the MIC devices.