[英]Load Native Shared Libraries in a HBase MapReduce task
Recently I'm trying to implementing my algorithm in JNI codes(using C++).I did that and generate a shared library. 最近我试图用JNI代码(使用C ++)实现我的算法,我这样做并生成了一个共享库。 Here is my JNI class. 这是我的JNI课程。
public class VideoFeature{
// JNI Code Begin
public native static float Match(byte[] testFileBytes, byte[] tempFileBytes);
static {
System.loadLibrary("JVideoFeatureMatch");
}
// JNI Code End
}
In the main function, I write 在主函数中,我写
// MapReduce
Configuration conf = HBaseConfiguration.create();
// DistributedCache shared library
DistributedCache.createSymlink(conf);
// Both following ways seem work.
// DistributedCache.addCacheFile(new URI("/home/danayan/Desktop/libJVideoFeatureMatch.so#JVideoFeatureMatch"), conf);
DistributedCache.addCacheFile(new URI("hdfs://danayan-pc:9000/lib/libJVideoFeatureMatch.so#libJVideoFeatureMatch.so"), conf);
In map method, codes following work. 在地图方法中,对以下工作进行编码。
public static class MatchMapper extends TableMapper<Text, IntWritable> {
@Override
public void map(ImmutableBytesWritable key, Result values, Context context) throws IOException, InterruptedException {
// Other codes
Path[] localFiles = DistributedCache.getLocalCacheFiles(context.getConfiguration());
for(Path temp:localFiles) {
String path = temp.toString();
if(path.contains("JVideoFeatureMatch")) {
System.out.println("JVideoFeatureMatch found !");
}
}
}
In other words, it seems that I 'DistributedCache' my shared library successfully.But I can't load it in the Map function. 换句话说,我似乎成功地'DistributedCache'了我的共享库,但是无法在Map函数中加载它。
public static class MatchMapper extends TableMapper<Text, IntWritable> {
@Override
public void map(ImmutableBytesWritable key, Result values, Context context) throws IOException, InterruptedException {
// Other codes
int score = (int)VideoFeature.Match(testBytes, tempBytes);
}
When I try to call the static function in the JNI class, a 'java.lang.Exception' is thrown : 当我尝试在JNI类中调用静态函数时,抛出“ java.lang.Exception”:
java.lang.UnsatisfiedLinkError: no libJVideoFeatureMatch in java.library.path.
I have also tried 'System.load()'. 我也尝试过'System.load()'。 And I have considered the use of prefix 'lib' and suffix '.so' in Linux system. 我已经考虑过在Linux系统中使用前缀“ lib”和后缀“ .so”。
What's more, I set a jvm argument (Removing it makes no difference): 而且,我设置了一个jvm参数(将其删除没有区别):
-Djava.library.path=/usr/local/hadoop/lib/native/Linux-amd64-64
And I have loaded the shared library in local machine successfully by moving the shared library to the 'Java.library.path'(set above). 通过将共享库移至“ Java.library.path”(如上所述),我已成功将共享库加载到本地计算机中。
I have browsed some web site below: 我浏览了下面的一些网站:
Issue loading a native library through the DistributedCache 通过DistributedCache加载本机库时出现问题
Native Libraries Guide loading native libraries in hadoop reducer? 本机库指南 在hadoop reducer中加载本机库?
I don't know if that I said clearly.If not, please let me know. 我不知道我说的是否清楚,否则请告诉我。
First copy the library to HDFS: 首先将库复制到HDFS:
bin/hadoop fs -copyFromLocal mylib.so.1 /libraries/mylib.so.1
The job launching program should contain the following: 职位启动程序应包含以下内容:
DistributedCache.createSymlink(conf); DistributedCache.addCacheFile("hdfs://host:port/libraries/mylib.so.1#mylib.so", conf);
The MapReduce task can contain: MapReduce任务可以包含:
System.load((new File("mylib.so")).getAbsolutePath());
The third point is different from the official documentation 第三点与官方文档不同
Official documentation : Native Shared Libraries 官方文档: 本地共享库
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.