使用紧缩MemPipeline编写时出现java.lang.UnsatisfiedLinkError

Question

I am using com.cloudera.crunch version: '0.3.0-3-cdh-5.2.1'.

I have a small program that reads some AVROs and filters out invalid data based on some criteria. I am using pipeline.write(PCollection, AvroFileTarget) to write the invalid data output. It works fine in production run.

For unit testing this piece of code, I use MemPipeline instance. But, it fails while writing the output in that case.

I get error:

java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
    at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)
    at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86)
    at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:428)
    at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:197)
    at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163)
    at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:144)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:78)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
    at java.io.DataOutputStream.writeBytes(DataOutputStream.java:276)
    at com.cloudera.crunch.impl.mem.MemPipeline.write(MemPipeline.java:159)

Any idea what's wrong?

Answer 1

Hadoop environment variable should be configured properly along with hadoop.dll and winutils.exe.

Also pass the JVM argument while executing MR job/application -Djava.library.path=HADOOP_HOME/lib/native