字数 C++ Hadoop 管道不起作用

Question

I am trying to run the example of wordcount in C++ like this link describes the way to do: Running the WordCount program in C++ . The compilation works fine, but when I tried to run my program, an error appeared:

bin/hadoop pipes -conf../dev/word.xml -input testtile.txt -output wordcount-out
11/06/06 14:23:40 WARN mapred.JobClient: No job jar file set. User classes may not be
found. See JobConf(Class) or JobConf#setJar(String).
11/06/06 14:23:40 INFO mapred.FileInputFormat: Total input paths to process: 1
11/06/06 14:23:40 INFO mapred.JobClient: Running job: job_201106061207_0007
11/06/06 14:23:41 INFO mapred.JobClient: map 0% reduce 0%
11/06/06 14:23:53 INFO mapred.JobClient: Task Id: attempt_201106061207_0007_m_000000_0, Status: FAILED
java.io.IOException
at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188) at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194) at org.apache.hadoop.mapred.pipes.Application.(Application.java:149) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.ZB6EFD606D118D0F62066E 31419FF04CCZ.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201106061207_0007_m_000000_0: Server failed to authenticate. Exiting

I am running Hadoop on Fedora on two nodes, and I followed instructions for configurations from that link: Running Hadoop on multi-node cluster . I tried the wordcount example of Hadoop with that command:

bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount testtile.txt wordcount-out

And that command works fine. That is why I don't understand why my program did not work. So I hope that someone have an idea about what I am doing wrong, or if someone had already resolve this error.

Answer 1

I do not know if I have to answer to my question in this way, or edit my question. Anyway I find the solution and I just want to tell it for everyone who will get the same error.

After few days of research and try, I understand that Fedora and C++ on 64bits for Hadoop is not a good match. I tried to compile the Hadoop wordcount C++ with ant like explained in the wiki. But ant gets me some error about: libssl and stdint.

First, if you are on Fedora you have to add -lcrypto to the LIBS variables in the .configure . That is cause the dependency on libcrypto must now be explicitely stated on these platform when linking to libssl.(see bug on Fedora ).

Second issue: ant produces a lot of error about C++ files: to resolve that you just have to add an include: stdint.h on the top of the file.

Then the build success. I tried then to run wordcount example on my Hadoop cluster and it works, while mine did not. I expected that issue come from the library that I just corrected and I was right: I tried to run Hadoop example with library from the hadoop install directory and it did not work and I get the same error.

That could be explained by the fact that ant recompile the C++ library needed for Hadoop(with correction that I did) and used it, instead library provides in the Hadoop install directory.