Hortonworks沙箱中的流Hadoop jar文件流，无contrib目录

Question

I'm demoing hadoop in the hortonworks sandbox virtual machine .

I've previously set up simple elastic map reduce streaming jobs on EC2, following patterns like this , or this .

However, it seems I have no streaming jar installed---in fact, I seem to be lacking many of the basic directories required in:

$HADOOP_HOME/mapred/contrib/

My ls -lah actually looks like this:

[root@sandbox ~]# ls -lah
total 60K
dr-xr-x---.  5 root root 4.0K Apr 10 18:52 .
dr-xr-xr-x. 24 root root 4.0K Apr 10 18:31 ..
-rw-------   1 root root  126 Oct 28 08:35 .bash_history
-rw-r--r--.  1 root root   18 May 20  2009 .bash_logout
-rw-r--r--.  1 root root  176 May 20  2009 .bash_profile
-rw-r--r--   1 root root  262 Oct 28 08:29 .bashrc
-rw-r--r--.  1 root root  100 Sep 22  2004 .cshrc
-rw-r--r--   1 root root    0 Oct 28 08:34 .hdfs_prepared
drwxr-xr-x   2 root root 4.0K Apr 10 18:22 .pip
drwxr-----   3 root root 4.0K Oct 20 16:21 .pki
-rw-------   1 root root 1.0K Oct 20 14:04 .rnd
drwx------   2 root root 4.0K Oct 20 09:21 .ssh
lrwxrwxrwx   1 root root   48 Oct 28 08:28 start_ambari.sh -> /usr/lib/hue/tools/start_scripts/start_ambari.sh
lrwxrwxrwx   1 root root   47 Oct 28 08:28 start_hbase.sh -> /usr/lib/hue/tools/start_scripts/start_hbase.sh
-rw-r--r--.  1 root root  129 Dec  3  2004 .tcshrc
-rw-------   1 root root 4.8K Oct 28 08:30 .viminfo
-rw-r--r--   1 root root  218 Oct 20 08:55 zero_machine.sh

Although using the hadoop commands, I see that the mapred exists, but contains no contrib.

[root@sandbox ~]# hadoop fs -ls /
Found 6 items
drwxrwxrwt   - yarn   hadoop          0 2014-04-10 19:14 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2013-10-20 15:08 /apps
drwxr-xr-x   - mapred hdfs            0 2013-10-20 15:10 /mapred
drwxr-xr-x   - hdfs   hdfs            0 2013-10-20 15:10 /mr-history
drwxrwxrwx   - hdfs   hdfs            0 2013-10-28 08:34 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2013-10-28 08:34 /user
[root@sandbox ~]# hadoop fs -ls /mapred/
Found 1 items
drwxr-xr-x   - mapred hdfs          0 2013-10-20 15:10 /mapred/system

Is there a special download page for the streaming .jar file? When I access the link at the bottom of this page , the link to streaming is dead .

Answer 1

The default location of hadoop streaming jar is /usr/lib/hadoop/contrib/streaming/hadoop-streaming-*.jar .

If you cannot find the jar in the above mentioned location. you could download hadoop-streaming-*.jar from the following Hortonworks repository:

http://repo.hortonworks.com/content/repositories/releases/org/apache/hadoop/hadoop-streaming/

Answer 2

There is a convenience link /usr/lib/hadoop-mapreduce/hadoop-streaming.jar to that jar. I tested it in Hortonworks Sandbox 2.1

#ls -AlF /usr/lib/hadoop-mapreduce/hadoop-streaming.jar
lrwxrwxrwx 1 root root 38 Apr 21 06:57 /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -> hadoop-streaming-2.4.0.2.1.1.0-385.jar