基本的hadoop mapreduce工作正在启动，但尚未完成

Question

I'm looking at Hadoop, but when I try to submit a mapreduce job Hadoop appears to start, but then hangs with no indication of progress, or other activity. The Application Status pages say it's been submitted and show the job, but nothing happens, and I'm curious of where to look to resolve this problem.

I'm using Hadoop version 2.7.1, installed in OS X 10.10.4 using Homebrew and Java 1.8.0_45. I configured it per these instructions: https://datarecipe.wordpress.com/2015/06/05/setup-hadoop-2-6-on-mac-osx-10-9/

The data is a simple text file called "purchases.txt" containing this (tab-delimited):

2013-03-29  2:30    miami   cup 2.43    visa
2013-04-23  1:34    miami   cup 2.43    visa
2013-04-23  10:15   LA  spoon   1.32    visa
2013-04-28  6:34    LA  bottle  3.56    cash
2013-05-23  1:43    miami   glass   3.21    visa

I've uploaded this into hadoop with (data folder already created):

hadoop fs -put purchases.txt /data/

I then set up the following mapper in python (per an online tutorial) and called it "mapper.py":

import sys

def mapper():
    for line in sys.stdin:
        tempdata = line.strip().split("\n")
        for l in tempdata:
            if (len(l.split("\t")) == 6):
                date, time, store, item, cost, payment = l.split("\t")
                print("{0}\t{1}".format(store,cost))

def main():
    mapper()

if __name__=="__main__":
    main()

I did the same for the reducer code, and called it "reducer.py":

import sys

def reducer():
    salesTotal = 0
    oldKey = None
    for line in sys.stdin:
        data = line.strip().split("\t")
        if len(data)!=2:
            continue
        thisKey, thisSale = data
        if oldKey and oldKey != thisKey:
            print("{0}\t{1}".format(oldKey,salesTotal))
            salesTotal=0
        oldKey = thisKey
        salesTotal+=float(thisSale)
    if oldKey != None:
        print("{0}\t{1}".format(oldKey,salesTotal))

def main():
    reducer()

if __name__=="__main__":
    main()

Testing these bits of code works on the command line:

Tophers-Retina-MBP:Hadoop tkessler$ cat purchases.txt | ./mapper.py | sort | ./reducer.py 
LA  4.88
miami   5.640000000000001

However, when I run the stream process to run it in Hadoop, it just stalls here:

Tophers-Retina-MBP:lib tkessler$ hadoop jar ./hadoop-streaming-2.7.1.jar -mapper ~/PycharmProjects/Hadoop/mapper.py -reducer ~/PycharmProjects/Hadoop/reducer.py -file ~/PycharmProjects/Hadoop/mapper.py -input /data -output /project1out
packageJobJar: [/Users/tkessler/PycharmProjects/Hadoop/mapper.py, /var/folders/f_/3zvmc1g95lqgt1cp2dtnrtqw0000gp/T/hadoop-unjar2355518779286421017/] [] /var/folders/f_/3zvmc1g95lqgt1cp2dtnrtqw0000gp/T/streamjob8766144507660069606.jar tmpDir=null

It seems to start the job just fine, and accept the mapper and reducer, and running "mapred job -list all" shows the jobs are all running, but it never completes, and the status is just listed as "unknown". I'm not sure if its a hadoop configuration issue, or some other problem, if anyone has any insight.

Addition:

When I run the following example command, the progress seems to just stop at the following line:

Tophers-Retina-MBP:~ tkessler$ hadoop jar /usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 4 1000
Number of Maps  = 4
Samples per Map = 1000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job

Answer 1

Cleared out Hadoop by shutting down the namenode and datanodes, and then uninstalled it with brew uninstall hadoop and then followed the instructions on this page for setting it up: http://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/

Seems to be working great now, so perhaps it was just a slight configuration change (likely with the temporary file location), but it processes the mapper and reducer quite nicely now.