在Mac OS X 10.8上設置hadoop的問題

Question

我在Mac上設置了一些東西來安裝hadoop。 但是有這樣的錯誤消息：

13/02/18 04:05:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).
13/02/18 04:05:53 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).
13/02/18 04:05:54 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).
13/02/18 04:05:55 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).
13/02/18 04:05:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).
13/02/18 04:05:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).
13/02/18 04:05:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).
13/02/18 04:05:59 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).
13/02/18 04:06:00 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).
13/02/18 04:06:01 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).
java.lang.RuntimeException: java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
    at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:546)
    at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:318)
    at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:265)
    at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1099)
    at org.apache.hadoop.ipc.Client.call(Client.java:1075)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
    at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:542)
    ... 17 more
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
    at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
    at org.apache.hadoop.ipc.Client.call(Client.java:1050)
    ... 31 more

然后輸入jps檢查服務，結果是：20635 Jps

20466 TaskTracker

20189數據節點

20291 SecondaryNameNode

我不知道該如何處理該錯誤。 有人可以給我答案嗎？ 多謝！！！

Answer 1

實際上，由於IP配置錯誤，所有（hadoop應該運行的）進程都沒有運行。我不熟悉Mac OS，但是在linux和Windows上，我們需要在主機文件（etc / hosts）中放置hadoop enterents以進行連接，我該死的確保它適用於Mac。現在關鍵是
您需要將hadoop條目作為本地mnachine放在該文件中，例如127.0.0.1
實際上，您需要將其放置在計算機的實際IP上
例如
hadoop-machine 127.0.0.1->（在此處放置回送IP是錯誤的，因為hadoop將嘗試與此IP連接）。
刪除此127.0.0.1，並將計算機的實際IP放置在此條目的前面。 您可以輕松地找到mac機的ip。以下是一些與hadoop沒有直接關系的問題，但我想它們對您會有所幫助。
問題1 ， 問題2 ， 問題3

Answer 2

這可能會有所幫助。 但是首先您必須使用以下命令刪除較早的安裝

rm -rf / usr / local / Cellar /usr/local/.git && brew清理

然后，U可能會開始在U're Mac上重新安裝Hadoop。

步驟1：安裝Homebrew

$ ruby​​ -e“ $（curl -fsSL https://raw.github.com/mxcl/homebrew/go ）”

步驟2：安裝Hadoop

$ brew install hadoop

假設brew安裝Hadoop 1.2.1

步驟3：配置Hadoop

$ cd /usr/local/Cellar/hadoop/1.2.1/libexec

將以下行添加到conf / hadoop-env.sh：

出口HADOOP_OPTS =“-Djava.security.krb5.realm = -Djava.security.krb5.kdc =”

將以下行添加到conf / core-site.xml中的配置標簽中：

 <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> 

將以下行添加到conf / hdfs-site.xml中的配置標簽中：

 <property> <name>dfs.replication</name> <value>1</value> </property> 

將以下行添加到conf / mapred-site.xml中的配置標簽中：

  <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> 

轉到系統偏好設置>共享。 確保選中“遠程登錄”。

$ ssh-keygen -t rsa $ cat〜/ .ssh / id_rsa.pub >>〜/ .ssh / authorized_keys

步驟4：啟用SSH到localhost

$ cat〜/ .ssh / id_rsa.pub >>〜/ .ssh / authorized_keys

步驟5：格式化Hadoop文件系統

$ bin / hadoop namenode-格式

步驟6：啟動Hadoop

$ bin / start-all.sh

確保所有Hadoop進程都在運行：

$ jps

運行Hadoop示例：

還有一件事 ！ “ brew update”會將hadoop二進制文件更新為最新版本（當前為1.2.1）。

Answer 3

我遇到了同樣的錯誤： ConnectException: Connection refused所有secondarynamenode日志文件中的ConnectException: Connection refused 。

但是我也在namenode的日志文件中找到了這個：

2015-10-25 16:35:15,720 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /private/tmp/hadoop-admin/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.

因此，我做了一個

hadoop namenode -format

問題就消失了。 因此，錯誤消息與namenode沒有成功啟動有關。