Apache Spark：工人已连接且免费，但不接受任务

Question

I have a simple Spark cluster - one master, and one slave. Worker is free and has no busy resources.

Web UI screenshot

But when I try execute any application (eg 'sc.parallelize(1 to 10).foreach(println)' in spark-shell) I see the following error:

WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

However when application is launched at the same server as the slave, it runs successfully. Looks like somethings listens a wrong network interface.

The configuration is default, cloned with spark from github .

I start master the following way:

192.168.111.204@spark > ./sbin/start-master.sh -h 192.168.111.204

slave:

192.168.111.230@spark > ./sbin/start-slave.sh  spark://192.168.111.204:7077 -h 192.168.111.230

application:

192.168.111.229@spark > ./bin/spark-shell --master spark://192.168.111.204:7077

What should I check?

UPD: just tried the same with two virtual machines. Works fine. Maybe servers have some problems with hostnames.

Answer 1

A few things you can try:

maybe for some reason the slaves do not have any core allocated, try starting the slave with -c .

-c CORES, --cores CORES Total CPU cores to allow Spark applications 
    to use on the machine (default: all available); only on worker

Answer 2

Thank to everyone, problem is solved. As I guessed before, networking was a cause of troubles.

When spark-shell and spark-submit start, they open a port to listen. However, I didn't find a flag to specify a host for this purpose. So, they started listening on the outer inteface, the port on which was blocked by a firewall. I had to add the following lines to conf/spark-env.sh :

export SPARK_LOCAL_IP=192.168.111.229