在 Spark 中将 IP 地址设置为 spark.driver.host 而不是主机名？

Question

I have a problem with declaring an spark.driver.host setting. Assume that i have an host with a HOSTNAME and binded address 192.168.1.1 (its linux, windows has not this problem at all).

If i set a property spark.driver.hostname = 192.168.1.1 workers will respond to HOSTNAME, which is really bad, because they don't have DNS with this hostname.

As this troubleshooting page says:

If SPARK_LOCAL_IP is set to an IP address, it will be resolved to a hostname.

And here is a question: how to NOT setting this IP address as a hostname?

Answer 1

你可以在你的spark环境中设置SPARK_LOCAL_HOSTNAME=ip地址来解决这个问题。

Answer 2

hostname -f gives your FQDN add this to /etc/hosts file as

[your ip address] [your hostname] on spark nodes

or add your hostname to dns used by your spark nodes