Spark started very slow

Question

I have a shell script that starts a spark application using spark-submit.

I found today that spark starts very slowly. The log is as follows

18/10/07 06:00:53  WARN  SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
18/10/07 06:00:53  WARN  SparkConf: Detected deprecated memory fraction settings: [spark.shuffle.memoryFraction, spark.storage.memoryFraction, spark.storage.unrollFraction]. As of Spark 1.6, execution and storage memory management are unified. All memory fractions used in the old model are now deprecated and no longer read. If you wish to use the old memory management, you may explicitly enable `spark.memory.useLegacyMode` (not recommended).
18/10/07 06:00:54  INFO  log: Logging initialized @47766ms
18/10/07 06:00:54  INFO  Server: jetty-9.2.z-SNAPSHOT
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@4a481728{/jobs,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@4ace284d{/jobs/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@77fa9381{/jobs/job,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@76d72b5b{/jobs/job/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@7cac93fe{/stages,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@479b5066{/stages/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@64deb58f{/stages/stage,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2b1cd7bc{/stages/stage/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@33379242{/stages/pool,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2e32fc22{/stages/pool/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@3f5156a6{/storage,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@72bdd7c7{/storage/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@7d17ee50{/storage/rdd,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@44fdce3c{/storage/rdd/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@712c5463{/environment,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@6d33a66e{/environment/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2f14b0f6{/executors,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5b20706{/executors/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@16aed49f{/executors/threadDump,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@66de00f2{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@6c479fdf{/static,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@79d82f66{/,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@19b206ee{/api,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@ffd4cba{/jobs/job/kill,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5f1483fd{/stages/stage/kill,null,AVAILABLE,@Spark}
18/10/07 06:00:54  INFO  ServerConnector: Started Spark@21263314{HTTP/1.1}{10.1.237.119:22640}
18/10/07 06:00:54  INFO  Server: Started @47955ms
18/10/07 06:06:46  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5d5574c7{/metrics/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@2328ec73{/SQL,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@58021c3c{/SQL/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@7bda01da{/SQL/execution,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@5c997de8{/SQL/execution/json,null,AVAILABLE,@Spark}
18/10/07 06:06:47  INFO  ContextHandler: Started o.s.j.s.ServletContextHandler@15d58530{/static/sql,null,AVAILABLE,@Spark}
18/10/07 06:06:48  WARN  SessionState: load mapred-default.xml, HIVE_CONF_DIR env not found!
18/10/07 06:06:50  WARN  SessionState: load mapred-default.xml, HIVE_CONF_DIR env not found!

From the log, it takes about 6 minutes to starts some server?

18/10/07 06:00:54  INFO  Server: Started @47955ms
18/10/07 06:06:46  INFO  ContextHandler: Started 

I would ask what may happen here, why does it take so much time?

Answer 1

我认为启动服务器只花了47秒，建议您查看链接，以解决未找到HIVE_CONF_DIR env的问题