How to run Spark structured streaming using local JAR files

Question

I'm using one of the Docker images of EMR on EKS (emr-6.5.0:20211119) and investigating how to work on Kafka with Spark Structured Programming (pyspark). As per the integration guide , I run a Python script as following.

$SPARK_HOME/bin/spark-submit \
  --deploy-mode client \
  --master local \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 \
  <myscript>.py

It download the package from Maven central and I see some JAR files are downloaded into ~/.ivy2/jars .

com.github.luben_zstd-jni-1.4.8-1.jar       org.apache.spark_spark-sql-kafka-0-10_2.12-3.1.2.jar             org.slf4j_slf4j-api-1.7.30.jar
org.apache.commons_commons-pool2-2.6.2.jar  org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.1.2.jar  org.spark-project.spark_unused-1.0.0.jar
org.apache.kafka_kafka-clients-2.6.0.jar    org.lz4_lz4-java-1.7.1.jar                                       org.xerial.snappy_snappy-java-1.1.8.2.jar

However I find the main JAR file is already download into $SPARK_HOME/external/lib and I wonder how to make use of it instead of downloading it.

spark-avro_2.12-3.1.2-amzn-1.jar          spark-ganglia-lgpl.jar                      spark-streaming-kafka-0-10-assembly_2.12-3.1.2-amzn-1.jar   spark-streaming-kinesis-asl-assembly.jar
spark-avro.jar                            **spark-sql-kafka-0-10_2.12-3.1.2-amzn-1.jar  spark-streaming-kafka-0-10-assembly.jar                     spark-token-provider-kafka-0-10_2.12-3.1.2-amzn-1.jar
spark-ganglia-lgpl_2.12-3.1.2-amzn-1.jar  **spark-sql-kafka-0-10.jar                    spark-streaming-kinesis-asl-assembly_2.12-3.1.2-amzn-1.jar  spark-token-provider-kafka-0-10.jar

UPDATE 2022-03-09

I tried after updating spark-defaults.conf as shown below - added the external lib folder.

spark.driver.extraClassPath      /usr/lib/spark/external/lib/*:...
spark.driver.extraLibraryPath    ...
spark.executor.extraClassPath    /usr/lib/spark/external/lib/*:...
spark.executor.extraLibraryPath  ...

I can run without --packages but it fails with the following error.

22/03/09 05:37:25 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: org/apache/commons/pool2/impl/GenericKeyedObjectPoolConfig
        at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer$.<init>(KafkaDataConsumer.scala:623)
        at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer$.<clinit>(KafkaDataConsumer.scala)
        at org.apache.spark.sql.kafka010.KafkaBatchPartitionReader.<init>(KafkaBatchPartitionReader.scala:52)
        at org.apache.spark.sql.kafka010.KafkaBatchReaderFactory$.createReader(KafkaBatchPartitionReader.scala:40)
        at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD.compute(DataSourceRDD.scala:60)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:131)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 33 more

It doesn't help although I tried with adding --packages org.apache.commons:commons-pool2:2.6.2 .

Answer 1

You would use --jars to refer to local filesystem in-place of --packages

Answer 2

Unfortunately I cannot submit an app only with the JAR files in $SPARK_HOME/external/lib due to an error. The details of the error are updated to the question. Instead I ended up pre-downloading the package JAR files and using those.

I first ran with the following command. Here foo.py is an empty file and it'll download the package JAR files into /home/hadoop/.ivy2/jars .

$SPARK_HOME/bin/spark-submit \
  --deploy-mode client \
  --master local \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 \
  foo.py

Then I updated spark-defaults.conf as following.

spark.driver.extraClassPath      /home/hadoop/.ivy2/jars/*:...
spark.driver.extraLibraryPath    ...
spark.executor.extraClassPath    /home/hadoop/.ivy2/jars/*:...
spark.executor.extraLibraryPath  ...

After that, I ran the submit command without --packages and it worked without an error.

$SPARK_HOME/bin/spark-submit \
  --deploy-mode client \
  --master local \
  <myscript>.py

This approach is likely to be useful when it takes long to download package JAR files as they can be pre-downloaded. Note EMR on EKS supports using a custom image from ECR.