spark shell dependency exception

Question

My host system windows 10 and I have cloudera vm and my spark version is 1.6 .I was trying to load the spark-shell with the below command.

spark-shell --packages org.apache.spark:spark-streaming-twitter_2.10:1.6.0

But it is throwing the below exception :

:::: ERRORS Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-twitter_2.10/1.6.0/spark-streaming-twitter_2.10-1.6.0.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)
    Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-twitter_2.10/1.6.0/spark-streaming-twitter_2.10-1.6.0.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS Exception in thread

"main" java.lang.RuntimeException: [unresolved dependency: org.apache.spark#spark-streaming-twitter_2.10;1.6.0: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1067) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

I reviewed: javax.net.ssl.SSLException: Received fatal alert: protocol_version

So it appears that it is related to the Java TLS protocol version defaults. If the Java version is outdated and results in an outdated TLS version for the request over HTTPS, then it gets blocked. I ran into this issue while trying to install a PySpark package. Many servers are now blocking outdated versions of TLS. See, for example: Github now blocks weak cryptography standards

The solution was to force TLS version 1.2 with an environment variable:

echo 'export JAVA_TOOL_OPTIONS="-Dhttps.protocols=TLSv1.2"' >> ~/.bashrc
source ~/.bashrc

When I re-ran the command to start PySpark with my package:

pyspark --packages com.databricks:spark-csv_2.10:1.5.0

The TLS version that I specified was detected immediately. It literally gave me the output:

Picked up JAVA_TOOL_OPTIONS: -Dhttps.protocols=TLSv1.2

Answer 2

It seems like issues while establishing connection with maven central.

Check this javax.net.ssl.SSLException: Received fatal alert: protocol_version

Maven central has TLS1.2 so do check yours

If it was working earlier Try to connect the url from browser and see that if its allowed

If so then do the necessary changes wrt to maven ( proxy and credentials probably)