starting spark session within R

Question

I can successfully start spark session using the sparkR command in SPARK_PATH. But it does give me a warning

In SparkR::sparkR.session() :
  Version mismatch between Spark JVM and SparkR package. JVM version was 2.3.0.xxxx , while R package version was 2.3.0

The installed spark version is 2.3.0.xxxx, where xxxx is a company specific version code for our company's own deployed spark version.

However, when I try to start a spark session with R using command.

library(SparkR)
sparkR.session()

I got the following error:

Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap,  :
  Unexpected EOF in JVM connection data. Mismatched versions?

It looks like we do have a mismatch in version numbers but standalone sparkR only gives a warning while the sparkR.session() within R will just fail. I won't be able to change the version number of our spark installation as it is a company specific version. Is there any way for me to get around this?

My R version is 3.5.1 and spark version is 2.3.0.

I'm trying to use sparkR within Jupyter notebook, so the ability to start the spark session within an established R session is much more preferable.

Any suggestion would be really appreciated.

Answer 1

I was facing the same issue , it is usually due to presence of a Space " " in the in the path of SPARK_HOME directory

For example if spark home is installed in "C:/users/root/Sam folder/SPARK" Change this to "C:/users/root/Samfolder/SPARK" ie remove spaces