How to use java runtime 11 in EMR cluster AWS

Question

I'm creating a cluter in EMR aws and when spark runs my application I'm getting error below:

Exception in thread "main" java.lang.UnsupportedClassVersionError: 
com/example/demodriver/MyClassFromJAR has been compiled by a more recent version of the Java Runtime (class file version 55.0), 
this version of the Java Runtime only recognizes class file versions up to 52.0

I'm using releaseLabel emr-6.5.0 on clusters and My driver jar is built in java11

How run java11 application in EMR? Or this error is about anything else?

Answer 1

In the most recent versions of EMR, java 11 is installed. To enable it, you can provide the following configuration.

[
    {
        "Classification": "spark-env",
        "Configurations": [
            {
                "Classification": "export",
                "Properties": {
                    "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
                }
            }
        ]
    },
    {
        "Classification": "spark-defaults",
        "Properties": {
            "spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
        }
    }
]

This does not seem to be documented.

The defaultJavaOptions and extraJavaOptions might contain incompatible options for java 11 which you still might need to adapt/update.

Answer 2

Here is the full configuration including the necessary JVM options:

[
    {
        "Classification": "spark-env",
        "Configurations": [
            {
                "Classification": "export",
                "Properties": {
                    "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
                }
            }
        ]
    },
    {
        "Classification": "spark-defaults",
        "Properties": {
            "spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64",
            "spark.driver.defaultJavaOptions": "-XX:OnOutOfMemoryError='kill -9 %p' -XX:MaxHeapFreeRatio=70",
            "spark.executor.defaultJavaOptions": "-verbose:gc -Xlog:gc*::time -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' -XX:MaxHeapFreeRatio=70 -XX:+IgnoreUnrecognizedVMOptions"
        }
    }
]

Answer 3

For running Spark (3.3.0) on EMR 6 (6.9.0) I had to provide the following (note the additional "hadoop-env" and the empty "Properties" elements):

[
{
    "Classification": "hadoop-env", 
    "Configurations": [
        {
            "Classification": "export", 
            "Configurations": [], 
            "Properties": {
                "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
            }
        }
    ], 
    "Properties": {}
}, 
{
    "Classification": "spark-env",
    "Configurations": [
        {
            "Classification": "export",
             "Configurations": [],
            "Properties": {
                "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
            }
        }
    ],
    "Properties": {}
},
{
    "Classification": "spark-defaults",
    "Properties": {
        "spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
    }
}

]