Quick Guide to bootstrapping Java 11 on EMR

Steps to configure EMR to run jars compiled using Java 11/bytecode version 55.0 and some “gotchas”!

Caused by: java.lang.UnsupportedClassVersionError: com/group/project/YourClass has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
...

Step 1: Add GC ignore options to spark configuration

When creating EMR cluster, in the configuration section, add these options to spark-defaults :

[
{
"classification": "spark-defaults",
"properties": {
"spark.driver.defaultJavaOptions": "-XX:OnOutOfMemoryError='kill -9 %p' -XX:MaxHeapFreeRatio=70",
"spark.executor.defaultJavaOptions": "-verbose:gc -Xlog:gc*::time -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' -XX:MaxHeapFreeRatio=70 -XX:+IgnoreUnrecognizedVMOptions"
},
"configurations": []
}
]
Unrecognized VM option 'UseGCLogFileRotation'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Step 2: Write and Store bootstrap.sh in S3

Add a bootstrap step to run the following script once the node is provisioned: