Why apache spark does not work with java 10? We get illegal reflective then java.lang.IllegalArgumentException

Question

Is there any technical reason why spark 2.3 does not work with java 1.10 (as of July 2018)?

Here is the output when I run SparkPi example using spark-submit

Answer 1

Committer here. It's actually a fair bit of work to support Java 9+: SPARK-24417

It's also almost done and should be ready for Spark 3.0, which should run on Java 8 through 11 and beyond.

The goal (well, mine) is to make it work without opening up module access. The key issues include:

• sun.misc.Unsafe usage has to be removed or worked around
• Changes to the structure of boot classloader
• Scala support for Java 9+
• A bunch of dependency updates to work with Java 9+
• JAXB no longer automatically available