Can I use regular Spark Core application library to talk to DSE Spark?

雨燕双飞 提交于 2020-01-25 21:11:29

问题


A couple of days ago we upgraded from DSE 4.6 to DSE 4.7. We're running Spark jobs from Java so I've upgraded spark-core_2.10 Maven dependency from 0.9.1 to 1.2.2 to support the newer Spark 1.2.2 that DSE bundles with. However, when I'm now submitting jobs to master it logs

ERROR [sparkMaster-akka.actor.default-dispatcher-2] 2015-11-17 17:50:42,442  Slf4jLogger.scala:66 - org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 7674242335164700840, local class serialVersionUID = 2812534333379744973
java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 7674242335164700840, local class serialVersionUID = 2812534333379744973
[...]

(see full exception here: https://gist.github.com/JensRantil/371d00701ff7f2021d7d)

Digging further into this, it turns out that DSE ships with spark-core_2.10-1.2.2.2.jar which is

  1. not AFAIK published as a Maven artefact on the Interwebz.
  2. not tagged as released in GIT repository in https://github.com/apache/spark.
  3. shipping with a different serialVersionUID of org.apache.spark.deploy.ApplicationDescription than spark-core_2.10-1.2.2.jar (1.2.2, that is). I have verified this using serialver.

Question: The Datastax Java Spark documentation never mentions anything about importing Spark Core. Should I be able to use the standard Spark Java Maven artifact with DSE?

Additionals:

  • I have added dse.jar to the classpath and that's not the issue.
  • The Datastax Java Spark documentation doesn't state which Spark Code version to use. See related What do I need to import to make `SparkConf` resolvable?.
  • The ApplicationDescription class doesn't fixate its serialVersionUID. Decompiling the ApplicationDescription.class for 1.2.2 and 1.2.2.2, it looks like they both use the same number of fields but have different methods, which leads to serialVersionUID being generated differently. One potential upstream DSE fix would be to simply fixate the serialVersionUID to 7674242335164700840.
  • Somewhat related; To make things even more complicated we have a repackaged spark-core_2.10-1.2.2.jar to shadow a package that broke our application. However, we could probably use the Maven shade plugin for this even if we don't have source code for 1.2.2.2.

来源:https://stackoverflow.com/questions/33782857/can-i-use-regular-spark-core-application-library-to-talk-to-dse-spark

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!