Can I have multiple spark versions installed in CDH?

Question

I\'m using cdh5.1.0, which already has default spark installed. However, I want to use Spark 1.3. Can I also install this version to cdh5.1.0? How is it possible to set thes

Answer 1

Yes, you can run any Apache Spark version you like. Just make sure it's built for the version of YARN you have (2.3 for CDH 5.1.0). You can then run your application as a YARN application with spark-submit. (See http://spark.apache.org/docs/latest/running-on-yarn.html.) It will be monitored like any other YARN application.

Spark doesn't need to be "installed" at all, you just run your application.

Answer 2

Please execute below commands from your console, with sudo access and password....it will update your spark current version to latest version

sudo yum install spark-core spark-master spark-worker spark-history-server spark-python
sudo yum install spark-core spark-history-server spark-python