mesosphere

According to the Spark on Mesos docs one needs to set the spark.executor.uri pointing to a Spark distribution: val conf = new SparkConf() .setMaster("mesos://HOST:5050") .setAppName("My app") .set("spark.executor.uri", "<path to spark-1.4.1.tar.gz uploaded above>") The docs also note that one can build a custom version of the Spark distribution. My question now is whether it is possible/desirable to pre-package external libraries such as spark-streaming-kafka elasticsearch-spark spark-csv which will be used in mostly all of the job-jars I'll submit via spark-submit to reduce the time sbt

I am running Apache Spark on cluster mode using Apache Mesos. But, when I start Spark-Shell to run a simple test command (sc.parallelize(0 to 10, 8).count) I receive the following warning message: 16/03/10 11:50:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources If I check on Mesos WebUI I can see that Spark-Shell is listed as a framework and I have listed one slave (my own machine). Any help how to troubleshoot it? While most of other answers focuses on resource allocation (cores,

What exactly is the difference between Apache's Mesos and Google's Kubernetes? I understand both are server cluster management software. Can anyone elaborate where the main differences are - when would which framework be preferred? Why would you want to use Kubernetes on top of Mesosphere ? Craig Mcluckie Kubernetes is an open source project that brings 'Google style' cluster management capabilities to the world of virtual machines, or 'on the metal' scenarios. It works very well with modern operating system environments (like CoreOS or Red Hat Atomic) that offer up lightweight computing