multiple spark application submission on standalone mode

。_饼干妹妹 提交于 2019-12-31 05:31:06

问题


i have 4 spark application (to find wordcount from text file) which written on 4 different language (R,python,java,scala)

./wordcount.R
./wordcount.py
./wordcount.java
./wordcount.scala

spark works in standalone mode... 1.4worker nodes 2.1 core for each worker node 3.1gb memory for each node 4.core_max set to 1

./conf/spark-env.sh

export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=1"

export SPARK_WORKER_OPTS="-Dspark.deploy.defaultCores=1"

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=1g

export SPARK_WORKER_INSTANCES=4

i submitted spark application using pgm.sh file on terminal

./bin/spark-submit  --master spark://-Aspire-E5-001:7077 ./wordcount.R  &

./bin/spark-submit  --master spark://-Aspire-E5-001:7077 ./wordcount.py &

./bin/spark-submit  --master spark://-Aspire-E5-001:7077 ./project_2.jar &

./bin/spark-submit  --master spark://-Aspire-E5-001:7077 ./project_2.jar 

when each process executing individually it takes 2sec. when all process executed using .sh file on terminal it takes 5 sec to 6sec

how do i run different spark applications parallelly? how to assign each spark application to individual core?

来源:https://stackoverflow.com/questions/44558491/multiple-spark-application-submission-on-standalone-mode

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!