spark-submit

I have recently found a way to use logback instead of log4j in Apache Spark (both for local use and spark-submit ). However, there is last piece missing. The issue is that Spark tries very hard not to see logback.xml settings in its classpath. I have already found a way to load it during local execution: What I have so far Basically, checking for System property logback.configurationFile , but loading logback.xml from my /src/main/resources/ just in case: // the same as default: https://logback.qos.ch/manual/configuration.html private val LogbackLocation = Option(System.getProperty("logback

I am geting the above error when i run a model training pipeline with spark `val inputData = spark.read .option("header", true) .option("mode","DROPMALFORMED") .csv(input) .repartition(500) .toDF("b", "c") .withColumn("b", lower(col("b"))) .withColumn("c", lower(col("c"))) .toDF("b", "c") .na.drop()` inputData has about 25 million rows and is about 2gb in size. the model building phase happens like so val tokenizer = new Tokenizer() .setInputCol("c") .setOutputCol("tokens") val cvSpec = new CountVectorizer() .setInputCol("tokens") .setOutputCol("features") .setMinDF(minDF) .setVocabSize

Related to this question , I got the tip that the getOrCreate idiom should be used to avoid this issues. But trying: JavaStreamingContextFactory contextFactory = new JavaStreamingContextFactory() { @Override public JavaStreamingContext create() { final SparkConf conf = new SparkConf().setAppName(NAME); return new JavaStreamingContext(conf, Durations.seconds(BATCH_SPAN)); } }; final JavaStreamingContext context = JavaStreamingContext.getOrCreate("/tmp/"+NAME, contextFactory); I'm still getting: Exception in thread "main" org.apache.spark.SparkException: Only one SparkContext may be running in

My spark application depends on adam_2.11-0.20.0.jar, every time I have to package my application with adam_2.11-0.20.0.jar as a fat jar to submit to spark. for example, my fat jar is myApp1-adam_2.11-0.20.0.jar, It's ok to submit as following spark-submit --class com.ano.adam.AnnoSp myApp1-adam_2.11-0.20.0.jar It reported Exception in thread "main" java.lang.NoClassDefFoundError: org/bdgenomics/adam/rdd using --jars spark-submit --class com.ano.adam.AnnoSp myApp1.jar --jars adam_2.11-0.20.0.jar My question is how to submit using 2 separate jars without package them together spark-submit -

I'm writing code for spark in java. When I use foreachAsync spark fails and gives me java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. In this code: JavaSparkContext sparkContext = new JavaSparkContext("local","MyAppName"); JavaPairRDD<String, String> wholeTextFiles = sparkContext.wholeTextFiles("somePath"); wholeTextFiles.foreach(new VoidFunction<Tuple2<String, String>>() { public void call(Tuple2<String, String> stringStringTuple2) throws Exception { //do something } }); It works fine. But in this code: JavaSparkContext sparkContext = new JavaSparkContext("local

I have two dataframes which has different types of columns. I need to join those two different dataframe. Please refer the below example val df1 has Customer_name Customer_phone Customer_age val df2 has Order_name Order_ID These two dataframe doesn't have any common column. Number of rows and Number of columns in the two dataframes also differs. I tried to insert a new dummy column to increase the row_index value as below val dfr=df1.withColumn("row_index",monotonically_increasing_id()). But As i am using spark-2, monotonically_increasing_id method is not supporting for me. Is there any way to