spark-streaming

问题 I am using spark-sql 2.4.x version , datastax-spark-cassandra-connector for Cassandra-3.x version. Along with kafka. I have rates meta data of currency sample as below : val ratesMetaDataDf = Seq( ("EUR","5/10/2019","1.130657","USD"), ("EUR","5/9/2019","1.13088","USD") ).toDF("base_code", "rate_date","rate_value","target_code") .withColumn("rate_date", to_date($"rate_date" ,"MM/dd/yyyy").cast(DateType)) .withColumn("rate_value", $"rate_value".cast(DoubleType)) Sales records which i received

问题 I am using spark-sql 2.4.x version , datastax-spark-cassandra-connector for Cassandra-3.x version. Along with kafka. I have rates meta data of currency sample as below : val ratesMetaDataDf = Seq( ("EUR","5/10/2019","1.130657","USD"), ("EUR","5/9/2019","1.13088","USD") ).toDF("base_code", "rate_date","rate_value","target_code") .withColumn("rate_date", to_date($"rate_date" ,"MM/dd/yyyy").cast(DateType)) .withColumn("rate_value", $"rate_value".cast(DoubleType)) Sales records which i received

问题 I want to integrate Spark Streaming with Standalone Solr. I am using Spark 1.6.1 and Solr 5.2 standalone on windows with no Zookeeper configuration. I am able to find some solution where they are connecting to Solr from spark by passing the Zookeeper config. How can I connect my spark program to standalone Solr? 回答1: Please see if this example is helpful http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd From example, you will need to

问题 I use pyspark streaming with enabled checkpoints. The first launch is successful but when restart crashes with the error: INFO scheduler.DAGScheduler: ResultStage 6 (runJob at PythonRDD.scala:441) failed in 1,160 s due to Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6.0 (TID 86, h-1.e-contenta.com, executor 2): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File"/data1/yarn/nm/usercache

问题 Below simple spark program runs absolutely fine, if I run as "sbt run". But if i run, 1) As " spark-submit.cmd eventfilter-assembly-0.1-SNAPSHOT.jar ". where the jar is created using "sbt assembly" with "streaming and sql"'s sbt rule with "% provided". 2) As " spark-submit.cmd --jar play-json_2.10-2.3.10.jar seventfilter_2.10-0.1-SNAPSHOT.jar ". Both cases, it is starting and waiting for new files to come. No problem till now. But as soon as I start putting the files, so that it can be

问题 Below simple spark program runs absolutely fine, if I run as "sbt run". But if i run, 1) As " spark-submit.cmd eventfilter-assembly-0.1-SNAPSHOT.jar ". where the jar is created using "sbt assembly" with "streaming and sql"'s sbt rule with "% provided". 2) As " spark-submit.cmd --jar play-json_2.10-2.3.10.jar seventfilter_2.10-0.1-SNAPSHOT.jar ". Both cases, it is starting and waiting for new files to come. No problem till now. But as soon as I start putting the files, so that it can be

问题 I'm struggling to find the correct packages dependency and their relative version to write to a Redshfit DB with a Pyspark micro-batch approach. What are the correct dependencies to achieve this goal? 回答1: As suggested from AWS tutorial is necessary to provide a JDBC driver wget https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.20.1043/RedshiftJDBC4-no-awssdk-1.2.20.1043.jar After this jar has been downloaded and make it available to the spark-submit command, this is how I provided

问题 I'm struggling to find the correct packages dependency and their relative version to write to a Redshfit DB with a Pyspark micro-batch approach. What are the correct dependencies to achieve this goal? 回答1: As suggested from AWS tutorial is necessary to provide a JDBC driver wget https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.20.1043/RedshiftJDBC4-no-awssdk-1.2.20.1043.jar After this jar has been downloaded and make it available to the spark-submit command, this is how I provided

问题 I am trying to add max and min to each RDD in a spark dstream..each of it's tuple. I wrote the following code, but can't understand how to pass the parameter min and max. Can anyone suggest a way to do this transformation? I tried the following: JavaPairDStream<Tuple2<Long, Integer>, Tuple3<Integer,Long,Long>> sortedtsStream = transformedMaxMintsStream.transformToPair(new Sort2()); class MinMax implements Function<JavaPairRDD<Tuple2<Long, Integer>, Integer>, JavaPairRDD<Tuple2<Long, Integer>,

问题 Scenario: I am doing some testing with spark streaming. The files with around 100 records comes in every 25 seconds. Problem: The processing is taking on average 23 seconds for 4 core pc using local[*] in the program. When i deploy the same app to server with 16 cores i was expecting an improvement in processing time. However, i see it is still taking same time in 16 cores as well (also checked cpu usages in ubuntu and cpu is being fully utilized). All the configurations are default provided