spark-streaming-kafka

问题 I am doing some transformation on the spark structured streaming dataframe. I am storing the transformed dataframe as parquet files in hdfs. Now I want that the write to hdfs should happen in batches instead of transforming the whole dataframe first and then storing the dataframe. 回答1: Here is a parquet sink example: # parquet sink example targetParquetHDFS = sourceTopicKAFKA .writeStream .format("parquet") # can be "orc", "json", "csv", etc. .outputMode("append") # can only be "append"

问题 I am doing some transformation on the spark structured streaming dataframe. I am storing the transformed dataframe as parquet files in hdfs. Now I want that the write to hdfs should happen in batches instead of transforming the whole dataframe first and then storing the dataframe. 回答1: Here is a parquet sink example: # parquet sink example targetParquetHDFS = sourceTopicKAFKA .writeStream .format("parquet") # can be "orc", "json", "csv", etc. .outputMode("append") # can only be "append"

问题 I have a streaming job that writes data into Kafka and I've noticed one of Kafka partitions (#3) takes more data then others. +-----------------------------------------------------+ | partition | messages | earlist offset | next offset| +-----------------------------------------------------+ |1 | 166522754 | 5861603324 | 6028126078 | |2 | 152251127 | 6010226633 | 6162477760 | |3 | 382935293 | 6332944925 | 6715880218 | |4 | 188126274 | 6171311709 | 6359437983 | |5 | 188270700 | 6100140089 |

问题 I have a streaming job that writes data into Kafka and I've noticed one of Kafka partitions (#3) takes more data then others. +-----------------------------------------------------+ | partition | messages | earlist offset | next offset| +-----------------------------------------------------+ |1 | 166522754 | 5861603324 | 6028126078 | |2 | 152251127 | 6010226633 | 6162477760 | |3 | 382935293 | 6332944925 | 6715880218 | |4 | 188126274 | 6171311709 | 6359437983 | |5 | 188270700 | 6100140089 |

问题 I have a Spark Structured Streaming consuming records from Kafka topic with 2 partition. Spark Job: 2 queries, each consuming from 2 separate partition, running from same spark session. val df1 = session.readStream.format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("assign", "{\"multi-stream1\" : [0]}") .option("startingOffsets", latest) .option("key.deserializer", classOf[StringDeserializer].getName) .option("value.deserializer", classOf[StringDeserializer]

问题 I have a Spark Structured Streaming consuming records from Kafka topic with 2 partition. Spark Job: 2 queries, each consuming from 2 separate partition, running from same spark session. val df1 = session.readStream.format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("assign", "{\"multi-stream1\" : [0]}") .option("startingOffsets", latest) .option("key.deserializer", classOf[StringDeserializer].getName) .option("value.deserializer", classOf[StringDeserializer]

问题 I would like to know about the unit testing side of Spark Structured Streaming. My scenario is, I am getting data from Kafka and I am consuming it using Spark Structured Streaming and applying some transformations on top of the data. I am not sure about how can I test this using Scala and Spark. Can someone tell me how to do unit testing in Structured Streaming using Scala. I am new to streaming. 回答1: tl;dr Use MemoryStream to add events and memory sink for the output. The following code

来源： https://stackoverflow.com/questions/58723314/pyspark-failed-to-find-data-source-kafka

来源： https://stackoverflow.com/questions/58723314/pyspark-failed-to-find-data-source-kafka

来源： https://stackoverflow.com/questions/58723314/pyspark-failed-to-find-data-source-kafka