How many RDDs does DStream generate for a batch interval?

前端 未结 3 1399
礼貌的吻别
礼貌的吻别 2021-02-19 20:39

Does one batch interval of data generate one and only one RDD in DStream regardless of how big is the quantity of the data?

3条回答
  •  不要未来只要你来
    2021-02-19 21:17

    It's very late to reply to this thread. But still, It's worth adding a few more points. Number of RDDs depends upon how many receivers you have in your application. That's why "sparkContext.read" will have multiple RDDs. But if you have only one receiver or Kafka as a source (receiver-less) in that case you will get only one RDD.

提交回复
热议问题