Multiple aggregations in Spark Structured Streaming

Question

I would like to do multiple aggregations in Spark Structured Streaming.

Something like this:

• Read a stream of input files (from a folder)
• Per

Answer 1

This is not supported, but there are other ways also. Like performing single aggregation and saving it to kafka. Read it from kafka and apply aggregation again. This has worked for me.

Answer 2

For spark 2.2 and above (not sure about earlier version), if you can design the aggregation to use flatMapGroupWithState with append mode, you can do as many aggregations as you want. The restriction is mentioned here Spark structured streaming - Output mode