Any clue how to join this spark-structured stream joins?

Question

I am using spark-sql-2.4.1 with spark-cassandra-connector-2_11.jar

I am trying to join to streaming datasets as below :

 Dataset&l         


        

Answer 1

AFAIK Spark structured streaming can't do joins after aggregations (or other non-map-like operations)

https://spark.apache.org/docs/2.4.3/structured-streaming-programming-guide.html#support-matrix-for-joins-in-streaming-queries

As of Spark 2.3, you cannot use other non-map-like operations before joins. Here are a few examples of what cannot be used.

• Cannot use streaming aggregations before joins.

I think this is still true in Spark 2.4