How to write stream to S3 with year, month and day of the day when records were received?

Question

I have a simple streams that reads some data from a Kafka topic:

 val ds = spark
      .readStream
      .format(\"kafka\")
      .option(\"kafka.bootstrap.s         


        

Answer 1

Use partitionBy clause:

import org.apache.spark.sql.functions._

df.select(
    dayofmonth(current_date()) as "day",
    month(current_date()) as "month",
    year(current_date()) as "year",
    $"*")
  .writeStream
  .partitionBy("year", "month", "day")
  ... // all other options