Improve performance of slow running structured streaming process that uses micro-batching

Question

I am trying to create an application to process 10 million json files where the size of a json can vary from 1mb to 50mb.

To avoid burdening the driver I am using the