Why is my spark structured streaming job not consistent at speed when streaming from a folder

Question

I\'m running a spark structured streaming job to do the following:

1. Stream a folder from S3 containting json files (gzip)
2. Call a HTTP endpoint