Why Google Cloud Dataflow doesn't scale to target workers with autoscaling enabled and no quota limitations

Question

I am running a Dataflow pipeline to parse about 45000 text files stored in a Cloud storage bucket. The parsed text is transformed into JSON and written to text files for subsequ