Apache Beam: beam.Flatten doesn't flatten files with SparkRunner, but does so with DirectRunner

Question

Python source code:

import apache_beam as beam
from apache_beam import coders
from apache_beam.options.pipeline_options import PipelineOptions
import glob

#