The builder pattern is popular to create immutable objects, but there is some programming overhead to create a builder. So I wonder why not simply using a config object.
Have you considered using builder-builder?
I do think the builder (with prefixes like "With") reads more natrually/fluently.