That does seem like a weird behavior at first, but try setting the navigationItem.largeTitleDisplayMode to always. The default is automatic - and it's not defined how that works in the docs.
Also wrote / will update an answer about large titles here.