How can I connect to on-premises TFS using Visual Studio Code? Is that possible the same way as in Visual Studio?
If you need to use Git, all you need is Visual Studio Code. Git is a built-in feature.
In order to also use TFVC you'll need to install an extension. You'll need Visual Studio Code and the Azure Repos Extension and a recent version of Team explorer and/or Team Explorer Command Line Client.
To edit Azure Pipelines (available in Azure DevOps Server 2019), you'll need to also install this Azure Pipelines extension.
The naming is a bit confusing, but these Azure DevOps extension also work with recent version of Team Foundation Server and Azure DevOps Server (new name).