In this post on ServerFault, they say you can't do it all through the ssh config file. But you can do it with the ssh config and your .bash_profile or whatever the terminal nerds call it.
in the ssh config file add
Host dev
Hostname server.com
User joe
then in your .bash_profile add an alias
alias domain1="ssh dev -t 'cd domains/domain1; bash'"
Here the dev
refers to what you set up in the config file.
In the Terminal, just type domain1
, you will be asked to put in your password and will go straight to the directory. Make a new alias for all your domains and it will make logging in to each one super easy.