I recommend Granados for SSH stack. It has been used in many products.
I recommend this code project page for telnet stack.
You can also download Poderosa terminal emulator. It's using Granados as the SSH stack and it has its own implementation of telnet stack.
Poderosa is very pluggable. It provides a good abstraction layer on the network stacks. If you don't have time to write your own unified interface, you should really check out their work. It provides access to both SSH and Telnet under the same interface like what you want. Unfortunately, it's written by Japanese engineers. All the inline comments are written in Japanese. It may take you some time to understand the codes.
You may also like to check out the terminal emulation library as well. Since you are going to process the data returning from the remote terminals, most terminals return escape character sequence for device control commands, font color/style or cursor movement commands.
You may not notice that even in a simple command "ls -al", the returned content may also involve some escape character sequences. Those escape sequences are to make the directory name and file name shown in different colors.
Again, for terminal emulation, the best C# library I can find is Poderosa. If you don't have to use C# library, PuTTY has the best terminal emulation support. It can recognize almost all escape character sequences I have seen so far.
Although I highly recommend you to look at Poderosa and Granados, please be aware that these projects seem to be no longer active. Well.. even so, it's still the best I have seen.