In addition to the Express tools that have already been mentioned, you'll also want IIS for web development. That comes included in the various pro/business versions of Windows.
The MSDN documentation is available online for free as well.
The main difference between the free Express and the paid versions of Visual Studio is that you don't get support for plug-ins, some debugging tools, some projects (such as Windows services), Crystal Reports, installers, and other tool which are useful in business, but aren't important for learning.
After you've learned C#, if you need features that aren't in the free versions, you can buy the tools you need separately from the MSDN and save some cash. Visual Studio standard is about $250 and SQL Developer is about $50. The primary advantage of the MSDN is that it gives you all of the tools you need for development using the Microsoft stack. Depending on the type of MSDN (pdf) you get desktop and server OSes, Office, Visual Studio, and various server applications. Those can be very useful, but certainly aren't needed for learning.
Microsoft also provides 30+ day evaluations of many of their professional tools. Windows Server, Visual Studio, SQL, Sharepoint, Expression
In my opinion, there is absolutely no reason you need an MSDN subscription to learn C# well. But it is useful for a professional developer who focuses on Microsoft tools.
