I would be wary of the other solutions offered here. Calling CodeDomProvider.CreateProvider requires finding and parsing the Machine.Config file, as well as your app.config file. That's likely to be several times slower than the time required to just check the string your self.
Instead I would advocate you make one of the following changes:
Cache the provider in a static variable.
This will cause you to take the hit of creating it only once, but it will slow down type loading.
Create the provider directly, by creating a Microsoft.CSharp.CSharpCodeProvider instance your self
This will skip the config file parsing all together.
Write the code to implement the check your self.
If you do this, you get the greatest control over how it's implemented, which can help you optimize performance if you need to. See section 2.2.4 of the C# language spec for the complete lexical grammar for C# identifiers.