The following code will always throw UnuthorizedAccessException (MemoryStream\'s internal buffer cannot be accessed.)
byte[] buf1 = { 2, 3, 5, 7, 11 }; var
To add to what others have already put in here...
Another way to get your code to work is change your code to the following line.
byte[] buf2 = ms.ToArray();