As usual the answer is that it depends... By itself, no, VB.Net is not slower than C#, at least nothing that you will notice. Yes, there will be slight differences in compiler optimization, but IL generated will be essentially the same.
However, VB.Net comes with a compatibility library for programmers used to VB6. I am remembering about those string methods like left, right, mid, old VB programmers would expect. Those string manipulation functions are slower. I'm not sure you would notice an impact, but depending on the intensity of their use, I'd bet the answer would be yes. Why are those methods slower than "native" .net string methods? Because they are less type-safe. Basically, you can throw almost anything at them and they will try to do what you want them to, just like in old VB6.
I am thinking about string manipulation, but if I think harder, I'm sure I'll remember about more methods thrown into that compatibility layer (I don't remember the assembly's name, but remember it is referenced by default in VB.Net) that would have a performance impact if used instead of their .net "native" equivalent.
So, if you keep on programming like you were in VB6, then you might notice an impact. If not, it's ok.