Assuming the following markup:
Radio Buttons
I believe this does it all. You didn't mention that it has to validate, however, so I used the inline-block (-moz-inline-box) display. One of my favorites, actually.
Here's a working copy
Tested in Safari 3, FireFox 3, and IE7.