I created a 57×57 circular icon (without shine) which has transparency outside of the circle for my app. I can successfully install the app on Simulator and iPhone. It works fi
I've looked into this for a client who would like a circular transparent icon to match his company's logo. Here's what I've found:
The 512x512 iTunes icon is uploading in JPEG or TIFF format, which don't support transparency, so you'd have to make up a square background for your circle. Since the home screen icon is supposed to look exactly like the large icon, your home screen icon will also have to have the same background.
As brandon suggests you could use a pre-rendered icon with a black background, but you'd be at the mercy of the App Store reviewers. The docs say your home screen icon must be a "flattened, square-image icon that is 57x57 pixels" (my emphasis). You might get away with a circular icon with a black background, but it really depends on how the reviewer tests that requirement. You might get away with it if they don't check, or only check that the alpha channel is 100% everywhere, but won't if they decide based on whether the icon looks square.
A circular icon will look odd next to all the other icons on the users' phones, so it's probably a bad idea anyway.