I created a 57×57 circular icon (without shine) which has transparency outside of the circle for my app. I can successfully install the app on Simulator and iPhone. It works fi
For the app, you can use your transparent PNG as the icon. When you upload to the itunes store, they require a .jpg which of course has no transparency.
As for if they will accept your app with a substandard shaped icon, maybe, maybe not. The could approve it one time and then reject an upgrade, it will probably depend if they are paying attention and notice the icon or not.