What will happen if I do the following:
3GS and iPhone 4 will both display the image improperly.
The fact is 3GS won't scale down the image. So it won't display the image properly. iPhone 4 will scale the non-@2x image (scale it twice) so it won't display the image properly too.
In 99% of cases this will work fine. However there are cases where images won't scale correctly (usually depending on the contentMode). I worked with someone who did this exact thing and never noticed a problem, although I think it is terrible practice.
You will get a lighter binary, but you are asking the older devices (with less memory) to do more work. If you don't want to support these devices then don't, I think this is better than giving users a potentially broken app.
There are better ways to reduce the size of images in your app, reusing them wherever possible, stretching, tiling etc.
Nothing will happen if you decide to do this. Older devices will try, but if you allow the device to even run your app, you will have quite a bit of work just to deal with the scaling issues. If you allow older devices to install your app, you will have to be prepared to get approved from Apple on older devices too. If the images don't work right on older devices, you won't get approved. You are also right that there are fewer and fewer retina displayed devices in service.
But... there are still quite a few non-retina devices still in service. Maintaining both is good for your users. Yes it adds a little extra heft to your binary, but with today's speeds on a variety of networks, that isn't the issue like it used to be.
While it is your right to choose which users to support, and there are many developers that share your sentiment, it is still up to you to decide which group of users you ultimately want to support. If you are fine drawing the line with retina displays, so be it.
I could finger-wag at you and tell you that you should support every single user, but I'm sure you've thought of that. If you are fine supporting retina devices only, go for it. If you are prepared to answer questions on why this one device is supported and another isn't, go for it. The good news is, no matter what you decide, you'll be right...eventually. Good luck.