Very good question. I've personally spend quite a bit of time on trying to find documentation on the private APIs, and even though I'm pretty sure I've come across it some time ago, right now I can't really find it anymore. However, I do have some answers to your questions. Let me share what I have so far:
Filesystem layout
- An old, but still mostly true guide on filesystem layout.
"Replacing" apps
I think most of the "hijacking" of original iPhone apps is based on catching and re-implementing objective-c messages through posing and comparable techniques. A good guide on this is available here. I'm note entirely sure that this is the mechanism at work though.
Tools
- A toolchain for iphone dev is available on google code. See this guide for info on how to install it.
- See this guide for info on how to use the default apple Xcode environment for development on a jailbroken phone.
