Otool usage
Here is official documentation
Here is the question about otool usage:
how to use otool
Detect private api usage
How does Apple know you are using private API?
Detecting Private APIs
Here I have a question. Why do you want to detect private apis? Usually, this question happens when you plan to send your app to AppStore. If so, probably iphone-privateapi tag isn't good fit for the question. Since, this tag is more about how to use private api (and implication that the app won't be send to Appstore)
Documentation on private API
This one is tough. There is no good documentation. Information is scattered and becoming outdate as soon as new version of iOS are released.
Look at these questions to find some interesting links:
iOS Private API Documentation
How to use iPhone SDK Private APIs
https://stackoverflow.com/questions/12734960/some-structured-information-on-ios-internals
