The first place to look would probably be BitTorrent.org. It lists the standard specification as well as a lot of the extensions that have become de facto standards (most of which have been developed by µTorrent or Azereus).
A thorough guide can also be found here, though it might not be as up-to-date as the first link.
Keep in mind that most of the complexity with bittorrent relates to the client implementations, whereas the server ("tracker") is fairly trivial.
