The DOCTYPE tells the consuming user agent (web browsers, web crawlers, validation tools) what type of document the file is. Using it ensures that the consumer correctly parses the HTML as you intended it.
There are several different DOCTYPES for HTML, XHTML, and Framesets and each of these has two modes Strict and Transitional. Strict says that your markup is using the defined standards exactly. See W3C DTDs page for further details.
Quirksmode is basically the layout method from the browser wars days when the standards were much less respected and defined. Generally a standards mode page, that is valid, will layout more consistently across various browsers, but may lack certain features that you require. One such features is the anchor tag's target attribute. The Quirksmode site is a great resource for these differences.
One final thought is that the new HTML5 standard proposes using a very simple DOCTYPE:
<!DOCTYPE html>
Using this DOCTYPE is a forward compatible way to specify that your pages are in standards mode, and are HTML. This is the method that Google uses, and is reasonably easy to remember. I recommend using this DOCTYPE unless you plan to use XHTML.
First of all there is no one doctype you should be using, but most designers try to make it work within XHTML 1.0 Strict.
A doctype is nothing more than a declaration of what tags you can use within your html (though the browsers can use more or less than what is defined) You can actually open up the doctype file and start reading (XHTML 1.0 Strict)
If you do not specify a doctype, the browser will try its best to guess but not always hits the correct type.
Quirks mode is just a technique used by browsers to be backwards compatible, a great example of quirks mode is how IE renders boxes