I am trying to understand the purpose behind CMake. Why it is designed like it is right now. Here are some questions I would like to have answered.
It can be difficult to understand how CMake works. I'll do my best to briefly answer some of the questions you posted.
Why does cmake generate makefiles instead of just building the project?
CMake is a cross-platform make system and is compiler independent. It doesn't generate just make build systems, but also generates Visual Studio solution files. With the same CMakeList.txt file someone can easily create a windows build (Visual Studio) or a Linux build (g++).
Why are cmake files [a series of] commands and not just configuration files eg: ini/xml/yaml
Compilers don't use a universal configuration, so CMake must adapt for the target compiler.
CMake files are a series of commands to generate the appropriate configuration for the target compiler.
What are the commands that I write into the CMakeLists.txt supposed to do? Just calling the compiler would be too easy I guess
They are suppose to generate the appropriate configuration for the target compiler, and they will if written correctly.
In which order am I supposed to do the commands?
In the order that matters, if the order matters at all. Yes it's a vague answer, but it depends on the project. For example the include() command will be used to add additional cmake files, and if you include them in the wrong order, it can break generation of the build system.
The first command in your cmake file must be the minimum required version with the latest version of CMake (3.0.1). From there it depends. :)
For example this command will not work with versions of CMake less than 2.6.
cmake_minimum_required (VERSION 2.6)
See some of the tutorial links at the end of this answer.
Is everything case insensitive? Can I write everything lower case?
As stated on the CMake wiki's language syntax page, all of the commands are case insensitive, but you must consider case for contents passed to a command.
Why do tutorials advise me to list every source file explicitly?
Because you have to list every source file explicitly. This can be done inside a cmake file or a separate file list that is referenced in the cmake file. Please see Antonio's answer for a caveat of using a separate file list.
CMake does provide a command aux_source_directory
which collects the names of all the source files in a specified directory and stores the list to a variable, but it should only be used for "generated" source files for the same reason mentioned in Antonio's answer.
How do I structure my CMakeLists.txt to keep it short and simple to maintain. Every file I looked up in real projects looked very cluttered.
It's very easy to clutter a CMake file, and one way to help with this is to use .cmake files that you reference using the include command.
A good resource (although still under development) can be found on the CMake about page.
It's broken up into several sections:
Tutorials
CMake Tutorial - From the official CMake documentation page
Getting Started With CMake - A tutorial that uses CMake with gcc.
Using CMake To Build QT Projects - A tutorial that uses CMake with QT.
Since jmstoker gave already a very complete answer, I'll focus on this specific point:
"Why do tutorials advise me to list every source file explicitly?"
In practice you can use globbing functions, however the problem is that if you add a file, then you have to run cmake explicitly to make cmake aware of its presence. It's usually not a problem if you are working "alone", but in a shared project it can be a problem if somebody add a file, you update, and then you compile facing strange errors.
With git there is a solution, that allow triggering "cmake ." every time a merge happens. Getting cmake to run before building after pulling from git