You have an ambitious goal. But execution is key.
Most of the structured approaches (textbook or college class) will walk you through the process, but they supply a lot of the nitty-gritty code that glosses over the arcane details of your chosen platform and lets you focus on the big-picture ideas: process scheduling, memory management, deadlock prevention, I/O, and so forth.
My advice is this: cut way back on your expectations and start with a basic question.
What is an operating system?
A computer scientist will (hopefully) never say that an OS is a graphical user interface, or a web browser, or a way to hook up USB devices, or anything that a user can actually see or touch. Instead, an OS at its most fundamental level are those things I mentioned above. They all fall under one big umbrella: resource management.
An operating system is nothing more than a program that manages the hardware resources of the computer: memory, CPU, and peripherals.
Here's a simple operating system: a program lets the user type in a program (in hexadecimal or binary) using a serial connection. Once the program has been typed in, it runs the program. When the program is finished, control is returned to the user where they can either run the program again or type in a new one.
Do this on a "clean" architecture such as an embedded ARM processor with 64K of memory or so. You can code this up in assembly after a few days of learning the ins and outs of the ARM. And voila!, you've got yourself an operating system.
It does everything an OS is supposed to do:
- It manages memory by not letting the user overwrite the OS itself.
- It schedules a single process to run.
- It handles the I/O to the single serial peripheral.
This gives you a building block to start from. You have many choices now. Perhaps one of them would be to allow two programs to be loaded into memory and let the user decide which one to run next.
Or you could let the user suspend execution of one program, switch to the other, suspend, and switch back. This is rudimentary multitasking, even though it's entirely manual.
Your choices are unlimited but each one is a baby step from what you had before.
It's fun if you don't set your sights too high!