Too broad question, but some points (related to Linux; the principles should be the same for Windows, but you probably are forbidden to understand all of it) :
The elementary system calls (those listed in syscalls(2)...) are invoked by an elementary machine instruction (e.g. SYSENTER or SYSCALL) which switches the processor into kernel mode (with the system call number and arguments passed through defined registers, following the ABI convention). Hence user-space code can be viewed as running in some virtual machine (defined by user-mode instructions + the system call primitives). BTW the Linux kernel can load kernel modules to e.g. add additional code (such as device drivers) in it, and that is done also thru system calls.
The inter-process communication facilities are built above these system calls (perhaps used by the standard library in higher level functions, e.g. getaddrinfo(3) might interact indirectly with some DNS service, see nsswitch.conf(5)). Read Advanced Linux Programming for more details. In practice you'll need several server programs (and that idea is pushed to its extreme in microkernel approaches), notably (on recent Linux) systemd. Drivers and kernel modules are loaded by specific system calls and later are part of the kernel so are usable thru other system calls. Play with strace(1) to understand the actual system calls done by some Linux program. Some information is provided by the kernel thru pseudo file systems (see proc(5)...) accessible thru system calls.
Every communication from user program to kernel is done by IPC (implemented by system calls). Sometimes, the kernel is doing an upcall to user code (on Linux, with signals).
The Linux framebuffer (and the physical keyboard & mouse) is generally only accessed by a single server which other desktop applications communicate with using usual IPC facilities -sockets-, that server is the X11 or Wayland server.
Read also some good book on Operating Systems, e.g. the freely downloadable Operating Systems: Three Easy Pieces
For Windows, MacOSX, Android, it is very similar. However, since Windows (etc...) is a proprietary software, you might not be able to know all the details (and you might not be allowed to reverse-engineer them). In contrast, Linux is free software, so you can study its source code.
My advice would be to understand in details how Linux work (this would take several years) and study some relevant source code (which is possible for free software). If you need an deep understanding of Windows, you might need to buy some source code license of it (probably millions of dollars) and sign an NDA. I don't know Windows at all, but AFAIK it is only defined by a huge API in C. Rumors tell that the Windows kernel is microkernel like, but Microsoft has economical interest to hide ugly implementation details.
See also osdev.
