Linux is a general-purpose OS (GPOS); its application to embedded systems is usually motivated by the availability of device support, file-systems, network connectivity, and UI support. All these things can be available in an RTOS, but often with less broad support, or at additional cost or integration effort.
Many RTOS are not full OS in the sense that Linux is, in that they comprise of a static link library providing only task scheduling, IPC, synchronisation timing and interrupt services and little more - essentially the scheduling kernel only. Such a library is linked with your application code to produce a single executable that your system boots directly (or via a bootloader). Most RTOS do not directly support the loading and unloading of code dynamically from a file system as you would with Linux - it is all there at start-up and runs until power down.
Critically Linux is not real-time capable. An RTOS provides scheduling guarantees to ensure deterministic behaviour and timely response events and interrupts. In most cases this is through a priority based pre-emptive scheduling algorithm, where the highest priority task ready to run always runs - immediately - pre-empting any lower priority task without a specific yield or relinquishing of the CPU, or completion of a time-slice.
Linux has a number of scheduling options, including a real-time scheduler, but this is at best "soft" real-time - a term I dislike since it is ill-defined, and essentially means real-time, most of the time, but sometimes not. If your application has no need of "hard" real-time, that's fine, but typical latencies in real-time Linux will be in the order of tens or hundreds of microseconds, whereas a typical RTOS real-time kernel can achieve latencies from zero to a few microseconds.
Another issue with embedded Linux is that it needs significant CPU resources, perhaps >200MIPS, 32bit processor, ideally with an MMU, 4Mb of ROM and 16MB of RAM to just about boot (which may take several seconds). An RTOS on the other hand can be up in milliseconds, run in less than 10Kb, on microcontrollers from 8-bit up. This can have a significant impact on system cost for volume production despite being ostensibly "free".
There are larger RTOS products that exhibit some of the features of a GPOS such as dynamic loading, filesystems, networking, GUI (for example, in QNX), and many RTOS provide a POSIX API (usually secondary to their native real-time API) for example VxWorks and again QNX, so that a great deal of code developed for Linux and Unix can be ported relatively easily. These larger more comprehensive RTOS products remain scalable, so that functionality not required is not included. Linux in comparison has far more limited scalability.
