I would avoid operating systems all together, Embedded Linux is an oxymoron, what are you really learning? You could learn the same thing with your home computer.
Yes, ARM is the king of embedded and may remain so for a while, eventually you will need to take a look.
Sparkfun.com is a great site, they offer a number of the olimex boards plus others like arduino derivatives. I prefer the msp430 to the avr, but the avr is more popular.
Outside sparkfun look at the ez430 from TI (msp430) direct or from digikey or mouser. Stellaris has a number of nice boards, the 811 is easy to brick, my first one lasted only a few hours, oops, I should have read the 17 warning labels stuck to everything in the box, dont touch the enables or direction on those jtag lines. stellaris packed a ton of stuff on the boards, very good platform for learning embedded programming, reading schematics, reading data sheets and programmers reference manuals, learning that all schematics and reference manuals are a little buggy and you have to hack your way through. If you want something to build your own projects on though you have to cut away (not literally) goodies on the boards, so I would go with an olimex header board for adding your own electronics to.
Back to sparkfun. The olimex header boards are good, sub $50, the various arm vendors are represented (except for ti/stellaris). Arduino pro mini for under $20. The armmite pro is good. Avoid the mbed2 and lpcxpresso, perhaps avoid lpc all together (other than the armmite pro).
qemu is also a good simulator, not as easy as the armulator or my thumbulator to dig into to see what your code is doing. But there are many rtoses and linuxes that run on it (arm and other processors are simulated), the arm integrator eval board is a popular target and is simulated by qemu. Just within arm there are many eval boards emulated by qemu.
The beagleboard is popular but I was quite disappointed with my beagleboard and always innovating netbook. It cost another $100 - $150 to make the beagleboard useful. Embest has a much better beagleboard clone, that is useful out of the box. I like the hawkboard.org much better than the beagleboard, slower by a bit, half the price, but a better board. You can work on linux or true embedded (no operating system) or whatever you like. Being TI omap based there is a dsp on chip as well in case you have some interests there.
I am not a fan of pic, and have been and will get flamed for it, doesnt bother me. I recommend learning assembler for the msp430, avr, arm, thumb (and pic), real embedded always involves a little assembler to manage the boot process, interrupt handling and the like at a minimum. The msp430 and arm are well suited instruction sets for C compilers, pic is horrible as is the 8051, that doesnt mean there arent C compilers, it is just horribly inefficient and adding a high level language makes the result that much worse. The pics are quite resource limited. The pic32 is a mips derivative and not what I am talking/complaining about. That is a whole other family and category of device. I would go with the msp430 over pic given the choice both for assembler and C, size, power, features, etc.
Mips is probaby arms biggest competitor, sadly it is the one platform I have not had the opportunity to learn.
8051, 6501, and the lsi11 used by the pdp11 are a nice trip down history lane. the lsi11 is what the C language was invented on and you can see parallels between the assembler and C. The msp430 instruction set has the same feel, both the msp430 and lsi11 are excellent instruction sets for beginning assembler. The 8051 is probably the oldest surviving workhorse, it had its day as the dominant embedded processor, and can still be found in new chip products. like the pic the instruction set is a tower of hanoi puzzle, in and out of the accumulator, in and out, in and out several instructions to do anything useful. sdcc is a free 8051 C compiler and just the right size if you want to dig into the guts of a compiler and have a chance at understanding it (without having to buy books or take classes).
I am a big fan of the gameboy advance and nintendo ds, the gba is easier, emulators are out there for both, although are geared toward playing rom games and not necessarily perfect at emulation you might do for homebrew development. the gba's other than the mini have serial ports making debug that much easier (mini has one but getting at it is harder). start with the gba if you can, much cheaper, much easier, once you get the hang of the the ds is a couple of gbas tied together with some extra complications.
Based on your post, not knowing if the cost is too much, my guess is the hawkboard (hawkboard.org) is a good choice for linux, embedded, algorithms and other items you listed. Go ahead and get the power supply and the otg usb cable if it doesnt come with it. If you are like I was as an EE student that may be too rich, look at the arduino pro or actually the lillypad because it comes with the pins already, no soldering or extra parts required, yes get the $14 usb to serial thing. For the price of the lillypad and usb to serial/power board you could get a armmite pro and not need anything else other than a usb cable (I know it is a lpc, its okay). I have web pages on how to remove the arduino like firmware from both platforms and get at the processor, not brain surgery, pretty easy, but first timers might hesitate. The ez430 falls into this same price category, you dont need anything else but the $20 kit, three more boards cost you $10 for the set. If that is too rich or even if not the emulators are free some are much easier to debug than running on real hardware as you have the source code and can compile in print statements or whatever. You can get your feet wet without expending anything but time, and see if this is something you are really interested in.
Remember even the best emulator is not like being on the real hardware (same as running your code under a debugger). You may have to start your project over on hardware, but that is the fun of embedded. board bring up...
