"embedded programming" is a very broad term. AVR is pretty well in that category, but it's a step below ARM, in that it's both simpler to use, as well as less powerful.
If you just want to play around with ARM, buy a Nintendo DS or a Gameboy Advance. These are very cheap compared to the hardware inside (wonders of mass production), and they both have free development toolchains based off of gcc which can compile to them.
If you want to play around with embedded linux, BeagleBoard is looking to be a good option, only $150 and it has a ton of features.
Personally I think AVR is best for the smaller-sized 8-bit platforms, and ARM is best for the larger, more powerful 32-bit based platforms. Like many AVR fans, I don't like PIC. It just seems worse in pretty much every way. Also avoid anything that requires you to write any type of BASIC.