For learning, you can't go past the AVR. The chips are cheap and they'll run with zero external components - they also supply enough current to drive an LED straight from the port.
You can start with a cheap programmer such as lady-ada's USBTinyISP (USD$22 for a kit) which can power your board with 5V from the USB port. Get the free tools WinAVR (GCC based) and AVRStudio and get a small project working in no time.
Yes the AVRs have limitations - but developing software for microcontrollers is largely about managing resources and coping with those problems. It's unlikely that you'll experience problems such as running out of stack space, RAM or ROM when you're making hobbist projects for powerful ARM platforms.
That said, ARM is also a great platform which is widely used in the industry, however, for learning I highly recommend AVRs.
