First, learn some basic electrical theory - resistive networks, measuring voltage and current, Ohm's law, and the basics of diodes and transistors.
Buy a few lamps, resistors, LEDs, transistors, and of course wires and wire strippers, electrical tape, batteries and preferably a multimeter. Mess around with some basic circuits. You might like to put some nails in a board and use that to help you out. Making an electromagnets can also be fun, especially for young ones. The components are pretty cheap so don't worry to much about accidentally burning them out - it's just a lesson learned.
The real magic happens when you create a circuit with a few transistors that feed off each other, such that when you flip a switch, a light comes on, and when you unflip the switch, the light stays on.
Once you've reached that point it's probably time to move away from those "low-level" components and start using pre-built chips, like NAND gates and potentially more complex integrated circuits.
An interesting approach from the programming side is to write a small program in assembly language that gets loaded when the computer boots. A "Hello World" operating system, if you will. And then expand it a bit. Use an emulator/debugger such as Bochs to save you time when testing, but then do the fun part: create a bootable floppy disk and let the computer boot off it! You may wish to have a look at this.
