Really, the hardware level is electrical engineering. Computers are complex clocked synchronous digital electronic circuits. Software is built on this hardware platform.
The EE in me thinks that you might want to consider getting a copy of The Art of Electronics by Horowitz and Hill. A lab manual is available. Possibly more information than you want to know, and definitely a big, fat, heavy book, but also a very good introduction to electronics that does not require advanced math. You shouldn't read it in one sitting. Digital and analog electronics are covered, as well as microprocessors.
The only criticism I have is it is a bit dated -- the integrated circuits discussed are now pretty old. The basics and the theory is still sound and well done, and a new edition may be in the works.
The books website: http://frank.harvard.edu/aoe/
The books Wikipedia page: http://en.wikipedia.org/wiki/The_Art_of_Electronics
Building circuits in the real world great fun, and quite different than computer simulation.
