I am a believer in writing a disassembler. Start with a very simple one or two line program that loads a register with a constant perhaps (gotta read a tutorial or something to learn that step). Assemble it. Save the binary in a format you are capable or willing to write a program to read (intel hex perhaps, or elf if they support that).
Write a program to read the binary file and extract the program then take those bytes and write a disassembler (even if the vendor has a disassembler you should still write one).
Now start iterating the process, learn a new instruction or a new way to use that instruction, one instruction at a time. Write code to disassemble that instruction or option. Try to write assembler to manipulate each of the bits in the instruction.
By the time you get through the instruction set you will know the instruction set better than most people that use it every day, you will know how to write assembler for each of the options for each of the opcodes, you may also learn why this instruction can only address N bytes from its location and others can access anything, or that instruction can only use an N bit immediate and others can use any value. That sort of thing.
I have used this process many times and learned many instruction sets, ymmv. After the first couple-three the process above may only take an afternoon to complete.
EDIT:
The goal here is education not the next great sourceforge project. The output can be as ugly or incomplete as you like you are the only one going to read it.
Note: A generic disassembler for variable length instruction sets can be somewhat difficult, you don't want to linearly disassemble the binary in that case you want to follow all the execution paths. I would avoid it. Taking simple programs that execute somewhat linearly assembling then disassembling is not difficult even on a variable length instruction set. You can learn quite a bit about an instruction set by disassembling and examining the output of a C compiler (or other high level language), if the compiler doesn't have an assembler output option or doesn't have a disassembler you may not get to take advantage of this (unless it is a fixed length instruction set).
Also note once you learn assembler for one processor, the second one is much easier, and so on. The things you need to learn from one to the next often become how big can this jump be, what are the rules on immediates, indirect addressing, basically all the things that are directly tied to examining the opcodes. You can learn it without looking at the opcodes, but you have to rely on the documentation or assembler error messages being high quality.
