I have some preferences (and pet peeves) from writing software to control media and display devices using RS232. Depending on your hardware, some of these may not apply:
I think it's a good idea to make your protocol more friendly for automation. If you need an interactive interface (command line or other), build it separately and have it use the automation protocol. I wouldn't worry too much about making it human readable, but it's up to you.
Always return a response, even (especially) if you get an invalid command. Something simple like $06 for ACK and $15 for NAK. Or spell it out if you want it to be slightly more human readable.
If you can set any value, make sure there's some way to query that same value. If you have a lot of values, it could take a while to query them all. Consider having one or just a few meta-queries that return several values at once.
If you have information that can't be set, but is important (model no, serial no, version, copyright, etc), make sure these can be queried instead of just displaying them once on startup or reset.
Never respond with an error for a valid command. You'd think this one would be obvious...
Speaking of obvious, document the serial settings your hardware supports. Especially if it's going to be used by anyone other than you and you don't want them to spend the first 30 minutes trying to figure out if they can't talk to the device because of the serial port, the connections, the cable or their software. Not that I'm bitter...
Use absolute commands instead of toggle values. For example, have separate commands for Power On and Power Off instead of sending the same command and having the power toggle on and off.
Responses should include information on the command they are responding to. This way any program doesn't need to remember the last thing it asked for in order to deal with the response (see extra credit option below).
If your device supports a standby mode (off, but not really off), make sure queries still work while you're in this state.
Depending on how paranoid you are about data completeness:
Wrap your message in an envelope. The header could include a starting character, the lengeth of the message and a closing character. Just in case you get partial or malformed messages. Maybe $02 for the start and $03 for the end.
If you're really paranoid about message integrity, include a checksum. They can be a bit of a pain, though.
For extra credit:
- If your hardware settings can be changed manually, maybe send this change out the serial port as if the user had queried it. For example, you might not want the user to be able to change the input source for a public display monitor.
I hope this helps.
Update:
I forgot something important. Before you either use this seriously and especially before you give it out to someone else, try it out on something trivial to make sure it works the way you expect it to and (more importantly) to make sure you haven't left anything out. It will take more time and effort to fix things if you find a problem in the middle of a bigger project.
This is a good rule of thumb whether you're designing a command protocol, a web service, a database schema or a class, etc.