What are the rules to write robust shell scripts?

Question

I recently erased part of my home directory with a shell script I wrote. Fortunately, I did hit Ctrl-C fast enough to avoid the worst.

My mistake h

Answer 1

General rule: you need to think about the stuff you are doing. Testing and debugging is a way to achieve the success.

Always decompose the problem, regardless of the language you are using. Write small procedures. Test them. Fix if necessary. Write more complicated ones using the small ones. And so one.

Learn about the debugging tools in a given language. In bash the -x option can do a wonders for you. Also, strategically placed echo foo or echo $variable can help you a lot.

If you need to do something potentially destructive (like removing the files) make a dry-run first. Replace all rm with echo and check the results. If something wrong happened, you will see that in the list of files-to-be-removed.

Test every step of data preparation. Even, if you write a one-liner in bash, check the result of each stage, before adding a pipe and another stage. Detect problems early and fix them early.

Make sure, the solution is scalable. If it worked for you for 20 data items, will it worked for 20 thousands?

If your program processes data given by user, make sure it would behave in a sane way even for garbage on input. Reporting an error and exiting is a sane way in such cases.

Aim for simplicity. The simpler the solution, the easier you can avoid mistakes.

And - for sure - I forgot about adding something important here :)

Good luck in your coding!

Answer 2

A couple of ideas:

1. Use -e flag in the shebang, for example #!/bin/sh -e. That way the script will stop at the first error. It's a bit like throwing a RuntimeException in Java. This probably did save my ass a couple of times, and I have a feeling it would helped you too in this case.

2. Handle the exit codes of all the statements in your scripts. Actually using -e in the shebang will force you to do this.

3. Don't chain commands with ;. Use && instead. Again, using -e in the shebang will force you to do this.

4. Properly quote paths that might contain spaces or other special characters.

5. It's best if a script doesn't do dangerous things when used without parameters.

6. For non-trivial scripts, make sure to -h and --help flags that print a helpful message. (I use this script to generate scripts with flag parsing.) This should be mandatory for scripts that can do dangerous things when called without parameters.

7. Exit your scripts with an explicit exit 1 on any non-normal exit. It's a common mistake to handle an error in an if block, echo some helpful message, and then exit without any arguments. Since exit uses the exit code of the last command, in this case echo, it will exit with success. Note that in the sample script I linked earlier, I exit 1 after printing the help message when handling the --help flag.

8. If you don't need bash features, use a #!/bin/sh shebang and try to stay compatible with older versions. Being portable is a kind of robustness, I think.

9. Use $() instead of ``. Easier to read ~~ harder to make mistakes.

10. Format your code nicely and consistently. Easier to read ~~ robustness.

11. Be aware of the differences between platforms. For example both date --iso and date +%F print dates in the format 2013-11-17, but the first only works in GNU systems, the second works in BSD and Solaris too. So use date +%F always, it works everywhere. There are of course 100s of examples like this. If you're doing something you're not used to every day, try to check if it works in a different system too.

12. Test for empty variables, especially in dangerous commands like rm -fr. For example the result of rm -rf "$TOPDIR/$OBJDIR" can be disastrous if TOPDIR or/and OBJDIR happens to be empty or unset. In general, double-check or triple check the possible parameter values of dangerous commands like this.

13. Don't push the limits of shell scripts. Scripts are supposed to be glue-code. If you catch yourself doing something tricky, or that you need obscure features, chances are you're better off moving to more powerful languages.

In the end, none of this will prevent you from making stupid mistakes.

PS: I will keep coming back and add more things as I remember them. Please feel free to suggest improvements and I'll add them. Thanks.