Cleanest way to write retry logic?

Question

Occasionally I have a need to retry an operation several times before giving up. My code is like:

int retries = 3;
while(true) {
  try {
    DoSomething();
         


        

Answer 1

This is possibly a bad idea. First, it is emblematic of the maxim "the definition of insanity is doing the same thing twice and expecting different results each time". Second, this coding pattern does not compose well with itself. For example:

Suppose your network hardware layer resends a packet three times on failure, waiting, say, a second between failures.

Now suppose the software layer resends an notification about a failure three times on packet failure.

Now suppose the notification layer reactivates the notification three times on an notification delivery failure.

Now suppose the error reporting layer reactivates the notification layer three times on a notification failure.

And now suppose the web server reactivates the error reporting three times on error failure.

And now suppose the web client resends the request three times upon getting an error from the server.

Now suppose the line on the network switch that is supposed to route the notification to the administrator is unplugged. When does the user of the web client finally get their error message? I make it at about twelve minutes later.

Lest you think this is just a silly example: we have seen this bug in customer code, though far, far worse than I've described here. In the particular customer code, the gap between the error condition happening and it finally being reported to the user was several weeks because so many layers were automatically retrying with waits. Just imagine what would happen if there were ten retries instead of three.

Usually the right thing to do with an error condition is report it immediately and let the user decide what to do. If the user wants to create a policy of automatic retries, let them create that policy at the appropriate level in the software abstraction.