Retry a Bash command with timeout

Question

How to retry a bash command until its status is ok or until a timeout is reached?

My best shot (I\'m looking for something simpler):

NEXT_WAIT_TIME=0         


        

Answer 1

Put together some tools.

retry: https://github.com/kadwanev/retry

timeout: http://manpages.courier-mta.org/htmlman1/timeout.1.html

Then see the magic

retry timeout 3 ping google.com

PING google.com (173.194.123.97): 56 data bytes
64 bytes from 173.194.123.97: icmp_seq=0 ttl=55 time=13.982 ms
64 bytes from 173.194.123.97: icmp_seq=1 ttl=55 time=44.857 ms
64 bytes from 173.194.123.97: icmp_seq=2 ttl=55 time=64.187 ms
Before retry #1: sleeping 0.3 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=56.549 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=60.220 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=8.872 ms
Before retry #2: sleeping 0.6 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=25.819 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=16.382 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=3.224 ms
Before retry #3: sleeping 1.2 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=58.438 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=94.828 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=61.075 ms
Before retry #4: sleeping 2.4 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=43.361 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=32.171 ms
...

Check exit status for ultimate pass/fail.

Answer 2

retry fuction is from:

http://fahdshariff.blogspot.com/2014/02/retrying-commands-in-shell-scripts.html

#!/bin/bash

# Retries a command on failure.
# $1 - the max number of attempts
# $2... - the command to run
retry() {
    local -r -i max_attempts="$1"; shift
    local -r cmd="$@"
    local -i attempt_num=1

    until $cmd
    do
        if (( attempt_num == max_attempts ))
        then
            echo "Attempt $attempt_num failed and there are no more attempts left!"
            return 1
        else
            echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
            sleep $(( attempt_num++ ))
        fi
    done
}

# example usage:
retry 5 ls -ltr foo

if you want to retry an function in your script, you should do like this:

# example usage:
foo()
{
   #whatever you want do.
}

declare -fxr foo
retry 3 timeout 60 bash -ce 'foo'

Answer 3

One line and shortest, and maybe the best approach:

timeout 12h bash -c 'until ssh root@mynewvm; do sleep 10; done'

Credited by http://jeromebelleman.gitlab.io/posts/devops/until/

Answer 4

You can simplify things a bit by putting command right in the test and doing increments a bit differently. Otherwise the script looks fine:

NEXT_WAIT_TIME=0
until [ $NEXT_WAIT_TIME -eq 5 ] || command; do
    sleep $(( NEXT_WAIT_TIME++ ))
done
[ $NEXT_WAIT_TIME -lt 5 ]

Answer 5

I made some tweaks to this answer which let you switch on whether the timeout was reached, or whether the command succeed. Also, in this version there is a retry every second:

ELAPSED=0
started=$(mktemp)
echo "False" > $started
until the_command_here && echo "True" > $started || [ $ELAPSED -eq 30 ]
do
   sleep 1
   (( ELAPSED++ ))
done

if [[ $(cat $started) == "True" ]]                                                                                                                                                                                                                            
then                                                                                                                    
    echo "the command completed after $ELAPSED seconds"                                                                                              
else                                                                                                                    
    echo "timed out after $ELAPSED seconds"                                                                               
    exit 111                                                                                                            
fi