How to continue a task when Fabric receives an error

Question

When I define a task to run on several remote servers, if the task runs on server one and exits with an error, Fabric will stop and abort the task. But I want to make fabric

Answer 1

You should set the abort_exception environment variable and catch the exception.

For example:

from fabric.api        import env
from fabric.operations import sudo

class FabricException(Exception):
    pass

env.abort_exception = FabricException
# ... set up the rest of the environment...

try:
    sudo('reboot')
except FabricException:
    pass  # This is expected, we can continue.

You can also set it in a with block. See the documentation here.

Answer 2

As of Fabric 1.5, there is a ContextManager that makes this easier:

from fabric.api import sudo, warn_only

with warn_only():
    sudo('mkdir foo')

Update: I re-confirmed that this works in ipython using the following code.

from fabric.api import local, warn_only

#aborted with SystemExit after 'bad command'
local('bad command'); local('bad command 2')

#executes both commands, printing errors for each
with warn_only():
    local('bad command'); local('bad command 2')

Answer 3

In my case, on Fabric >= 1.4 this answer was the correct one.

You can skip bad hosts by adding this:

env.skip_bad_hosts = True

Or passing the --skip-bad-hosts flag/

Answer 4

You can also set the entire script's warn_only setting to be true with

def local():
    env.warn_only = True

Answer 5

From the docs:

... Fabric defaults to a “fail-fast” behavior pattern: if anything goes wrong, such as a remote program returning a nonzero return value or your fabfile’s Python code encountering an exception, execution will halt immediately.

This is typically the desired behavior, but there are many exceptions to the rule, so Fabric provides env.warn_only, a Boolean setting. It defaults to False, meaning an error condition will result in the program aborting immediately. However, if env.warn_only is set to True at the time of failure – with, say, the settings context manager – Fabric will emit a warning message but continue executing.

Looks like you can exercise fine-grained control over where errors are ignored by using the settings context manager, something like so:

from fabric.api import settings

sudo('mkdir tmp') # can't fail
with settings(warn_only=True):
    sudo('touch tmp/test') # can fail
sudo('rm tmp') # can't fail

Answer 6

In Fabric 1.3.2 at least, you can recover the exception by catching the SystemExit exception. That's helpful if you have more than one command to run in a batch (like a deploy) and want to cleanup if one of them fails.