How do I use a circuit breaker?

Question

I\'m looking for ways to make remote calls to services out of my control until a connect is successful. I also don\'t want to simply set a timer where an action gets executed ev

Answer 1

This is an interesting idea if you have lots of threads hitting the same resource. The way this works is by pooling the count for attempts from all threads. Rather than worrying about writing a loop to try and hit the database 5 times before actually failing, you have the circuit breaker keep track of all attempts to hit the resource.

In one example, you have say 5 threads running a loop like this (pseudo-code):

int errorCount = 0;
while(errorCount < 10) // 10 tries
{
    if(tryConnect() == false)
      errorCount++;
    else
      break;
}

Assuming your error handling is correct and all, this loop could be run 5 times, and ping the resource a total of 50 times.

The circuit breaker tries to reduce the total number of times it attempts to reach the resource. Each thread, or request attempt, will increment a single error counter. Once the error limit is reached, the circuit breaker will not try to connect to it's resource for any more calls on any threads until the timeout has elapsed. It's still the same effect of polling the resource until it's ready, but you reduce the total load.

static volatile int errorCount = 0;

while(errorCount < 10)
{
   if(tryConnect() == false)
      errorCount++;
   else
       break;
}

With this interceptor implementation, the interceptor is being registered as a singleton. So, all instances of your resource class will have code redirected through the circuit breaker first for any call made to any method. The interceptor is just a proxy to your class. It basically overrides your methods and calls the interceptor method first before calling your method.

The Open/Closed bit might be confusing if you don't have any circuit theory knowledge. wiki:

An electric circuit is an "open circuit" if it lacks a complete path between the positive and negative terminals of its power source

In theory, this circuit is Open when the connection is down and Closed when the connection is available. The important part of your example is this:

public void Intercept(IInvocation invocation)
    {
        using (TimedLock.Lock(monitor))
        {
            state.ProtectedCodeIsAboutToBeCalled(); /* only throws an exception when state is Open, otherwise, it doesn't do anything. */
        }

        try
        {
            invocation.Proceed(); /* tells the interceptor to call the 'actual' method for the class that's being proxied.*/
        }
        catch (Exception e)
        {
            using (TimedLock.Lock(monitor))
            {
                failures++; /* increments the shared error count */
                state.ActUponException(e); /* only implemented in the ClosedState class, so it changes the state to Open if the error count is at it's threshold. */ 
            }
            throw;
        }

        using (TimedLock.Lock(monitor))
        {
            state.ProtectedCodeHasBeenCalled(); /* only implemented in HalfOpen, if it succeeds the "switch" is thrown in the closed position */
        }
    }

Answer 2

I've created a library called CircuitBreaker.Net that encapsulates all serving logic to safely perform calls. It's easy to use, an example could look like:

// Initialize the circuit breaker
var circuitBreaker = new CircuitBreaker(
    TaskScheduler.Default,
    maxFailures: 3,
    invocationTimeout: TimeSpan.FromMilliseconds(100),
    circuitResetTimeout: TimeSpan.FromMilliseconds(10000));

try
{
    // perform a potentially fragile call through the circuit breaker
    circuitBreaker.Execute(externalService.Call);
    // or its async version
    // await circuitBreaker.ExecuteAsync(externalService.CallAsync);
}
catch (CircuitBreakerOpenException)
{
    // the service is unavailable, failover here
}
catch (CircuitBreakerTimeoutException)
{
    // handle timeouts
}
catch (Exception)
{
    // handle other unexpected exceptions
}

It's available via a nuget package. You can find the sources on github.