How do I use a circuit breaker?

问题

I'm looking for ways to make remote calls to services out of my control until a connect is successful. I also don't want to simply set a timer where an action gets executed every n seconds/minutes until successful. After a bunch of research it appears that the circuit breaker pattern is a great fit.

I found an implementation that uses an Castle Windsor interceptor, which looks awesome. The only problem is I don't know how to use it. From the few articles I found regarding the topic the only usage example I was able to find was to simply use the circuit breaker to call an action only once, which doesn't seem very useful. From that it seems I need to simply run my action using the circuit breaker in a while(true) loop.

How do I use the Windsor interceptor to execute an action making a call to an external service until it is successful without slamming our servers?

Could someone please fill in the missing pieces?

Here is what I was able to come up with

while(true)
{
    try
    {
        service.Subscribe();
        break;
    }
    catch (Exception e)
    {
        Console.WriteLine("Gotcha!");

        Thread.Sleep(TimeSpan.FromSeconds(10));
    }
}

Console.WriteLine("Success!");

public interface IService
{
    void Subscribe();
}

public class Service : IService
{
    private readonly Random _random = new Random();

    public void Subscribe()
    {
        var a = _random.Next(0, 10) % 2421;
        if(_random.Next(0, 10) % 2 != 0)
            throw new AbandonedMutexException();
    }
}

Based on that I think I now understand this concept as well as how to apply it.

回答1:

This is an interesting idea if you have lots of threads hitting the same resource. The way this works is by pooling the count for attempts from all threads. Rather than worrying about writing a loop to try and hit the database 5 times before actually failing, you have the circuit breaker keep track of all attempts to hit the resource.

In one example, you have say 5 threads running a loop like this (pseudo-code):

int errorCount = 0;
while(errorCount < 10) // 10 tries
{
    if(tryConnect() == false)
      errorCount++;
    else
      break;
}

Assuming your error handling is correct and all, this loop could be run 5 times, and ping the resource a total of 50 times.

The circuit breaker tries to reduce the total number of times it attempts to reach the resource. Each thread, or request attempt, will increment a single error counter. Once the error limit is reached, the circuit breaker will not try to connect to it's resource for any more calls on any threads until the timeout has elapsed. It's still the same effect of polling the resource until it's ready, but you reduce the total load.

static volatile int errorCount = 0;

while(errorCount < 10)
{
   if(tryConnect() == false)
      errorCount++;
   else
       break;
}

With this interceptor implementation, the interceptor is being registered as a singleton. So, all instances of your resource class will have code redirected through the circuit breaker first for any call made to any method. The interceptor is just a proxy to your class. It basically overrides your methods and calls the interceptor method first before calling your method.

The Open/Closed bit might be confusing if you don't have any circuit theory knowledge. wiki:

An electric circuit is an "open circuit" if it lacks a complete path between the positive and negative terminals of its power source

In theory, this circuit is Open when the connection is down and Closed when the connection is available. The important part of your example is this:

public void Intercept(IInvocation invocation)
    {
        using (TimedLock.Lock(monitor))
        {
            state.ProtectedCodeIsAboutToBeCalled(); /* only throws an exception when state is Open, otherwise, it doesn't do anything. */
        }

        try
        {
            invocation.Proceed(); /* tells the interceptor to call the 'actual' method for the class that's being proxied.*/
        }
        catch (Exception e)
        {
            using (TimedLock.Lock(monitor))
            {
                failures++; /* increments the shared error count */
                state.ActUponException(e); /* only implemented in the ClosedState class, so it changes the state to Open if the error count is at it's threshold. */ 
            }
            throw;
        }

        using (TimedLock.Lock(monitor))
        {
            state.ProtectedCodeHasBeenCalled(); /* only implemented in HalfOpen, if it succeeds the "switch" is thrown in the closed position */
        }
    }

回答2:

I've created a library called CircuitBreaker.Net that encapsulates all serving logic to safely perform calls. It's easy to use, an example could look like:

// Initialize the circuit breaker
var circuitBreaker = new CircuitBreaker(
    TaskScheduler.Default,
    maxFailures: 3,
    invocationTimeout: TimeSpan.FromMilliseconds(100),
    circuitResetTimeout: TimeSpan.FromMilliseconds(10000));

try
{
    // perform a potentially fragile call through the circuit breaker
    circuitBreaker.Execute(externalService.Call);
    // or its async version
    // await circuitBreaker.ExecuteAsync(externalService.CallAsync);
}
catch (CircuitBreakerOpenException)
{
    // the service is unavailable, failover here
}
catch (CircuitBreakerTimeoutException)
{
    // handle timeouts
}
catch (Exception)
{
    // handle other unexpected exceptions
}

It's available via a nuget package. You can find the sources on github.

来源：https://stackoverflow.com/questions/7675535/how-do-i-use-a-circuit-breaker

标签

design-patterns

dependency-injection

interceptor

circuit-breaker