Sonntag, 19. April 2009

Circuit Breaker Pattern on Google App Engine

This week I came across the Circuit Breaker Pattern, which seems to be a good approach for handling one or more dependencies to external services in cloud/mashup applications.
As applications on GAE tend to have such dependencies, I wrote a memcached and datastore-backed Circuit Breaker Pattern implementation for easy and straightforward usage on GAE Java (sorry, no Python this time - but should actually be easy to adapt and I am glad to post a link here to anyone who migrates the Java implementation).

So what exactly is that Circuit Breaker Pattern I am talking about?!

Think of your GAE application as a washing machine (pretty profane I know) - and think of your external service you like to use with your GAE application as the water you need for getting your clothes clean (thats what a GAE application should do, at last). So if you start your washing machine without the water, your clothes probably won't get much cleaner than they were before you put them into the machine - or worse: your washing machine may break down because its running hot and your clothes burst into flames.
To minimize casualties on that scenario, the Circuit Breaker Pattern comes into play: it basically tracks the health of your external services (the water, electricity and detergent) and trips if they are not healthy or completely unavailable (e.g. your washing machine will only start if all "external services" are available and healthy).

Technical stuff

From a technical perspective, it basically tracks how often a service request failed in the past and prevents you from making unnecessary calls to that service in the future (or act accordingly by informing the visitor to come back later, sending you an email, kicking the service in the butt or [you name it]).
To make this a little more dynamic, the Circuit Breaker Pattern allows you to add some sort of fuzziness to your logic, by not shutting down your whole app when an arbitrary external service fails only once (even though that could be done achieved the pattern) but only after a specific given number of times (the failure threshold).
Also, the Circuit Breaker Pattern defines a time period (timeout) which needs to elapse until the next call to that external service is allowed to get through and check if the service is back up again.

So how does this look in your GAE code?

Java pseudocode:

int failureThreshold = 3;
int defaultTimeout = 5; // seconds
ServiceCircuit ebayCircuit = CircuitBreaker
   .getOrCreateServiceCircuit("ebay", failureThreshold,
    defaultTimeout);

if (CircuitBreaker.isClosed(ebayCircuit)) {
    // circuit is closed,
    // we are allowed to request the ebay service
    
    // Here comes the call to your external service
    boolean success = ?;
    
    if(success) {
        // Service request returned successfully.
        
        // inform the circuit breaker,
        // the request was a success
        CircuitBreaker.success(ebayCircuit);
        
        //do something with your data
        System.out.println(yourData);
    } else {
        // Service failed
       
        // tell the circuit breaker, the request failed
        CircuitBreaker.fail(ebayCircuit);
        
        // inform your visitor
        System.out.println("ebay could not be reached!");
    }
} else {
    // circuit was not closed, so we are not allowed
    // to make a call to
    // our external service for now.
    // We need to wait for the timeout to
    // elapse
    
    // inform your visitor to come back later
    System.out.println("Outage: Please come back later!");
}

Thats more or less all the magic you need to add to your app to make use of the Circuit Breaker Pattern.
To use more than one service, you can either instantiate another circuit, or create a compound circuit (just change the ID from "ebay" to "ebay_and_anotherservice") to track the vitality of both services (depends on how they are needed: sequentially or parallel).

GWT sample application

Beside the code I also wrote a small GAE GWT sample application, which shows you how to use the Circuit Breaker Pattern and comes with some example code:

Code

Download the complete eclipse project. This includes the GWT sample and a whole lot of comments.

The Pattern implementation itself is in two files:
  • /CircuitBreakerExample/src/com/feth/appengine/circuitbreaker/CircuitBreaker.java
  • /CircuitBreakerExample/src/com/feth/appengine/circuitbreaker/ServiceCircuit.java

Please let me know what you think, if the Circuit Breaker is helpful or not, if you have additions or comments or encounter bugs...