Spring Cloud: Hystrix

Overview

In this article, we'll introduce you to Spring Cloud Netflix Hystrix. It is a fault tolerance library, which implements the Circuit Breaker enterprise pattern - a pattern designed to prevent cascading failures.

In a typical microservice architecture we have many small applications running separately. It's quite common that one service calls another service (may be more than one) to complete a particular request. There is always a possibility that one of these downstream services won't respond correctly or simply fails completely.

A single crash may result in cascading failures which leads to too many failed network calls, wastage in memory and makes the top service which actually got the request slow.

The principle of the circuit breaker pattern is analogous to electronics:

There are circuit breakers in our houses that watch for any failures. When a failure does occur, it opens the circuit (disconnecting the circuit) and thus isolates the failed area. Once you have rectified the problem, you could manually close the circuit by flipping the switch. This prevents your house from burning down.

In the same way, Hystrix is watching methods for failing calls to downstream services. If a downstream service call is failing more often than allowed, Hystrix will "open the circuit", isolating calls to that service.

We can add a fallback method which will be called if the circuit is open, which allows us to recover the application from the failing state. Hystrix automatically closes the circuit after a certain time, which gives time to the downstream failing service to recover.

Setup

This article assumes that you already have knowledge of Netflix's Eureka project, which is used as a service registry and for load balancing.

Here's how our back-end service looks like:

• Eureka Server: Acts as a service registry and runs on port 8761.
• Recommendation Service: A simple REST service that has a single endpoint of /recommendations and runs on port 8070.
• User Service: A simple REST service that has a single endpoint of /personalized/{id} and runs on port 8060.

The user service internally calls recommendation service to get result for /personalized/{id}:

@RestController
public class UserRestController {

    @Autowired
    RestTemplate restTemplate;

    @GetMapping(value = "/personalized/{id}")
    public Product[] personalized(@PathVariable int id) {
        Product[] result = restTemplate.getForObject("http://recommendation-service/recommendations", Product[].class);
        return result;
    }
}

Let's start all services navigate your browser to the Eureka server at http://localhost:8761/:

Now, let's test user-service endpoint by navigating the browser to the http://localhost:8060/personalized/1 endpoint:

Now, just to see what happens, stop recommendation-service and hit the above endpoint again:

So we got a 500 error response, and this may change depending on the implementation.

It would be bad if this were to happen when an end-user is relying on our application. And what if another service was waiting for a response as well?

To avoid these situations, let's go ahead and implement Hystrix in our application.

Adding Hystrix

To add Hystrix to our user-service, we need to import the appropriate dependency in our pom.xml:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>

In the case you are building a project from scratch you can select this dependency from Spring Initializr:

To enable Hystrix, we have to annotate our main class with either @EnableCircuitBreaker or @EnableHystrix. The former option allows us to implement the circuit breaker pattern with other technologies as well. The latter option allows us to implement the circuit breaker pattern only with Hystrix:

@SpringBootApplication
@EnableDiscoveryClient
@EnableCircuitBreaker
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @LoadBalanced
    @Bean
    RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

Now, we want to scout for adequate places to implement the circuit breaker pattern. Once we find a point that could fail and where we want to break the circuit if need be, we'll annotate it with @HystrixCommand.

This wraps a potentially risky method and allows us to define a fallback and breakpoint for it, using the fallbackMethod attribute.

Let's add this to our .personalized() method:

@GetMapping(value = "/personalized/{id}")
@HystrixCommand(fallbackMethod = "recommendationFallback")
public Product[] personalized(@PathVariable int id) {
    Product[] result = restTemplate.getForObject("http://recommendation-service/recommendations", Product[].class);
    return result;
}

public Product[] recommendationFallback(int id) {
    System.out.println("=======recommendationFallback=========" + id);
    return new Product[0];
}

In the @HystrixCommand annotation, we added an attribute of fallbackMethod, which points to the fallback method we want to run. This attribute has to exactly match our method name, of course.

Notice that the return type and the parameter of the fallback method are the same as the original method. This is because the fallback method must have the same signature as the original. Although, instead of returning any actual data we just return an empty array.

Let's rebuild and start off the user service again. It will work the same as before when the recommendation-service is up.

But if we stop the recommendation-service and hit the user service's /personalized/{id} endpoint again, we will get a very different message:

To configure the @HystrixCommand you can use the commandProperties attribute with a list of @HystrixProperty annotations.

Hystrix Dashboard

Hystrix also provides an optional feature to monitor all of your circuit breakers in a visually-friendly fashion.

Let's create a new project for this dashboard. As always, the best way to start with a skeleton project is to use Spring Initializr.

Select your preferred version of Spring Boot and add the "Hystrix Dashboard" dependency, and generate it as a Maven project:

To enable it we have to add the @EnableHystrixDashboard annotation to our main class:

@SpringBootApplication
@EnableHystrixDashboard
public class HystrixDashboardApplication {
    public static void main(String[] args) {
        SpringApplication.run(HystrixDashboardApplication.class, args);
    }
}

Let's start this service on port 9903 and navigate our browser to the http://localhost:9903/hystrix endpoint:

Adding Hystrix Metrics Stream

To be able to use the Hystrix dashboard, our fallback-enabled microservices need to provide data to the dashboard via a Hystrix Metrics Stream.

To add this in our user service we need to include the actuator dependency to our project:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Also, in your application.properties let's include the stream:

management.endpoints.web.exposure.include= hystrix.stream

Doing so exposes the /actuator/hystrix.stream as a management endpoint.

Monitoring a Hystrix Stream

We need to monitor the user-service hystrix metric stream. For that, let's type in http://localhost:8060/actuator/hystrix.stream in the box and click "Monitor Stream".

You could see many metrics regarding the user service endpoint through this stream.

Monitoring many Hystrix-enabled applications could be challenging and so Spring provided another project called Turbine, which will aggregate streams to present in one Hystrix Dashboard.

Conclusion

In this article, we've covered how to use the Hystrix fallback mechanism to our microservices to prevent cascading failures. We also set up Hystrix Metrics Streams in the Hystrix Dashboard to monitor metrics for our endpoints.

As always, the code for the examples used in this article can be found on GitHub.