Monitoring Spring Boot Apps with Micrometer, Prometheus, and Grafana

Introduction

Monitoring an application's health and metrics helps us manage it better, notice unoptimized behavior and get closer to its performance. This especially holds true when we're developing a system with many microservices, where monitoring each service can prove to be crucial when it comes to maintaining our system.

Based on this information, we can draw conclusions and decide which microservice needs to scale if further performance improvements can't be achieved with the current setup.

In this article, we'll cover how to monitor Spring Boot web applications. We will be using three projects to achieve this:

• Micrometer: Exposes the metrics from our application
• Prometheus: Stores our metric data
• Grafana: Visualizes our data in graphs

This might look like a lot, especially compared to just using the Spring Boot Actuator project, but it's very easy to implement all of them with just a few configurations.

To make things even easier, we'll be using Docker to run Prometheus and Grafana since they both have official Docker images. If you're not familiar with Docker, you can check out our article Docker: A High-Level Introduction.

Please note that these metrics will give you aggregated information over an interval of time. If you want to check information about an individual request at a particular time and what happened to it, then this might not be the solution for you.

In that case, you probably need a distributed tracing system which we have covered in detail in Distributed Tracing with Sleuth.

Spring Boot Actuator

We'll start off with a simple REST service using Spring Initializr that contains a single endpoint of /hello and running on the default port of 8080.

Besides, this application also has the spring-boot-starter-actuator dependency, which provides production-ready endpoints that you can use for your application. These endpoints fall under a common prefix of /actuator and are, by default, protected.

Expose them individually, or all at once, by adding the following properties in application.properties:

management.endpoints.web.exposure.include=*

To check, let's navigate our browser to http://localhost:8080/actuator:

You can see all the endpoints that Actuator exposes such as /health, /metrics, /mappings, etc. Let's open up the /metrics endpoint of the Actuator by navigating our browser to http://localhost:8080/actuator/metrics:

As you can see, there's a bunch of information about our application here, such as information about threads, Tomcat sessions, classes, the buffer, etc. Let's go deeper and retrieve information about the JVM memory used:

Now, using the Spring Boot Actuator like this does yield a lot of information about your application, but it's not very user-friendly. It can be integrated with Spring Boot Admin for visualization, but it has its limitations and is less popular.

Tools like Prometheus, Netflix Atlas, and Grafana are more commonly used for the monitoring and visualization and are language/framework-independent.

Each of these tools has its own set of data formats and converting the /metrics data for each one would be a pain. To avoid converting them ourselves, we need a vendor-neutral data provider, such as Micrometer.

Micrometer

To solve this problem of being a vendor-neutral data provider, Micrometer came to be. It exposes Actuator metrics to external monitoring systems such as Prometheus, Netflix Atlas, AWS CloudWatch, and many more.

They quite correctly describe themselves as:

Think SLF4J, but for metrics.

Just as a refresher, SLF4J is a logging facade for other Java logging frameworks. SLF4J itself does not have any logging implementation. The idea is that you write code using SLF4J API and the real implementation of it comes from the framework you choose. It could be any of the popular frameworks like Log4j, Logback, etc.

Similarly, Micrometer automatically exposes /actuator/metrics data into something your monitoring system can understand. All you need to do is include that vendor-specific micrometer dependency in your application.

Micrometer is a separate open-sourced project and is not in the Spring ecosystem, so we have to explicitly add it as a dependency. Since we will be using Prometheus, let's add it's specific dependency in our pom.xml:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Rebuild and start the application and navigate our browser to http://localhost:8080/actuator:

This will generate a new endpoint - /actuator/prometheus. Opening it, you will see data formatted specific for Prometheus:

Prometheus

Prometheus is a time-series database that stores our metric data by pulling it (using a built-in data scraper) periodically over HTTP. The intervals between pulls can be configured, of course, and we have to provide the URL to pull from. It also has a simple user interface where we can visualize/query on all of the collected metrics.

Let's configure Prometheus, and more precisely the scrape interval, the targets, etc. To do that, we'll be using the prometheus.yml file:

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'spring_micrometer'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.2.8:8080']

As you can see, we have a scrape_configs root key where we can define a list of jobs and specify the URL, metrics path, and the interval. If you'd like to read more about Prometheus configurations, feel free to visit the official documentation.

Note: Since we are using Docker to run Prometheus, it will be running in a Docker network that won't understand localhost/120.0.01, as you might expect. Since our app is running on localhost, and for the Docker container, localhost means its own network, we have to specify our system IP in place of it.

So instead of using locahost:8080, 192.168.2.8:8080 is used where 192.168.2.8 is my PC IP at the moment.

To check your system IP you can run ipconfig or ifconfig in your terminal, depending upon your OS.

Now, we can run Prometheus using the Docker command:

$ docker run -d -p 9090:9090 -v <path-to-prometheus.yml>:/etc/prometheus/prometheus.yml prom/prometheus

<path-to-prometheus.yml> is where your own prometheus.yml is starting from the root. For an example, this works on my local Windows PC:

$ docker run -d -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

To see Prometheus dashboard, navigate your browser to http://localhost:9090:

To check if Prometheus is actually listening to the Spring app, you can go to the /targets endpoint:

Let's go back to the home page and select a metric from the list and click Execute:

Prometheus Query Language - PromQL

Another thing to note is - Prometheus has its own query language called PromQL. It allows the user to select and aggregate time series data in real-time, storing it either in graph or tabular format. Alternatively, you can feed it to an external API through HTTP.

If you'd like to read more about PromQL, the official documentation covers it quite nicely.

Grafana

While Prometheus does provide some crude visualization, Grafana offers a rich UI where you can build up custom graphs quickly and create a dashboard out of many graphs in no time. You can also import many community built dashboards for free and get going.

Grafana can pull data from various data sources like Prometheus, Elasticsearch, InfluxDB, etc. It also allows you to set rule-based alerts, which then can notify you over Slack, Email, HipChat, and similar.

Let's start off by running Grafana using Docker:

$ docker run -d -p 3000:3000 grafana/grafana

If you visit http://localhost:3000, you will be redirected to a login page:

The default username is admin and the default password is admin. You can change these in the next step, which is highly recommended:

Since Grafana works with many data sources, we need to define which one we're relying on. Select Prometheus as your data source:

Now, add the URL that Prometheus is running on, in our case http://localhost:9090 and select Access to be through a browser.

At this point, we can save and test to see if the data source is working correctly:

As previously said, Grafana has a ton of pre-built dashboards. For Spring Boot projects, the JVM dashboard is popular:

Input the URL for the dashboard, select "Already created Prometheus datasource" and then click Import:

Conclusion

Monitoring an application's health and metrics helps us manage it better, notice unoptimized behavior, and better understand its performance. This especially holds true when we're developing a system with many microservices, where monitoring each service can prove to be crucial when it comes to maintaining our system.

Based on this information, we can draw conclusions and decide which microservice needs to scale if further performance improvements can't be achieved with the current setup.

In this article, we used Micrometer to reformat the metrics data provided by Spring Boot Actuator and expose it in a new endpoint. This data was then regularly pulled and stored by Prometheus, which is a time-series database. Ultimately, we've used Grafana to visualize this information with a user-friendly dashboard.

As always, the code for the examples used in this article can be found on GitHub.