Microservices Monitoring and Alerting using Prometheus, Grafana and AlertManager.

There is a huge industry wide shift towards Microservices replacing monolithic application stacks to provide simplified and quick application deployments, scale up/down capabilities and accelerated development. But with all the functional and delivery advantages, it also increases the operational complexity.

Monitoring and troubleshooting microservices is more complex due to its modular and distributed nature. Lets look at Monitoring and Alerting microservices using Prometheus, Grafana and Alert Manager. These are the most widely adopted monitoring tools as of today and so have wide support.

Monitoring Architecture

image

There are 4 components in the above architecture:

  • A microservice developed using using spring boot, actuator, micrometer etc to expose metrics as an endpoint like /prometheus.
  • Prometheus service with configurations necessary to scrape the /prometheus endpoint and integrate with AlertManager.
  • Grafana service which will be pulling data from Prometheus and use it for data visualizations.
  • AlertManager service which will be purely used for alerting based on alert rules defined in Prometheus.

We are not going to go into details of these services as there are numerous blogs and articles on these services.

Setup

Download or clone the below GIT repository. It has all the necessary services shown in the architecture document for getting started.

https://github.com/Abmun/techmanyu-springboot-prometheus

There is a docker compose file provided in the project which will spin up 4 containers and setup the services.

Build Steps:

mvn clean package 
docker build -t techmanyu-springboot-prometheus .
docker-compose up -d

Four containers will spin up exposing 4 services-

Techmanyu-Service – http://localhost:8080

Metrics Endpoint- http://localhost:8080/prometheus or http://techmanyu-service:8080/prometheus

Prometheus Service – http://localhost:9090

AlertManager – http://localhost:9093

Grafana –http://localhost:3000 (Creds – admin/admin)

Health Checks in Prometheus:

image-1

In Grafana, go to DataSources and add prometheus endpoint.

image-2

Note: In alertmanager.yml file replace the slack webhool url and channel name to start receiving alerts. Update the alertmanager.yml file to swicth to any othe notification channel like email, webhook etc.

All set. Please comment in case of any questions or issues.

Categories
Comments
All comments.
Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  1. Jeremy

    Thanks for detailed article. Example works well.
    Can you also do a post using Elastic? or a comparison?