Performance & Capacity Management using Elasticsearch
ELK can collect performance data from Nagios or any other existing monitoring tool in the infrastructure. Using ELK we can see the trends of resource utilization at multiple levels for example per location, per environment, per physical server etc. Looking at trend we can find when compute resources are idle and when they are highly used. We can plan compute resource for some other processing when they are idle
Capacity of the servers can be managed by looking at trends. Most of the time you will notice that resources are sitting idle or not more than 5% utilized. For such servers we can either reduces allocated compute or may utilize server compute for some other processing or services.
Capacity of infrastructure can be planned in advance by looking at resource utilization trend.
It is easy to setup ELK cluster and integrate with existing monitoring tool in your infrastructure. Look at above figure, Here Nagios is monitoring tool which is collecting performance data from entire IT infrastructure. and forward required data to Logstash using any data shipper like Filebeat. Use Redis for data buffering between log shipper and logstash. Redis is an open source, in-memory data structure store, used as a database, cache and message broker. Load balancer can be used to distribute the traffic between multiple logstash instances. Logstash will be used to structure the performance data and send it elasticsearch.
Use Kibana to plot the graphs for CPU/Memory/Disk Utilization trends.