WSO2EI JVM Monitoring with ELK

Here is to visualize JVM statistics using Kibana dashboard. WSO2EI is running on JVM and ELK setup should be already implemented.

Below statistic will be available in Kibana after this integration

1. Thread count
2. Class count
3. Heap memory
4. Non-heap memory

Tested Environment:

wso2ei-6.1.1

jdk1.8.0_172-amd64

nrpe  3.2.1

filebeat 6.8.0

logstash 6.8.0

ElasticSearch 6.8.0

Kibana 6.8.0

Step 1:

Install check_jvm plugin in the VM as mentioned in below article,

nrpe plugin for JVM

Step 2:

Make sure that you have installed NRPE plugin successfully. Execute below commands and check output

Command

       
/usr/local/nagios/libexec/check_jvm -u igwstuser -n org.wso2.carbon.bootstrap.Bootstrap -p heap -w 2008096384 -c 3000218931
       

Output

       
OK - 826M |max=9544663040;commited=4462215168;used=865669640;
       

Command

       
/usr/local/nagios/libexec/check_jvm -u igwstuser -n org.wso2.carbon.bootstrap.Bootstrap -p non-heap -w 268435456 -c 300870912
       

Output

       
WARNING 267M |max=-1;;; commited=352649216;;;used=279245856;;;
       

Command

       
/usr/local/nagios/libexec/check_jvm -u igwstuser -n org.wso2.carbon.bootstrap.Bootstrap -p classes -w 25000 -c 30000
       

Output

       
OK - 22816 |classes=22816;;;
       

Command

       
/usr/local/nagios/libexec/check_jvm -u igwstuser -n org.wso2.carbon.bootstrap.Bootstrap -p threads -w 700 -c 800
       

Output

       
OK - 655 |threads=655;;;
       

Don’t worry about the warning and critical threshold values, here we are considering values only.

Important: You should execute the above commands from the user who is running WSO2EI.

Step 3:

Generate shell script to execute the above commands and write output to a log file

check_jvm.sh

Step 4:

Add cronjob to execute the above shell script every 2 minutes.

       
*/2 * * * * cd /log/jvm-scripts; sh check_jvm.sh
       

Now data will be collected on the given location.

Step 5:

Here I’m assuming that ELK stack is already installed and configured. I will mention the configurations which are related to this topic only. You can refer below for filebeat and logstash configurations. make sure to restart filebeat and logstash service after adding new configurations.

Filebeat : read log file and push to logstash, please refer filebeat.yml in the below link.

Logstash : read the logs sent by filebeat, match to a correct index after applying to grok patterns  and then puch to elasticsearch, please refer logstash.conf in below link

jvm-monitor-elk

Step 6:

Go to Kibana portal à Management à Elasticsearch à Index Management

Search for jvm and then you should be able to see relevant indexes were created in Elasticsearch  

Step 7:

Go to Kibana à Index patterns à Create index pattern à type “jvm*” and create index pattern.

Step 8:

Now you can create your own visualizations and dashboard to monitor JVM statistics.

You can download the Kibana visualizations here.

Feel free to comment here if anything.

NRPE Custom Plugin for Apigate ELK

We can use both Nagios and Elastic Alerts to generate alarms based on different matrices in ELK module. Here i'm going to describe how to integrate Nagios [NRPE custom plugin ] to  generate alarms based on API delays.

Here I have used default Apigate analytic module and all indices were based on Apigate product.

Step 1:

Create check_api_delay.py file in /usr/local/nagios/libexec/ directory or your own custom plugin directory.
check_api_delay.py

Step 2:

Go to nrpe.cfg file in /usr/local/nagios/etc or your default location and define NRPE command as below

       
#define check_api_delay command
#define check_api_delay command
define command{
        command_name    check_api_delay
        command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_api_delay -a $ARG1$ $ARG2$ $ARG3$ $ARG4$ $ARG5$
}

#define services
define service{
        use                     local-service
        host_name               MONITORING_01
        service_description     DelayProfile-Payment
        check_command           check_api_delay!DOB-Payment!amount!5000!100!150
        normal_check_interval   2
}

define service{
        use                     local-service
        host_name               MONITORING_01
        service_description     DelayProfile-QueryProfile-subscribertype
        check_command           check_api_delay!QueryProfile!subscribertype!5000!80!100
        normal_check_interval   2
}

#define service groups
define servicegroup{
        servicegroup_name       IGW-API-Delay-Profile
        alias                   IGW API Delay Profile
        members                 MONITORING_01,DelayProfile-Payment,MONITORING_01,DelayProfile-QueryProfile-subscribertype
}
       

Step 4:

Restart nrpe and Nagios agent