Graphite Plugin

Purpose

The Graphite plugin gives the possibility to collect metrics, alarms and metadata in Graphite plateform. Once active, this plugin will collect all generated alerts, metrics and metadata and store them waiting for the probe to collect them To limit memory usage the number of element temporally stored is limited

Create/Edit

Press Create or select Edit in the table action menu to open the Graphite Plugin editor

Set Graphite Plugin parameter:

Active

Name

Host

Port

URL

Secure mode

Send alarms

Send metrics

Send Redpeaks stats

Max chunk size

Installing graphite

Prerequisites

Linux VM with a Cent OS/RedHat distribution

200 GB disk space

2 cores

4 GB RAM

Installation

sudo docker run -d --name graphite --restart=always -p 80:80 -p 2003-2004:2003-2004 -p 2023-2024:2023-2024 -p 8125:8125/udp -p 8126:8126 graphiteapp/graphite-statsd

open ports 2003/2004

CentOS/RHEL6 :

sudo iptables -I INPUT -p tcp -m tcp --dport 2003 -j ACCEPT

service iptables save

CentOS/RHEL7 :

sudo firewall-cmd --zone=public --add-port=2003/tcp --permanent

Default credentials: root/root

Configure

docker exec -i -t graphite bash

vi /opt/graphite/conf/storage-schemas.conf

Configure the retention granularity.

The following will define retention as:

Keep 7 days with a granularity of 1 minute

Keep 30 days with a granularity of 5 minutes

Keep 90 days with a granularity of 90d

[agentil metrics availability]
pattern = ^agentil\.promonitor\.
retentions = 1m:7d,5m:30d,1h:90d
[agentil sap]
pattern = ^agentil\.sap\.
retentions = 1m:7d,5m:30d,1h:90d

you can define custom granularity based on metric path

Then remove everything in /opt/graphite/storage/whisper

Then restart graphite via docker:

sudo docker stop graphite

If iptables problem issue when starting graphite, try to restart docker:

sudo systemctl restart docker

Installing Grafana

Install

sudo docker run -d -p 3000:3000 -v /var/lib/grafana:/var/lib/grafana -e “GF_SECURITY_ADMIN_PASSWORD=secret” grafana/grafana

open port 3000

Start

Check /var/lib/grafana is writable

sudo docker start <image ID>

Does not restart automatically if docker is restarted

Configure

Go to http://host:3000

Login: admin/secret

Admin->Datasource->Add data source

Choose graphite

HTTP settings:

URL: Graphite FQDN

Access: server

Troubleshooting

No metrics created in Graphite

Check plugin connectivity (port number must be 2003), test must show success.

Check disk space is not full

Try the metric sender script from local and distant severs.

Some metrics seems to be missing

Check “use title as source”. If title contains spaces, metric will be rejected.

Check /var/log/carbon.log for ignore messages

Restart Redpeaks, can be some cache issue

Disk full.

Can be caused by huge log files : carbon.log

run the following commands:

sudo docker exec -i -t graphite bash
cd /var/log
rm -f carbon.log
exit
sudo docker stop graphite
sudo docker start graphite