Metric Beat

Saurabh Sharma

All right, its been a busy week and finally some time at hand to write about something I have been doing.

Mostly I am working on ELK stack these days, securing, validating, data analytics etc. Today, had the opportunity to go the Secure Elastic way, by enabling the TLS for secure communication within Nodes (Elastic cluster) and finally setup a Metric beat to get the status of the node.

This blog will be my explanation of how to go about it.

Helpful Links

Step 1

Installing the MetricBeat on my local system

System specs

cat /etc/redhat-release
>> CentOS Linux release 7.7.1908 (Core)

uname -a 
>> Linux Nodename 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Download and install the public signing key

sudo rpm –import https://packages.elastic.co/GPG-KEY-elasticsearch

If success no message only prompt should appear

Create a elastic.repo at /etc/yum/repos.d

sudo vi /etc/yum.repos.d/elastic.repo

[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Install metricbeat

sudo yum install metricbeat

  • Check the installation locations
whereis metricbeat 
>> metricbeat: /usr/bin/metricbeat /etc/metricbeat /usr/share/metricbeat

Default paths location

Metricbeat uses the following default (paths can be changed).

TypeDescriptionLocation
homeHome of the Metricbeat installation./usr/share/metricbeat
binThe location for the binary files./usr/share/metricbeat/bin
configThe location for configuration files./etc/metricbeat
dataThe location for persistent data files./var/lib/metricbeat
logsThe location for the logs created by Metricbeat./var/log/metricbeat

Configuring for the secured elasticsearch

The steps below allows to configure the metricbeat for a secured elastic instance.

metricbeat keystore create

>> Created metricbeat keystore

It creates the keystore in the data folder. To validate

ls /var/lib/metricbeat/ 
meta.json  metricbeat.keystore

Metricbeat provides a couple of different ways to enable modules and metricsets

  • Enable module configs in module.d directory
  • Enable module config in metricbeat.yml

metricbeat.yml

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "https://myhost.com.localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  space.id: saurabh
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["myhost.com.localhost:9200"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "EXTRAsecuredPassword"
metricbeat modules enable elasticsearch

Enable the elasticsearch module for collection of metrics

metricbeat setup 
or
systemctl start metricbeat 
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://myhost.com.localhost:9200: Get https://myhost.com.localhost:9200: x509: certificate is not valid for any names, but wanted to match myhost.com.localhost]
metricbeat modules list

On my local system the output is as under.

Enabled:
elasticsearch
system

Disabled:
aerospike
apache
appsearch
aws
azure
beat
beat-xpack
ceph
cockroachdb
consul
coredns
couchbase
couchdb
docker
dropwizard
elasticsearch-xpack
envoyproxy
etcd
golang
graphite
haproxy
http
jolokia
kafka
kibana
kibana-xpack
kubernetes
kvm
logstash
logstash-xpack
memcached
mongodb
mssql
munin
mysql
nats
nginx
oracle
php_fpm
postgresql
prometheus
rabbitmq
redis
statsd
tomcat
traefik
uwsgi
vsphere
windows
zookeeper

Since the elastic is a secure instance I need to modify the elastic instance to include the certificate and key entries.

  protocol: "https"
  username: "elastic"
  password: "EXTRAsecuredPassword"
  ssl.certificate_authorities: ["/etc/metricbeat/ca.pem"]
  ssl.certificate: "/etc/metricbeat/instance.crt"
  ssl.key: "/etc/metricbeat/instance.key"

and for kibana as well

  setup.kibana.ssl.enabled: true
  ssl.kibana.ssl.certificate_authorities: ["/etc/metricbeat/ca.pem"]
  ssl.kibana.ssl.certificate: "/etc/metricbeat/instance.crt"
  ssl.kibana.ssl.key: "/etc/metricbeat/instance.key"

— THE – END —