{"id":325,"date":"2020-01-24T12:45:34","date_gmt":"2020-01-24T12:45:34","guid":{"rendered":"https:\/\/www.samarthya.me\/wps\/?p=325"},"modified":"2020-01-24T12:46:27","modified_gmt":"2020-01-24T12:46:27","slug":"metric-beat","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2020\/01\/24\/metric-beat\/","title":{"rendered":"Metric Beat"},"content":{"rendered":"\n<p>All right, its been a busy week and finally some time at hand to write about something I have been doing.<\/p>\n\n\n\n<p>Mostly I am working on ELK stack these days, securing, validating, data analytics etc. Today, had the opportunity to go the Secure Elastic way, by enabling the TLS for secure communication within Nodes (Elastic cluster) and finally setup a Metric beat to get the status of the node.<\/p>\n\n\n\n<p>This blog will be my explanation of how to go about it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Helpful Links<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/metricbeat\/7.5\/setup-repositories.html#_yum\">https:\/\/www.elastic.co\/guide\/en\/beats\/metricbeat\/7.5\/setup-repositories.html#_yum<\/a><\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1<\/h2>\n\n\n\n<p>Installing the MetricBeat on my local system<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">System specs<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">cat \/etc\/redhat-release\n&gt;&gt; CentOS Linux release 7.7.1908 (Core)\n\nuname -a \n&gt;&gt; Linux Nodename 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU\/Linux<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Download and install the public signing key <\/h3>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>sudo rpm &#8211;import https:\/\/packages.elastic.co\/GPG-KEY-elasticsearch<\/p><cite>If success no message only prompt should appear<\/cite><\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Create a elastic.repo at \/etc\/yum\/repos.d<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo vi \/etc\/yum.repos.d\/elastic.repo\n\n[elastic-7.x]\nname=Elastic repository for 7.x packages\nbaseurl=https:\/\/artifacts.elastic.co\/packages\/7.x\/yum\ngpgcheck=1\ngpgkey=https:\/\/artifacts.elastic.co\/GPG-KEY-elasticsearch\nenabled=1\nautorefresh=1\ntype=rpm-md<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Install metricbeat<\/h3>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>sudo yum install <code>metricbeat<\/code><\/p><\/blockquote>\n\n\n\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"375\" src=\"https:\/\/www.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-12.21.57-PM-1024x375.png\" alt=\"\" class=\"wp-image-328\" srcset=\"https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-12.21.57-PM-1024x375.png 1024w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-12.21.57-PM-300x110.png 300w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-12.21.57-PM-768x281.png 768w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-12.21.57-PM-850x311.png 850w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-12.21.57-PM.png 1381w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li>Check the installation locations<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">whereis metricbeat \n&gt;&gt; metricbeat: \/usr\/bin\/metricbeat \/etc\/metricbeat \/usr\/share\/metricbeat<\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>Helpful link for <a href=\"https:\/\/www.elastic.co\/guide\/en\/beats\/metricbeat\/7.5\/directory-layout.html\">directory layout<\/a>.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Default paths location<\/h3>\n\n\n\n<p>Metricbeat uses the following default (paths can be changed).<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"\"><thead><tr><th>Type<\/th><th>Description<\/th><th>Location<\/th><\/tr><\/thead><tbody><tr><td><strong>home<\/strong><\/td><td>Home of the Metricbeat installation.<\/td><td><code>\/usr\/share\/metricbeat<\/code><\/td><\/tr><tr><td><strong>bin<\/strong><\/td><td>The location for the binary files.<\/td><td><code>\/usr\/share\/metricbeat\/bin<\/code><\/td><\/tr><tr><td><strong>config<\/strong><\/td><td>The location for configuration files.<\/td><td><code>\/etc\/metricbeat<\/code><\/td><\/tr><tr><td><strong>data<\/strong><\/td><td>The location for persistent data files.<\/td><td><code>\/var\/lib\/metricbeat<\/code><\/td><\/tr><tr><td><strong>logs<\/strong><\/td><td>The location for the logs created by Metricbeat.<\/td><td><code>\/var\/log\/metricbeat<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Configuring for the secured elasticsearch<\/h3>\n\n\n\n<p>The steps below allows to configure the metricbeat for a secured elastic instance.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>metricbeat keystore create <\/p><cite>        &gt;&gt; Created metricbeat keystore    <\/cite><\/blockquote>\n\n\n\n<p>It creates the keystore in the data folder. To validate<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">ls \/var\/lib\/metricbeat\/ \nmeta.json  metricbeat.keystore<\/pre>\n\n\n\n<p>Metricbeat provides a couple of different ways to enable modules and metricsets<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Enable module configs in module.d directory<\/li><li>Enable module config in metricbeat.yml<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">metricbeat.yml <\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">setup.kibana:\n\n  # Kibana Host\n  # Scheme and port can be left out and will be set to the default (http and 5601)\n  # In case you specify and additional path, the scheme is required: http:\/\/localhost:5601\/path\n  # IPv6 addresses should always be defined as: https:\/\/[2001:db8::1]:5601\n  host: \"https:\/\/myhost.com.localhost:5601\"\n\n  # Kibana Space ID\n  # ID of the Kibana Space into which the dashboards should be loaded. By default,\n  # the Default Space will be used.\n  space.id: saurabh\n<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">output.elasticsearch:\n  # Array of hosts to connect to.\n  hosts: [\"myhost.com.localhost:9200\"]\n\n  # Optional protocol and basic auth credentials.\n  protocol: \"https\"\n  username: \"elastic\"\n  password: \"EXTRAsecuredPassword\"<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">metricbeat modules enable elasticsearch<\/pre>\n\n\n\n<p>Enable the elasticsearch module for collection of metrics<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">metricbeat setup \nor\nsystemctl start metricbeat <\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https:\/\/myhost.com.localhost:9200: Get https:\/\/myhost.com.localhost:9200: x509: certificate is not valid for any names, but wanted to match myhost.com.localhost]<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">metricbeat modules list<\/pre>\n\n\n\n<p>On my local system the output is as under.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Enabled:\nelasticsearch\nsystem\n\nDisabled:\naerospike\napache\nappsearch\naws\nazure\nbeat\nbeat-xpack\nceph\ncockroachdb\nconsul\ncoredns\ncouchbase\ncouchdb\ndocker\ndropwizard\nelasticsearch-xpack\nenvoyproxy\netcd\ngolang\ngraphite\nhaproxy\nhttp\njolokia\nkafka\nkibana\nkibana-xpack\nkubernetes\nkvm\nlogstash\nlogstash-xpack\nmemcached\nmongodb\nmssql\nmunin\nmysql\nnats\nnginx\noracle\nphp_fpm\npostgresql\nprometheus\nrabbitmq\nredis\nstatsd\ntomcat\ntraefik\nuwsgi\nvsphere\nwindows\nzookeeper<\/pre>\n\n\n\n<p>Since the elastic is a secure instance I need to modify the elastic instance to include the certificate and key entries.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">  protocol: \"https\"\n  username: \"elastic\"\n  password: \"EXTRAsecuredPassword\"\n  ssl.certificate_authorities: [\"\/etc\/metricbeat\/ca.pem\"]\n  ssl.certificate: \"\/etc\/metricbeat\/instance.crt\"\n  ssl.key: \"\/etc\/metricbeat\/instance.key\"<\/pre>\n\n\n\n<p>and for kibana as well<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">  setup.kibana.ssl.enabled: true\n  ssl.kibana.ssl.certificate_authorities: [\"\/etc\/metricbeat\/ca.pem\"]\n  ssl.kibana.ssl.certificate: \"\/etc\/metricbeat\/instance.crt\"\n  ssl.kibana.ssl.key: \"\/etc\/metricbeat\/instance.key\"<\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"331\" src=\"https:\/\/www.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM-1024x331.png\" alt=\"\" class=\"wp-image-331\" srcset=\"https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM-1024x331.png 1024w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM-300x97.png 300w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM-768x248.png 768w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM-1536x496.png 1536w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM-850x274.png 850w, https:\/\/blog.samarthya.me\/wps\/wp-content\/uploads\/2020\/01\/Screenshot-2020-01-24-at-5.01.39-PM.png 1955w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h3 class=\"has-text-align-center wp-block-heading\">&#8212; THE &#8211; END &#8212; <\/h3>\n","protected":false},"excerpt":{"rendered":"<p>My encounter with metricbeat<\/p>\n","protected":false},"author":2,"featured_media":335,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[1,34],"tags":[32,39,40],"class_list":["post-325","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-others","category-technical","tag-elastic","tag-metricbeats","tag-secured"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=325"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/325\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/335"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}