{"id":1141,"date":"2020-12-09T08:01:49","date_gmt":"2020-12-09T08:01:49","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=1141"},"modified":"2020-12-09T08:48:36","modified_gmt":"2020-12-09T08:48:36","slug":"metric-server","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2020\/12\/09\/metric-server\/","title":{"rendered":"Metric Server"},"content":{"rendered":"\n

For my local K8S cluster of 4 nodes I wanted to get the resource usage metrics. From the official documentation of kubernetes.io <\/p>\n\n\n\n

These metrics can be accessed either directly by the user with the kubectl top<\/code> command, or by a controller in the cluster, for example Horizontal Pod Autoscaler, to make decisions.<\/p>https:\/\/kubernetes.io<\/cite><\/blockquote>\n\n\n\n

Resource usage metrics, such as container CPU and memory usage, are available in Kubernetes through the Metrics API.<\/p><\/blockquote><\/figure>\n\n\n\n

I deployed Metric Server v0.4.1<\/p>\n\n\n\n

kubectl apply -f https:\/\/github.com\/kubernetes-sigs\/metrics-server\/releases\/download\/v0.4.1\/components.yaml<\/code><\/pre>\n\n\n\n
serviceaccount\/metrics-server configured\nclusterrole.rbac.authorization.k8s.io\/system:aggregated-metrics-reader configured\nclusterrole.rbac.authorization.k8s.io\/system:metrics-server configured\nrolebinding.rbac.authorization.k8s.io\/metrics-server-auth-reader configured\nclusterrolebinding.rbac.authorization.k8s.io\/metrics-server:system:auth-delegator configured\nclusterrolebinding.rbac.authorization.k8s.io\/system:metrics-server configured\nservice\/metrics-server created\ndeployment.apps\/metrics-server created\napiservice.apiregistration.k8s.io\/v1beta1.metrics.k8s.io configured<\/code><\/pre>\n\n\n\n
After installation is complete, fire in to check the top node stats.<\/p>\n\n\n\n
k top node<\/code><\/pre>\n\n\n\n
For me I was getting<\/p>\n\n\n\n
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)<\/code><\/pre>\n\n\n\n
I could see the POD and deployment <\/p>\n\n\n\n
kube-system       pod\/metrics-server-5d5c49f488-8dhdf                        0\/1     Running        0          16s<\/code><\/pre>\n\n\n\n
kube-system       deployment.apps\/metrics-server            0\/1     1            0           16s<\/code><\/pre>\n\n\n\n
A quick describe for pod revealed<\/p>\n\n\n\n
k describe pod\/metrics-server-5d5c49f488-8dhdf -n kube-system<\/code><\/pre>\n\n\n\n
Events:\n  Type     Reason     Age               From               Message\n  ----     ------     ----              ----               -------\n  Normal   Scheduled  37s               default-scheduler  Successfully assigned kube-system\/metrics-server-5d5c49f488-8dhdf to i01.samarthya.me\n  Normal   Pulling    36s               kubelet            Pulling image \"k8s.gcr.io\/metrics-server\/metrics-server:v0.4.1\"\n  Normal   Pulled     25s               kubelet            Successfully pulled image \"k8s.gcr.io\/metrics-server\/metrics-server:v0.4.1\" in 11.718855147s\n  Normal   Created    24s               kubelet            Created container metrics-server\n  Normal   Started    24s               kubelet            Started container metrics-server\n  Warning  Unhealthy  7s (x2 over 17s)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500\n  Warning  Unhealthy  5s (x2 over 15s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500<\/code><\/pre>\n\n\n\n
Looking at the logs<\/p>\n\n\n\n
k log pod\/metrics-server-5d5c49f488-8dhdf  -n kube-system<\/code><\/pre>\n\n\n\n
E1209 07:23:57.351296       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node mymachine.cluster.samarthya.me: unable to fetch metrics from node mymachine.cluster.samarthya.me: Get \"https:\/\/10.80.241.70:10250\/stats\/summary?only_cpu_and_memory=true\": x509: cannot validate certificate for 10.80.241.70 because it doesn't contain any IP SANs, unable to fully scrape metrics from node i01.samarthya.me: unable to fetch metrics from node i02.samarthya.me: Get \"https:\/\/10.80.120.149:10250\/stats\/summary?only_cpu_and_memory=true\": x509: cannot validate certificate for 10.80.120.149 because it doesn't contain any IP SANs, unable to fully scrape metrics from node ibndev003277samarthya.me: unable to fetch metrics from node ibndev003277samarthya.me: Get \"https:\/\/10.80.120.148:10250\/stats\/summary?only_cpu_and_memory=true\": x509: cannot validate certificate for 10.80.120.148 because it doesn't contain any IP SANs, unable to fully scrape metrics from node i03.samarthya.me: unable to fetch metrics from node i04.samarthya.me: Get \"https:\/\/10.80.241.80:10250\/stats\/summary?only_cpu_and_memory=true\": x509: cannot validate certificate for 10.80.241.80 because it doesn't contain any IP SANs, unable to fully scrape metrics from node i03.samarthya.me: unable to fetch metrics from node i03.samarthya.me: Get \"https:\/\/10.80.241.78:10250\/stats\/summary?only_cpu_and_memory=true\": x509: cannot validate certificate for 10.80.241.78 because it doesn't contain any IP SANs]<\/code><\/pre>\n\n\n\n
and after some time I could see<\/p>\n\n\n\n
kube-system       pod\/metrics-server-5d5c49f488-8dhdf                        0\/1     CrashLoopBackOff   4          2m48s<\/code><\/pre>\n\n\n\n
I could see the selfsigned certificares were problematic for the configuration<\/p>\n\n\n\n
k edit deployment.apps\/metrics-server -n kube-system<\/code><\/pre>\n\n\n\n
Added `<\/p>\n\n\n\n
--kubelet-insecure-tls<\/code><\/pre>\n\n\n\n
to the deployment and saved it.<\/p>\n\n\n\n
 containers:\n      - args:\n        - --cert-dir=\/tmp\n        - --secure-port=4443\n        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname\n        - --kubelet-use-node-status-port\n        - --kubelet-insecure-tls<\/code><\/pre>\n\n\n\n
It fired up the new container<\/p>\n\n\n\n
kube-system       pod\/metrics-server-56c59cf9ff-qcxlc                        0\/1     ContainerCreating   0          5s<\/code><\/pre>\n\n\n\n
k describe  pod\/metrics-server-56c59cf9ff-qcxlc  -n kube-system<\/code><\/pre>\n\n\n\n
  Normal  Created    23s   kubelet            Created container metrics-server\n  Normal  Started    23s   kubelet            Started container metrics-server<\/code><\/pre>\n\n\n\n
k logs  pod\/metrics-server-56c59cf9ff-qcxlc  -n kube-system<\/code><\/pre>\n\n\n\n
I1209 07:27:03.832542       1 serving.go:325] Generated self-signed cert (\/tmp\/apiserver.crt, \/tmp\/apiserver.key)\nI1209 07:27:04.575200       1 secure_serving.go:197] Serving securely on [::]:4443\nI1209 07:27:04.575282       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController\nI1209 07:27:04.575295       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController\nI1209 07:27:04.575372       1 dynamic_serving_content.go:130] Starting serving-cert::\/tmp\/apiserver.crt::\/tmp\/apiserver.key\nI1209 07:27:04.575450       1 tlsconfig.go:240] Starting DynamicServingCertificateController\nI1209 07:27:04.575546       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file\nI1209 07:27:04.575551       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file\nI1209 07:27:04.575563       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file\nI1209 07:27:04.575567       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file\nI1209 07:27:04.675539       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController \nI1209 07:27:04.675584       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file \nI1209 07:27:04.675731       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file <\/code><\/pre>\n\n\n\n
With things appearing to be running fine time to find the top node<\/p>\n\n\n\n
k top node<\/code><\/pre>\n\n\n\n
NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   \ni01.samarthya.me    131m         0%     7004Mi          21%       \ni02.samarthya.me    85m          0%     5542Mi          17%       \ni03.samarthya.me    163m         1%     13011Mi         40%       \ni04.samarthya.me    395m         2%     7191Mi          22%       \ni05.samarthya.me    94m          0%     11536Mi         36% <\/code><\/pre>\n\n\n\n
Help<\/h2>\n\n\n\n
  • https:\/\/kubernetes.io\/docs\/tasks\/debug-application-cluster\/resource-metrics-pipeline\/<\/li>
  • https:\/\/github.com\/kubernetes\/community\/blob\/master\/contributors\/design-proposals\/instrumentation\/metrics-server.md<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
    For my local K8S cluster of 4 nodes I wanted to get the resource usage metrics. From the official documentation of kubernetes.io These metrics can be accessed either directly by the user with the kubectl top command, or by a controller in the cluster, for example Horizontal Pod Autoscaler, to make decisions. https:\/\/kubernetes.io Resource usage […]<\/p>\n","protected":false},"author":2,"featured_media":1143,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[86,34],"tags":[18,126],"class_list":["post-1141","post","type-post","status-publish","format-image","has-post-thumbnail","hentry","category-docker","category-technical","tag-k8s","tag-metricserver","post_format-post-format-image"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/1141","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=1141"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/1141\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/1143"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=1141"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=1141"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=1141"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}