{"id":944,"date":"2020-08-22T10:53:05","date_gmt":"2020-08-22T10:53:05","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=944"},"modified":"2020-08-23T15:56:01","modified_gmt":"2020-08-23T15:56:01","slug":"probes-k8s","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2020\/08\/22\/probes-k8s\/","title":{"rendered":"Probes: K8S"},"content":{"rendered":"

In this blog we will be talking about Liveness, Readiness and Startup probes of a container.<\/p>\n\n\n

A Service<\/a><\/em><\/strong> in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them.<\/p>– Life without a service<\/cite><\/blockquote><\/figure>\n\n\n\n

What is Kubelet?<\/h2>\n\n\n\n
  • Kubelet <\/code>is a primary node agent that runs on every node.<\/li>
  • Works in term of PodSpec, that describes a POD (JSON or Yaml).<\/li><\/ul>\n\n\n\n

    Liveness Probe<\/h2>\n\n\n\n
    • It is used by kubelet<\/code> to decide when to restart the container.<\/li>
    • periodSeconds<\/code> defines when the kubelet<\/code> check periodically.<\/li>
    • initialDelaySeconds<\/code> specifies the initial wait before checking.<\/li><\/ul>\n\n\n\n

      I just created a single container pod that will run touch<\/code> command to create a file and then it deletes that specific file after a delay, on which the livenessProbe is listening on and then once liveness fails we can check the container.<\/p>\n\n\n\n

      --- # Liveness Probe example\napiVersion: v1\nkind: Pod\nmetadata:\n  name: pod-bbox-lp\n  labels:\n    name: liveness-probe-pod\n# Container specs\nspec:\n  containers:\n    - name: bbox-pod\n      image: busybox:latest\n      livenessProbe:\n        exec:\n          command:\n            - cat\n            - \/tmp\/healthy\n        initialDelaySeconds: 5\n        periodSeconds: 5\n      args:\n        - \/bin\/sh\n        - -c\n        - touch \/tmp\/healthy; sleep 30; rm -rf \/tmp\/healthy; sleep 600\n      resources:\n        limits:\n          memory: \"128Mi\"\n          cpu: \"500m\"<\/code><\/pre>\n\n\n\n
      Liveness will be checked by issuing command cat \/tmp\/healthy<\/code><\/p>\n\n\n\n
       k get pods\nNAME          READY   STATUS              RESTARTS   AGE\npod-bbox-lp   0\/1     ContainerCreating   0          2s<\/code><\/pre>\n\n\n\n
      Describe to see details k describe pod\/pod-bbox-lp<\/code><\/p><\/blockquote><\/figure>\n\n\n\n
      Name:         pod-bbox-lp\nNamespace:    default\nPriority:     0\nNode:         docker-desktop\/192.168.65.3\nStart Time:   Sat, 22 Aug 2020 16:30:52 +0530\nLabels:       name=liveness-probe-pod\nAnnotations:  kubectl.kubernetes.io\/last-applied-configuration:\n                {\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"name\":\"liveness-probe-pod\"},\"name\":\"pod-bbox-lp\",\"namespace\":\"defa...\nStatus:       Running\nIP:           10.1.0.92\nIPs:\n  IP:  10.1.0.92\nContainers:\n  bbox-pod:\n    Container ID:  docker:\/\/817a11380e645cbde80f66fa34aa5652dcd6744f89788d0f1fe337c685280a10\n    Image:         busybox:latest\n    Image ID:      docker-pullable:\/\/busybox@sha256:4f47c01fa91355af2865ac10fef5bf6ec9c7f42ad2321377c21e844427972977\n    Port:          <none>\n    Host Port:     <none>\n    Args:\n      \/bin\/sh\n      -c\n      touch \/tmp\/healthy; sleep 30; rm -rf \/tmp\/healthy; sleep 600\n    State:          Running\n      Started:      Sat, 22 Aug 2020 16:31:00 +0530\n    Ready:          True\n    Restart Count:  0\n    Limits:\n      cpu:     500m\n      memory:  128Mi\n    Requests:\n      cpu:        500m\n      memory:     128Mi\n    Liveness:     exec [cat \/tmp\/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3\n    Environment:  <none>\n    Mounts:\n      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from default-token-ksbmz (ro)\nConditions:\n  Type              Status\n  Initialized       True \n  Ready             True \n  ContainersReady   True \n  PodScheduled      True \nVolumes:\n  default-token-ksbmz:\n    Type:        Secret (a volume populated by a Secret)\n    SecretName:  default-token-ksbmz\n    Optional:    false\nQoS Class:       Guaranteed\nNode-Selectors:  <none>\nTolerations:     node.kubernetes.io\/not-ready:NoExecute for 300s\n                 node.kubernetes.io\/unreachable:NoExecute for 300s\nEvents:\n  Type    Reason     Age        From                     Message\n  ----    ------     ----       ----                     -------\n  Normal  Scheduled  <unknown>  default-scheduler        Successfully assigned default\/pod-bbox-lp to docker-desktop\n  Normal  Pulling    18s        kubelet, docker-desktop  Pulling image \"busybox:latest\"\n  Normal  Pulled     10s        kubelet, docker-desktop  Successfully pulled image \"busybox:latest\"\n  Normal  Created    10s        kubelet, docker-desktop  Created container bbox-pod\n  Normal  Started    10s        kubelet, docker-desktop  Started container bbox-pod<\/code><\/pre>\n\n\n\n
      These commands will be run as bootstrapping the container.<\/p>\n\n\n\n
      \/bin\/sh -c touch \/tmp\/healthy; sleep 30; rm -rf \/tmp\/healthy; sleep 600<\/code><\/pre>\n\n\n\n
      The liveness probe is looking for the file \/tmp\/healthy<\/code><\/p>\n\n\n\n
      Liveness:     exec [cat \/tmp\/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3<\/code><\/pre>\n\n\n\n
      Check for the file if it exists while the container sleeps.<\/p>\n\n\n\n
      > k exec pod\/pod-bbox-lp -- ls -als \/tmp\/healthy\n\n0 -rw-r--r--    1 root     root             0 Aug 22 11:02 \/tmp\/healthy<\/code><\/pre>\n\n\n\n
      Let us run describe<\/code> after the sleep delay – 30 seconds.<\/p>\n\n\n\n
      > k describe pod\/pod-bbox-lp<\/code><\/pre>\n\n\n\n
      Name:         pod-bbox-lp\nNamespace:    default\nPriority:     0\nNode:         docker-desktop\/192.168.65.3\nStart Time:   Sat, 22 Aug 2020 16:30:52 +0530\nLabels:       name=liveness-probe-pod\nAnnotations:  kubectl.kubernetes.io\/last-applied-configuration:\n                {\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"labels\":{\"name\":\"liveness-probe-pod\"},\"name\":\"pod-bbox-lp\",\"namespace\":\"defa...\nStatus:       Running\nIP:           10.1.0.92\nIPs:\n  IP:  10.1.0.92\nContainers:\n  bbox-pod:\n    Container ID:  docker:\/\/302120c0b732718491733e9073830d40375df47c26e1b7df9cdd09391d74602f\n    Image:         busybox:latest\n    Image ID:      docker-pullable:\/\/busybox@sha256:4f47c01fa91355af2865ac10fef5bf6ec9c7f42ad2321377c21e844427972977\n    Port:          <none>\n    Host Port:     <none>\n    Args:\n      \/bin\/sh\n      -c\n      touch \/tmp\/healthy; sleep 30; rm -rf \/tmp\/healthy; sleep 600\n    State:          Running\n      Started:      Sat, 22 Aug 2020 16:32:17 +0530\n    Last State:     Terminated\n      Reason:       Error\n      Exit Code:    137\n      Started:      Sat, 22 Aug 2020 16:31:00 +0530\n      Finished:     Sat, 22 Aug 2020 16:32:11 +0530\n    Ready:          True\n    Restart Count:  1\n    Limits:\n      cpu:     500m\n      memory:  128Mi\n    Requests:\n      cpu:        500m\n      memory:     128Mi\n    Liveness:     exec [cat \/tmp\/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3\n    Environment:  <none>\n    Mounts:\n      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from default-token-ksbmz (ro)\nConditions:\n  Type              Status\n  Initialized       True \n  Ready             True \n  ContainersReady   True \n  PodScheduled      True \nVolumes:\n  default-token-ksbmz:\n    Type:        Secret (a volume populated by a Secret)\n    SecretName:  default-token-ksbmz\n    Optional:    false\nQoS Class:       Guaranteed\nNode-Selectors:  <none>\nTolerations:     node.kubernetes.io\/not-ready:NoExecute for 300s\n                 node.kubernetes.io\/unreachable:NoExecute for 300s\nEvents:\n  Type     Reason     Age                  From                     Message\n  ----     ------     ----                 ----                     -------\n  Normal   Scheduled  <unknown>            default-scheduler        Successfully assigned default\/pod-bbox-lp to docker-desktop\n  Normal   Pulling    57s (x2 over 2m17s)  kubelet, docker-desktop  Pulling image \"busybox:latest\"\n  Normal   Pulled     52s (x2 over 2m9s)   kubelet, docker-desktop  Successfully pulled image \"busybox:latest\"\n  Normal   Created    52s (x2 over 2m9s)   kubelet, docker-desktop  Created container bbox-pod\n  Normal   Started    52s (x2 over 2m9s)   kubelet, docker-desktop  Started container bbox-pod\n  Warning  Unhealthy  8s (x6 over 98s)     kubelet, docker-desktop  Liveness probe failed: cat: can't open '\/tmp\/healthy': No such file or directory\n  Normal   Killing    8s (x2 over 88s)     kubelet, docker-desktop  Container bbox-pod failed liveness probe, will be restarted<\/code><\/pre>\n\n\n\n
      Notice the error<\/p>\n\n\n\n
       Last State:     Terminated\n      Reason:       Error\n      Exit Code:    137\n      Started:      Sat, 22 Aug 2020 16:31:00 +0530\n      Finished:     Sat, 22 Aug 2020 16:32:11 +0530<\/code><\/pre>\n\n\n\n
      desktop  Started container bbox-pod\n  Warning  Unhealthy  8s (x6 over 98s)     kubelet, docker-desktop  Liveness probe failed: cat: can't open '\/tmp\/healthy': No such file or directory<\/code><\/pre>\n\n\n\n
      You can also use an httpGet<\/code> liveness probe as well.<\/p>\n\n\n\n
      --- # Livenessprobe example for HTTP endpoint\napiVersion: v1\nkind: Pod\nmetadata:\n  name: liveness-probe-example\n  labels:\n    name: liveness-probe-app\nspec:\n  containers:\n    - name: liveness-probe-pod\n      image: samarthya\/epserver:1.0\n      livenessProbe:\n        httpGet:\n          path: \/hello\n          port: 8090\n          httpHeaders:\n            - name: CustomHeader\n              value: Awesome\n        initialDelaySeconds: 5\n        periodSeconds: 3\n      resources:\n        limits:\n          memory: \"128Mi\"\n          cpu: \"500m\"\n      ports:\n        - containerPort: 8090<\/code><\/pre>\n\n\n\n
      If you look at the logs you can see the endpoint being called regularly<\/p>\n\n\n\n
      > k logs pod\/liveness-probe-example\n2020\/08\/22 18:20:52 My Simple Http Server\n2020\/08\/22 18:20:52  DBG: Accepting traffic for \/hello\n -------- Welcome to my server ------- \n2020\/08\/22 18:20:59  DBG: Method -  GET\n2020\/08\/22 18:20:59  DBG: Body -  {}\n2020\/08\/22 18:20:59  DBG: URL -  \/hello\n2020\/08\/22 18:20:59     { \n2020\/08\/22 18:20:59      DBG: Hello World!\n2020\/08\/22 18:20:59      DBG: 2020-08-22 18:20:59.6744393 +0000 UTC\n2020\/08\/22 18:20:59     } \n2020\/08\/22 18:20:59  DBG: {\"message\":\"Hello World!\",\"timestamp\":\"2020-08-22T18:20:59.6744393Z\"}\n2020\/08\/22 18:21:02  DBG: Method -  GET\n2020\/08\/22 18:21:02  DBG: Body -  {}\n2020\/08\/22 18:21:02  DBG: URL -  \/hello\n2020\/08\/22 18:21:02     { \n2020\/08\/22 18:21:02      DBG: Hello World!\n2020\/08\/22 18:21:02      DBG: 2020-08-22 18:21:02.6743863 +0000 UTC\n2020\/08\/22 18:21:02     } \n2020\/08\/22 18:21:02  DBG: {\"message\":\"Hello World!\",\"timestamp\":\"2020-08-22T18:21:02.6743863Z\"}<\/code><\/pre>\n\n\n\n
      \"\"\/
      From the official K8S documentation.<\/figcaption><\/figure>\n\n\n\n
      Readiness Probe<\/h2>\n\n\n\n
      • It is used to decide when the pod is ready to accept the incoming traffic.<\/li><\/ul>\n\n\n\n
        Startup Probe<\/h2>\n\n\n\n
        • The kubelet<\/code> uses startup probes to know when a container application has started.<\/li><\/ul>\n\n\n\n
          References<\/h2>\n\n\n\n
          • https:\/\/kubernetes.io\/docs\/reference\/command-line-tools-reference\/kubelet\/<\/li>
          • https:\/\/hub.docker.com\/repository\/docker\/samarthya\/epserver<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
            In this blog we will be talking about Liveness, Readiness and Startup probes of a container. A Service in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them. – Life without a service What is Kubelet? Kubelet is a primary node agent that runs on […]<\/p>\n","protected":false},"author":2,"featured_media":947,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[34],"tags":[18,97],"class_list":["post-944","post","type-post","status-publish","format-image","has-post-thumbnail","hentry","category-technical","tag-k8s","tag-probes","post_format-post-format-image"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/944","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=944"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/944\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/947"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=944"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=944"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=944"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}