{"id":1767,"date":"2021-10-28T07:46:58","date_gmt":"2021-10-28T07:46:58","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=1767"},"modified":"2021-10-28T07:46:59","modified_gmt":"2021-10-28T07:46:59","slug":"calico-cni-failures","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2021\/10\/28\/calico-cni-failures\/","title":{"rendered":"CALICO CNI failures"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

In my cluster things went haywire all of a sudden without a proper showcase, I tried multiple things but it was just not working. Specifically my core-dns<\/code> and calico-kube-controllers<\/code> were not coming up and were constantly showing not ready<\/code> and CrashLoop<\/code> respectively<\/p>\n\n\n\n

k describe pod -nkube-system  calico-kube-controllers-75f8f6cc59-xmlx2<\/code><\/pre>\n\n\n\n
The error was evident as under – networkPlugin cni failed to set up pod \"calico-kube-controllers-75f8f6cc59-xmlx2_kube-system\"<\/code><\/p>\n\n\n\n
Events:\n  Type     Reason                  Age                From               Message\n  ----     ------                  ----               ----               -------\n  Normal   Scheduled               17m                default-scheduler  Successfully assigned kube-system\/calico-kube-controllers-75f8f6cc59-xmlx2 to mymachine.local.net\n  Warning  FailedCreatePodSandBox  17m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container \"cc4c71d1969bfe80b08055c3f685f3218b17a865ef7e069cbe37b9c830f791f5\" network for pod \"calico-kube-controllers-75f8f6cc59-xmlx2\": networkPlugin cni failed to set up pod \"calico-kube-controllers-75f8f6cc59-xmlx2_kube-system\" network: error getting ClusterInformation: connection is unauthorized: Unauthorized, failed to clean up sandbox container \"cc4c71d1969bfe80b08055c3f685f3218b17a865ef7e069cbe37b9c830f791f5\" network for pod \"calico-kube-controllers-75f8f6cc59-xmlx2\": networkPlugin cni failed to teardown pod \"calico-kube-controllers-75f8f6cc59-xmlx2_kube-system\" network: error getting ClusterInformation: connection is unauthorized: Unauthorized]<\/code><\/pre>\n\n\n\n
I looked at the not read coredns<\/code> and it was timing out.<\/p>\n\n\n\n
k logs -n kube-system coredns-78fcd69978-8sqqd<\/code><\/pre>\n\n\n\n
[INFO] 127.0.0.1:53136 - 58052 \"HINFO IN 1705020418207355632.1388822312682969349. udp 57 false 512\" NOERROR - 0 2.000338997s\n[ERROR] plugin\/errors: 2 1705020418207355632.1388822312682969349. HINFO: read udp 10.98.34.204:50841->192.19.189.10:53: i\/o timeout\n[INFO] plugin\/ready: Still waiting on: \"kubernetes\"<\/code><\/pre>\n\n\n\n
The thing that helped me solve the problem was setting up the iptables policy as under<\/p>\n\n\n\n
iptables -P INPUT ACCEPT\niptables -P FORWARD ACCEPT\niptables -P OUTPUT ACCEPT\niptables -F<\/code><\/pre>\n\n\n\n
This resulted in containers coming up for both coredns<\/code> and controllers<\/code><\/p>\n\n\n\n
Normal   SandboxChanged          16m (x2 over 17m)  kubelet            Pod sandbox changed, it will be killed and re-created.\n  Warning  Unhealthy               16m (x3 over 16m)  kubelet            Readiness probe failed: Failed to read status file \/status\/status.json: unexpected end of JSON input\n  Warning  BackOff                 16m (x7 over 16m)  kubelet            Back-off restarting failed container\n  Normal   Pulled                  15m (x4 over 16m)  kubelet            Container image \"docker.io\/calico\/kube-controllers:v3.20.2\" already present on machine\n  Normal   Created                 15m (x4 over 16m)  kubelet            Created container calico-kube-controllers\n  Normal   Started                 15m (x4 over 16m)  kubelet            Started container calico-kube-controllers<\/code><\/pre>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
In my cluster things went haywire all of a sudden without a proper showcase, I tried multiple things but it was just not working. Specifically my core-dns and calico-kube-controllers were not coming up and were constantly showing not ready and CrashLoop respectively The error was evident as under – networkPlugin cni failed to set up […]<\/p>\n","protected":false},"author":2,"featured_media":1769,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[86,195,34],"tags":[196,197,198],"class_list":["post-1767","post","type-post","status-publish","format-image","has-post-thumbnail","hentry","category-docker","category-kubernetes","category-technical","tag-calico","tag-cni","tag-errors","post_format-post-format-image"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/1767","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=1767"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/1767\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/1769"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=1767"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=1767"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=1767"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}