Coredns couldn't reach to Host Nameserver

Question

I've created a kubernetes cluster from scratch as the directions of Kelsey Hightower. As I checked, everything works without any error but after I've deployed my first application, I figured out my application couldn't resolve the DNSes.

I've checked my coredns logs and saw below entries :

.:53
2018/10/16 12:31:45 [INFO] CoreDNS-1.2.2
2018/10/16 12:31:45 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2018/10/16 12:31:45 [INFO] plugin/reload: Running configuration MD5 = 06122de1a2d6c43092ab48d05478dc82
2018/10/16 12:44:27 [ERROR] 2 google.com. A: unreachable backend: read udp 192.168.65.142:51219->172.10.0.2:53: i/o timeout
2018/10/16 12:44:29 [ERROR] 2 google.com. A: unreachable backend: read udp 192.168.65.142:39967->172.10.0.2:53: i/o timeout
2018/10/16 12:44:31 [ERROR] 2 google.com. A: unreachable backend: read udp 192.168.65.142:40187->172.10.0.2:53: i/o timeout

Plus to Kelsey's documentation, I've installed calico on top of Docker.

My Suspect :

I've installed Calico on top of Docker, but my kubelet runtime has been configured to work with Containerd. I'm sshing into calico pod and seems it has network. But none of the containerd pods. But I couldn't find a way to run calico over Containerd.

My Kubelet Service Configuration :

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/bin/kubelet \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --image-pull-progress-deadline=2m \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig-kubelet \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/cni/bin \
  --register-node=true \
  --cloud-provider=aws \
  --v=2

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

And my kubelet config yaml file :

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/etc/kubernetes/pki/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "${cluster_domain}"
clusterDNS:
  - "172.10.0.10"
podCIDR: "172.10.0.0/16"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/etc/kubernetes/pki/worker.pem"
tlsPrivateKeyFile: "/etc/kubernetes/pki/worker-key.pem"

My resolv.conf file on the node machine :

nameserver 172.10.0.2

I can see that pods could connect to coredns pod but coredns couldn't connect to 172.10.0.2 over port 53. On the host machine, I could telnet to this port and getting answer.

best,

Answer 1

It is a tricky question. I experienced the same issue and solved in the following way, It should work for you too. To install Calico to your cluster you need to patch the Calico YAML. Rely on documentation, how to install Calico:

To achieve your goal you need to: Create RBAC for Calico:

kubectl apply -f \
https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml

Download the YAML with the configuration of Calico:

curl \
https://docs.projectcalico.org/v3.2/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml -o

Edit the file calico.yaml :

- name: CALICO_IPV4POOL_CIDR
   value: "10.200.0.0/16"

Paste into the section value the 10.200.0.0/16 than save. and apply it:

kubectl apply -f calico.yaml