How do I add a nameserver to all pods in Google Container Engine [GKE]?

Question

I am attempting to migrate my on premises cluster to GKE. In order to facilitate this transition I need to be able to resolve the names of legacy services.

Assume that t

Answer 1

I solved this by setting up a dnsmasq service in the k8s cluster and point all pods nameserver except dnsmasq to the dnsmasq service.
dnsmasq will forward requests to the correct nameserver based on domain suffix. So both internal and external vpn lookups will work.

1. setup a dnsmasq service.
   The pods can look something like this, make sure this is has at least 2 pods as it needs to be HA.

   apiVersion: v1
   kind: Pod
   metadata:
     name: dnsmasq
   spec:
     containers:
     - name: dnsmasq
       image: "andyshinn/dnsmasq:2.76"
       ports:
       - containerPort: 53
         hostPort: 53
         protocol: UDP
       - containerPort: 53
         hostPort: 53
         protocol: TCP
       args: [
         "-S", "/consul/10.3.20.86",
         "-S", "/consul/10.3.20.88",
         "-S", "/consul/10.3.20.90",
         "-S", "/your-vpn-domain.dom/10.3.128.22",
         "-S", "/your-vpn-domain.dom/10.3.128.23"
       ]
       securityContext:
         capabilities:
           add:
           - NET_ADMIN
   

2. Add a resolv-conf config map.

   #!/bin/bash
   
   DNS_IP=$(kubectl get svc --template '{{.spec.clusterIP}}' dnsmasq)
   DNS_POD=$(kubectl get pod -n kube-system | grep -v kube-dns-autoscaler | grep ^kube-dns  | head -1 | awk '{ print $1; }')
   DOMAIN=$(kubectl describe -n kube-system pod/${DNS_POD} | grep -- --domain= | sed -Ee 's/.*--domain=(.*)\..*/\1/')
   SEARCH=$(kubectl exec -n kube-system  ${DNS_POD} -c kubedns -- cat /etc/resolv.conf | grep ^search | cut -d' '  -f2-)
   VPN_SEARCH="your-vpn-domain.dom"
   
   kubectl create -f - <

3. Mount the cfgmap in your services/pods. add this to your pods

     volumeMounts:
     - mountPath: /etc/resolv.conf
       name: resolv-conf
       subPath: resolv.conf
       readOnly: true
   volumes:
     - name: resolv-conf
       configMap:
         name: resolv-conf
   

This solution can perhaps can be considered a bit ugly, but currently there aren't many other options. In the future I would hope to see a dns forward feature to Google Cloud or kube-dns.

It's kind of crazy that Google Cloud doesn't offer a DNS forward feature for specified domains/zones.