TL;DR - The solution to the problem, thanks to Paul
If you have the problem described below, the easiest way to solve it is to execute the followin
Here is the complete single node k8s start up script that makes my problem go away. Thanks to Paul Morie for providing me w/ the solution (the magic first line in the script).
Update
Here is an update that Paul sent me on why chcon is used:
basically what it does is change the SELinux type for the volume directory that holds all the pod
volumes to svirt_sandbox_file_t, which is the context that most SELinux policies allow containers
(typically running with svirt_lxc_net_t) to use.
So, TLDR, that command makes the kube volume directory usable by docker containers (though of course containers
only have access to the volumes that are consumed in their pod and then mounted into the container).
My understanding of this is that normally Docker container run in isolation and can't see each others file systems, the chcon allows us to break this isolation, in a controlled fashion, such that only using volume mount directives is this sharing allowed to happen. This explanation seems relevant.
# magic selinux context set command is required. for details, see: http://stackoverflow.com/questions/34777111/cannot-create-a-shared-volume-mount-via-emptydir-on-single-node-kubernetes-on
#
sudo chcon -Rt svirt_sandbox_file_t /var/lib/kubelet
docker run --net=host -d gcr.io/google_containers/etcd:2.0.12 /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data
docker run \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:ro \
--volume=/dev:/dev \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
--volume=/var/run:/var/run:rw \
--net=host \
--pid=host \
--privileged=true \
-d \
gcr.io/google_containers/hyperkube:v1.0.1 \
/hyperkube kubelet --containerized --hostname-override="127.0.0.1" --address="0.0.0.0" --api-servers=http://localhost:8080 --config=/etc/kubernetes/manifests
docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v1.0.1 /hyperkube proxy --master=http://127.0.0.1:8080 --v=2
sleep 20 # give everything time to launch