在JupyterHub for K8s使用GlusterFS存储

【推荐】2019 Java 开发者跳槽指南.pdf(吐血整理) >>>

在Kubernetes中使用GlusterFS(https://www.gluster.org/)有endpoint（外置存储）和heketi（k8s内置GlusterFS服务）两种方式。这里主要介绍使用endpoint将GlusterFS存储设置到JupyterHub for K8s中使用。为了简化起见，使用缺省的JupyterHub helm进行安装。按照《快速设置JupyterHub for K8s》安装后，会在Jupyterhub的安装命名空间下出现一个hub-db-dir的pvc，我将使用GlusterFS的volume来提供这个pvc。

1、创建GlusterFS的volume的endpoint

保存下面内容到文件0a-glusterfs-gvzr00-endpoint-jupyter.yaml：

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-gvzr00
  namespace: jupyter
subsets:
- addresses:
  - ip: 10.1.1.193
  - ip: 10.1.1.205
  - ip: 10.1.1.112
  ports:
  - port: 10000
    protocol: TCP

其中的addresss是自己的GlusterFS的对应复制卷的peer节点访问地址（这里是三节点复制提供冗余存储）。

创建服务，保存下面内容到文件0b-glusterfs-gvzr00-service-jupyter.yaml ：

apiVersion: v1
kind: Service
metadata:
  name: glusterfs-gvzr00
  namespace: jupyter
spec:
  ports:
  - port: 10000
    protocol: TCP
    targetPort: 10000
  sessionAffinity: None
  type: ClusterIP

2、创建jupyterhub系统的hub-db-dir的pv和pvc

创建jupyterhub主服务程序的pv和pvc，用于存储系统数据。

2.1 创建pv

保存下面内容到文件 1a-glusterfs-gvzr00-pv-jupyter-hub.yaml：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: hub-db-dir
  namespace: jupyter
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: "glusterfs-gvzr00"
    path: "gvzr00/jupyterhub/hub-db-dir"
    readOnly: false

2.2 创建pvc

首先删除pvc。

kubectl delete pvc/hub-db-dir -n jhub

保存下面内容到文件1b-glusterfs-gvzr00-pvc-jupyter-hub.yaml：

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: hub-db-dir
  namespace: jupyter
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 8Gi

3、创建jupyterhub系统的supermap的pv和pvc

每个用户自己的pv和pvc，用于存储notebook server的用户数据。

3.1 创建pv

保存下面内容到文件2a-glusterfs-gvzr00-pv-jupyter-supermap.yaml：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: claim-supermap
  namespace: jupyter
spec:
  capacity:
    storage: 16Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: "glusterfs-gvzr00"
    path: "gvzr00/jupyterhub/claim-supermap"
    readOnly: false

3.2 创建pvc

保存下面内容到文件2b-glusterfs-gvzr00-pvc-jupyter-supermap.yaml：

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: claim-supermap
  namespace: jupyter
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 16Gi

4、运行设置

根据自己的集群地址和存储容量修改上面的几个文件。

4.1 创建pv和pvc

保存下面内容到文件apply.sh：

echo "Create endpoint and svc, glusterfs-gvzr00 ..."
kubectl apply -f 0a-glusterfs-gvzr00-endpoint-jupyter.yaml
kubectl apply -f 0b-glusterfs-gvzr00-service-jupyter.yaml

echo "Create pv and pvc, hub-db-dir ..."
kubectl apply -f 1a-glusterfs-gvzr00-pv-jupyter-hub.yaml
kubectl apply -f 1b-glusterfs-gvzr00-pvc-jupyter-hub.yaml

echo "Create pv and pvc, claim--supermap ..."
kubectl apply -f 2a-glusterfs-gvzr00-pv-jupyter-supermap.yaml
kubectl apply -f 2b-glusterfs-gvzr00-pvc-jupyter-supermap.yaml

echo "Finished."
echo ""

然后运行apply.sh。

4.2 删除pv和pvc

保存下面内容到文件delete.sh

# 
echo "Delete pv and pvc, hub-db-dir ..."
kubectl delete pvc/hub-db-dir -n jupyter
kubectl delete pv/hub-db-dir -n jupyter

echo "Delete pv and pvc, claim--supermap ..."
kubectl delete pvc/claim-supermap -n jupyter
kubectl delete pv/claim--supermap -n jupyter

echo "Delete endpoint and svc, glusterfs-gvzr00 ..."
kubectl delete svc/glusterfs-gvzr00 -n jupyter
kubectl delete ep/glusterfs-gvzr00 -n jupyter

echo "Finished."
echo ""

需要全部删除时，运行delete.sh。

4.3 查看pv和pvc

通过Dashboard或命令：

kubectl get pv -n jhub

kubectl get pvc -n jhub

注意：如果升级到最近的python3版本，JupyterHub运行时出现错误，Notebook Server无法启动，进pod日志发现提示信息"NoneType"，可以采用下面的方法修复：

kubectl patch deploy -n jupyter hub --type json \
--patch '[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["bash", "-c", "\nmkdir -p ~/hotfix\ncp \
-r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix\nls -R ~/hotfix\npatch ~/hotfix/kubespawner/spawner.py \
<< EOT\n72c72\n<             key=lambda x: x.last_timestamp,\n---\n>             key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,\nEOT\n\nPYTHONPATH=$HOME/hotfix \
jupyterhub --config /srv/jupyterhub_config.py --upgrade-db\n"]}]'

再去访问JupyterHub的服务，恢复正常。

5、更多参考

来源：oschina

链接：https://my.oschina.net/u/2306127/blog/3153940

标签

GlusterFS

JupyterHub