在Kubernetes上使用Sateful Set部署Redis

2017-08-04

前面写过过几篇关于在Kubernetes上运行有状态服务相关的博文:

最近需要在我们的一个Kubernetes集群上部署Redis，因此重新整理，写一下如何在Kubernetes上使用Sateful Set部署Redis。

1.需求和环境 #

我们的需求是需要部署三节点的Redis主从复制，并部署三个节点的Redis Sentinel实现Redis的高可用。

环境信息如下：

Kubernetes 1.6.7集群
Ceph 11.2.0集群

Kubernetes的官方examples中已经给出了一个在k8s集群上部署Redis的例子Reliable, Scalable Redis on Kubernetes，就是基于Redis主从复制+Sentinel实现的，但是这个例子是以无状态服务形式部署的，如果整个k8s集群重启了，Redis的状态就会丢失，因此不能用于生产环境。但我们可以参考这个例子，以Satefult Set的形式部署。

我们的线上环境主要使用Ceph的块存储RBD作为Kubernetes的存储卷，这里可以将Redis服务的状态保存在Ceph RBD中。

关于Kubernetes和Ceph的部署可以参考我之前写过的几篇博文，这里不再展开：

2.Storage Classes和Dynamic Storage Provision #

Kubernetes 1.6开始Storage Classes和Dynamic Storage Provision已经是稳定可用的了。 StorageClass是Dynamic Storage Provision的基础，k8s的管理员可以定义底层存储平台抽象。用户通过在PVC(Persistent Volume Claim)中通过名字引用StorageClass，PV(Persistent Volume)将使用StorageClass来动态创建，这样就节省了集群管理员手动创建PV的时间。

2.1 在Ceph中创建存储池Pool #

我们需要先在Ceph中创建一个k8s集群专用的Ceph Pool，在创建之前我们先看一下当前Ceph集群中的存储池：

1ceph osd lspools
20 rbd,1 .rgw.root,2 default.rgw.control,3 default.rgw.data.root,4 default.rgw.gc,5 default.rgw.lc,6 default.rgw.log,7 default.rgw.users.uid,8 default.rgw.users.email,9 default.rgw.users.keys,10 default.rgw.buckets.index,11 default.rgw.buckets.data,

一个Ceph集群可以有多个pool，pool是逻辑上的存储池。不同的pool可以有不一样的数据处理方式，例如replica size, placement groups, crush rules,snapshot等等。可以看到因为我们这个环境还是用Ceph的RGW作为我们的对象存储，因此除了默认的名称为rbd的pool外，还有很多rgw的pool。

下面创建一个专门给k8s集群专用的pool kube:

1ceph osd pool create kube 128
2pool 'kube' created
3
4ceph osd lspools
50 rbd,1 .rgw.root,2 default.rgw.control,3 default.rgw.data.root,4 default.rgw.gc,5 default.rgw.lc,6 default.rgw.log,7 default.rgw.users.uid,8 default.rgw.users.email,9 default.rgw.users.keys,10 default.rgw.buckets.index,11 default.rgw.buckets.data,12 kube,

当前这个ceph集群只有3个osd，所以设置pg_num为128，可参考PLACEMENT GROUPS

2.2 配置k8s Node节点访问Ceph #

为了让Kubernetes的Node可以调用rbd，如果Ceph集群和Kubernetes集群不是在相同的机器上，还需要在Kubernetes的Node上安装ceph-common:

1yum install -y ceph-common

接下来在Kubernetes上创建ceph-secret，这个Secret将用于Kubernetes集群的StorageClass上。

我们先查看一下ceph集群上的所有用户列表：

1ceph auth list

这个命令会列出针对Ceph的每种类型的进程已经创建的不同权限的用户，同时也会列出client.admin用户，这个是Ceph集群的管理员用户。

接下来我们创建一个client.kube用户：

1ceph auth get-or-create client.kube
2[client.kube]
3        key = AQAzcYVZ6sbJLhAA7qCBywM+iPRgAG97FtoXIw==

创建好的client.kube用户用户还没有任何权限，下面给其授权：

1ceph auth caps client.kube mon 'allow r' osd 'allow rwx pool=kube'
2updated caps for client.kube

查看用户和权限信息：

1ceph auth get client.kube
2exported keyring for client.kube
3[client.kube]
4        key = AQAzcYVZ6sbJLhAA7qCBywM+iPRgAG97FtoXIw==
5        caps mon = "allow r"
6        caps osd = "allow rwx pool=kube"

因为Kubernetes的Secret需要Base64编码，下面将这个keyring转换成Base64编码：

1ceph auth get-key client.kube | base64
2QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==

接下来创建Secret，ceph-secret.yaml:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: ceph-secret
5  namespace: kube-system
6type: kubernetes.io/rbd
7data:
8  key: QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==

1kubectl create -f ceph-secret.yaml
2secret "ceph-secret" created

2.3 在k8s集群创建StorageClass #

首先检查我们的集群中是否有默认的StorageClass：

1kubectl get storageclass
2No resources found.

我们这里使用的k8s集群是使用ansible部署的Kubernetes 1.6 高可用集群，可以看出我们部署的这个集群并没有创建默认的StorageClass。

我们现在集群中创建默认的Storage Class， storege.yaml文件如下：

 1---
 2apiVersion: storage.k8s.io/v1
 3kind: StorageClass
 4metadata:
 5  name: default
 6  annotations:
 7    storageclass.kubernetes.io/is-default-class: "true"
 8  labels:
 9    kubernetes.io/cluster-service: "true"
10provisioner: kubernetes.io/rbd
11parameters:
12    monitors: 192.168.61.3:6789,192.168.61.4:6789,192.168.61.5:6789
13    adminId: kube
14    adminSecretName: ceph-secret
15    adminSecretNamespace: kube-system
16    pool: kube
17    userId: kube
18    userSecretName: ceph-secret-user

annotations中storageclass.kubernetes.io/is-default-class: "true"表示这个StorageClass是集群默认的StorageClass
provisioner: kubernetes.io/rbd表示这个StorageClass的类型时Ceph RBD
parameters配置了这个StorageClass使用的Ceph集群以及RBD的相关参数
monitors是逗号分隔的Ceph Mon节点地址
adminId指定Ceph client 的ID需要具有能在配置的Ceph RBD Pool中创建镜像的权限。默认值为admin
adminSecret：adminId的Secret Name,该Secret的type必须是"kubernetes.io/rbd"，该参数是必须的
adminSecretNamespace： adminSecret的namespace，默认为"default"
pool： Ceph RBD Pool，默认为"rbd"
userId: Ceph client Id，用来映射RBD镜像。
userSecretName: userId在映射RBD镜像时所需要的Secret的名称。该Secret要求必须出现在和PVC相同的namespace内，并且type必须是"kubernetes.io/rbd"。该参数是必须的

创建这个默认的StorageClass：

1kubectl create -f storage.yaml
2storageclass "default" created
3
4kubectl get storageclass
5NAME                TYPE
6default (default)   kubernetes.io/rbd

(default)表示这个名称为default的StorageClass是k8s集群默认的StorageClass

3.构建Redis的Docker镜像 #

参考Reliable, Scalable Redis on Kubernetes中的Redis镜像，我们的Redis的Dockerfile定制如下：

 1FROM harbor.frognew.com/rg/alpine-glibc:0.1
 2
 3RUN apk add --no-cache redis sed bash
 4
 5COPY redis-master.conf /redis-master/redis.conf
 6COPY redis-slave.conf /redis-slave/redis.conf
 7COPY run.sh /run.sh
 8RUN chmod u+x /run.sh
 9CMD [ "/run.sh" ]
10
11ENTRYPOINT [ "bash", "-c" ]

alpine-glibc:0.1是我们的基础镜像，在alpine:3.6的基础上增加了glibc，并将时区设置为Asia/Shanghai

参考Reliable, Scalable Redis on Kubernetes中的run.sh做如下定制，原来的run.sh不支持对redis设置密码，加上从环境变量$REDIS_PASS读取redis密码：

 1#!/bin/bash
 2
 3# Copyright 2014 The Kubernetes Authors.
 4#
 5# Licensed under the Apache License, Version 2.0 (the "License");
 6# you may not use this file except in compliance with the License.
 7# You may obtain a copy of the License at
 8#
 9#     http://www.apache.org/licenses/LICENSE-2.0
10#
11# Unless required by applicable law or agreed to in writing, software
12# distributed under the License is distributed on an "AS IS" BASIS,
13# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14# See the License for the specific language governing permissions and
15# limitations under the License.
16
17function launchmaster() {
18  if [[ ! -e /redis-master-data ]]; then
19    echo "Redis master data doesn't exist, data won't be persistent!"
20    mkdir /redis-master-data
21  fi
22  sed -i "s/%redis-pass%/${REDIS_PASS}/" /redis-master/redis.conf
23  redis-server /redis-master/redis.conf --protected-mode no
24}
25
26function launchsentinel() {
27  while true; do
28    master=$(redis-cli -a $REDIS_PASS -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1)
29    if [[ -n ${master} ]]; then
30      master="${master//\"}"
31    else
32      master=${REDIS_MASTER_SERVICE_HOST}
33    fi
34
35    redis-cli -a $REDIS_PASS -h ${master} INFO
36    if [[ "$?" == "0" ]]; then
37      break
38    fi
39    echo "Connecting to master failed.  Waiting..."
40    sleep 10
41  done
42
43  sentinel_conf=sentinel.conf
44
45  echo "sentinel monitor mymaster ${master} 6379 2" > ${sentinel_conf}
46  echo "sentinel auth-pass mymaster ${REDIS_PASS}" >> ${sentinel_conf}
47  echo "sentinel down-after-milliseconds mymaster 60000" >> ${sentinel_conf}
48  echo "sentinel failover-timeout mymaster 180000" >> ${sentinel_conf}
49  echo "sentinel parallel-syncs mymaster 1" >> ${sentinel_conf}
50  echo "bind 0.0.0.0" >> ${sentinel_conf}
51
52  redis-sentinel ${sentinel_conf} --protected-mode no
53}
54
55function launchslave() {
56  while true; do
57    master=$(redis-cli -a $REDIS_PASS -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT} --csv SENTINEL get-master-addr-by-name mymaster | tr ',' ' ' | cut -d' ' -f1)
58    if [[ -n ${master} ]]; then
59      master="${master//\"}"
60    else
61      echo "Failed to find master."
62      sleep 60
63      exit 1
64    fi 
65    redis-cli -a $REDIS_PASS -h ${master} INFO
66    if [[ "$?" == "0" ]]; then
67      break
68    fi
69    echo "Connecting to master failed.  Waiting..."
70    sleep 10
71  done
72  sed -i "s/%master-ip%/${master}/" /redis-slave/redis.conf
73  sed -i "s/%master-port%/6379/" /redis-slave/redis.conf
74  sed -i "s/%redis-pass%/${REDIS_PASS}/" /redis-slave/redis.conf
75  redis-server /redis-slave/redis.conf --protected-mode no
76}
77
78if [[ "${MASTER}" == "true" ]]; then
79  launchmaster
80  exit 0
81fi
82
83if [[ "${SENTINEL}" == "true" ]]; then
84  launchsentinel
85  exit 0
86fi
87
88launchslave

这个脚本根据环境变量MASTER, SENTINEL来判断是启动不同类型的redis进程，如果MASTER为true，则启动redis master，否则如果SENTINEL为true则启动redis sentinel，否则启动redis salve
从环境变量REDIS_PASS中读取并设置redis的密码

redis-master.conf的配置文件内容如下：

 1daemonize no
 2pidfile /var/run/redis.pid
 3port 6379
 4tcp-backlog 511
 5bind 0.0.0.0
 6timeout 0
 7tcp-keepalive 60
 8loglevel notice
 9logfile ""
10databases 16
11save 900 1
12save 300 10
13save 60 10000
14stop-writes-on-bgsave-error yes
15rdbcompression yes
16rdbchecksum yes
17dbfilename dump.rdb
18dir /redis-master-data
19slave-serve-stale-data yes
20rename-command FLUSHALL ""
21rename-command FLUSHDB ""
22slave-read-only yes
23repl-diskless-sync no
24repl-diskless-sync-delay 5
25repl-disable-tcp-nodelay no
26slave-priority 100
27requirepass %redis-pass%
28appendonly yes
29appendfilename "appendonly.aof"
30appendfsync everysec
31no-appendfsync-on-rewrite no
32auto-aof-rewrite-percentage 100
33auto-aof-rewrite-min-size 64mb
34aof-load-truncated yes
35lua-time-limit 5000
36slowlog-log-slower-than 10000
37slowlog-max-len 128
38latency-monitor-threshold 0
39notify-keyspace-events ""
40hash-max-ziplist-entries 512
41hash-max-ziplist-value 64
42list-max-ziplist-entries 512
43list-max-ziplist-value 64
44set-max-intset-entries 512
45zset-max-ziplist-entries 128
46zset-max-ziplist-value 64
47hll-sparse-max-bytes 3000
48activerehashing yes
49client-output-buffer-limit normal 0 0 0
50client-output-buffer-limit slave 256mb 64mb 60
51client-output-buffer-limit pubsub 32mb 8mb 60
52hz 10
53aof-rewrite-incremental-fsync yes

redis-slave.conf配置文件内容如下：

 1daemonize no
 2pidfile /var/run/redis.pid
 3port 6379
 4tcp-backlog 511
 5bind 0.0.0.0
 6timeout 0
 7tcp-keepalive 60
 8loglevel notice
 9logfile ""
10databases 16
11save 900 1
12save 300 10
13save 60 10000
14stop-writes-on-bgsave-error yes
15rdbcompression yes
16rdbchecksum yes
17dbfilename dump.rdb
18dir "/data"
19slaveof %master-ip% %master-port% 
20masterauth %redis-pass%
21slave-serve-stale-data yes
22rename-command FLUSHALL ""
23rename-command FLUSHDB ""
24slave-read-only yes
25repl-diskless-sync no
26repl-diskless-sync-delay 5
27repl-disable-tcp-nodelay no
28slave-priority 100
29requirepass %redis-pass%
30appendonly yes
31appendfilename "appendonly.aof"
32appendfsync everysec
33no-appendfsync-on-rewrite no
34auto-aof-rewrite-percentage 100
35auto-aof-rewrite-min-size 64mb
36aof-load-truncated yes
37lua-time-limit 5000
38slowlog-log-slower-than 10000
39slowlog-max-len 128
40latency-monitor-threshold 0
41notify-keyspace-events ""
42hash-max-ziplist-entries 512
43hash-max-ziplist-value 64
44list-max-ziplist-entries 512
45list-max-ziplist-value 64
46set-max-intset-entries 512
47zset-max-ziplist-entries 128
48zset-max-ziplist-value 64
49hll-sparse-max-bytes 3000
50activerehashing yes
51client-output-buffer-limit normal 0 0 0
52client-output-buffer-limit slave 256mb 64mb 60
53client-output-buffer-limit pubsub 32mb 8mb 60
54hz 10
55aof-rewrite-incremental-fsync yes

构建redis镜像并推送到我们的私有仓库：

1docker build -t harbor.frognew.com/rg/redis:1.0 .
2docker push harbor.frognew.com/rg/rg/redis

4.在Kubernetes集群上部署Redis #

假设我们的redis要部署在devops这个namespace下，先在这个namespace下创建ceph-secret-user这个Secret：

1apiVersion: v1
2kind: Secret
3metadata:
4  name: ceph-secret-user
5  namespace: devops
6type: kubernetes.io/rbd
7data:
8  key: QVFBemNZVlo2c2JKTGhBQTdxQ0J5d00raVBSZ0FHOTdGdG9YSXc9PQ==

1kubectl crate -f ceph-secret-user.yaml

4.1 redis-master.statefulset.yaml #

redis-master.statefulset.yaml是redis master的Service和StatefulSet。

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  name: redis-master
 5  namespace: devops
 6  labels:
 7    name: redis-master
 8spec:
 9  ports:
10    - port: 6379
11  selector:
12    redis-master: "true"
13    
14---
15apiVersion: apps/v1beta1
16kind: StatefulSet
17metadata:
18  name: redis-master
19  namespace: devops
20  labels:
21    name: redis-master
22spec:
23  serviceName: redis-master
24  replicas: 1
25  template:
26    metadata:
27      labels:
28        app: redis-master
29        redis-master: "true"
30    spec:
31      terminationGracePeriodSeconds: 10
32      containers:
33      - name: redis
34        image: harbor.frognew.com/rg/redis:1.0
35        imagePullPolicy: IfNotPresent
36        ports:
37        - containerPort: 6379
38        env:
39        - name: MASTER
40          value: "true"
41        - name: REDIS_PASS
42          valueFrom:
43            secretKeyRef:
44              name: devopssecret
45              key: redisAuthPass
46        resources:
47          requests:
48            memory: "256Mi"
49            cpu: "250m"
50          limits:
51            memory: "512Mi"
52            cpu: "500m"
53        volumeMounts:
54        - name: redis-master-volume
55          mountPath: /redis-master-data
56      imagePullSecrets: 
57        - name: regsecret
58  volumeClaimTemplates:
59  - metadata:
60      name: redis-master-volume
61    spec:
62      accessModes: [ "ReadWriteOnce" ]
63      resources:
64        requests:
65          storage: 5Gi

通过设置环境变量MASTER为true，表明以master形式启动redis，而环境变量REDIS_PASS从devopssecret这个Secret中获取的值，这里略过devopssecret这个Secret的内容
volumeClaimTemplates中定义了PVC，因为没有给定storageClassName，所以将使用我们前面创建的默认的StorageClass，会根据PVC动态创建StatefulSet中Pod所需的PV

4.2 redis-sentinel.statefulset.yaml #

redis-sentinel.statefulset.yaml定义了redis-sentinel的Service和StatefulSet：

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  name: redis-sentinel
 5  namespace: devops
 6  labels:
 7    name: redis-sentinel
 8spec:
 9  ports:
10    - port: 26379
11      targetPort: 26379
12  selector:
13    redis-sentinel: "true"
14---
15apiVersion: apps/v1beta1
16kind: StatefulSet
17metadata:
18  name: redis-sentinel
19  namespace: devops
20spec:
21  serviceName: redis-sentinel
22  replicas: 3
23  template:
24    metadata:
25      labels:
26        redis-sentinel: "true"
27    spec:
28      terminationGracePeriodSeconds: 10
29      containers:
30      - name: redis-sentinel
31        image: harbor.frognew.com/rg/redis:1.0
32        imagePullPolicy: IfNotPresent
33        ports:
34        - containerPort: 26379
35          name: redis-sentinel
36        env:
37          - name: SENTINEL
38            value: "true"
39          - name: REDIS_PASS
40            valueFrom:
41              secretKeyRef:
42                name: devopssecret
43                key: redisAuthPass
44      imagePullSecrets: 
45        - name: regsecret

sentinel的启动逻辑可以查看Docker镜像中的run.sh中launchsentinel()的逻辑

4.5 redis.statefulset.yaml #

redis.statefulset.yaml定义了redis slave的Service和SatefulSet：

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  name: redis
 5  namespace: devops
 6  labels:
 7    app: redis
 8spec:
 9  ports:
10    - port: 6379
11  clusterIP: None
12  selector:
13    app: redis
14---
15apiVersion: apps/v1beta1
16kind: StatefulSet
17metadata:
18  name: redis
19  namespace: devops
20  labels:
21    name: redis
22spec:
23  serviceName: redis
24  replicas: 2
25  template:
26    metadata:
27      labels:
28        app: redis
29    spec:
30      terminationGracePeriodSeconds: 10
31      containers:
32      - name: redis
33        image: harbor.frognew.com/rg/redis:1.0
34        imagePullPolicy: IfNotPresent
35        ports:
36        - containerPort: 6379
37        env:
38        - name: REDIS_PASS
39          valueFrom:
40            secretKeyRef:
41              name: devopssecret
42              key: redisAuthPass
43        resources:
44          requests:
45            memory: "256Mi"
46            cpu: "250m"
47          limits:
48            memory: "512Mi"
49            cpu: "500m"
50        volumeMounts:
51        - name: redis-volume
52          mountPath: /data
53      imagePullSecrets: 
54        - name: regsecret
55  volumeClaimTemplates:
56  - metadata:
57      name: redis-volume
58    spec:
59      accessModes: [ "ReadWriteOnce" ]
60      resources:
61        requests:
62          storage: 5Gi

volumeClaimTemplates中定义了PVC，因为没有给定storageClassName，所以将使用我们前面创建的默认的StorageClass，会根据PVC动态创建StatefulSet中Pod所需的PV。

4.4 以StatefulSet的形式部署Redis #

下面实际操作一遍基于StatefulSet的Redis的部署。

先创建redis-master的Service和StatefulSet：

1kubectl create -f redis-master.statefulset.yaml
2service "redis-master" created
3statefulset "redis-master" created

确保这redis master Pod处于running状态：

1kubectl get pods -l redis-master="true" -n devops
2NAME             READY     STATUS    RESTARTS   AGE
3redis-master-0   1/1       Running   0          48s
4
5kubectl get svc -l name="redis-master" -n devops
6NAME           CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
7redis-master   10.104.132.220   <none>        6379/TCP   1m

下面创建redis-sentinel的Service和StatefulSet：

1kubectl create -f redis-sentinel.statefulset.yaml
2service "redis-sentinel" created
3statefulset "redis-sentinel" created
4
5kubectl get svc -l name="redis-sentinel" -n devops -o wide
6NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE       SELECTOR
7redis-sentinel   10.97.4.9    <none>        26379/TCP   16s       redis-sentinel=true

查看StatefulSet确保DESIRED和CURRENT的数量是相同的。

1kubectl get statefulset -n devops
2NAME             DESIRED   CURRENT   AGE
3redis-master     1         1         3m
4redis-sentinel   3         3         42s

查看sentinel Pod：

1kubectl get pod -l redis-sentinel="true" -n devops
2NAME               READY     STATUS    RESTARTS   AGE
3redis-sentinel-0   1/1       Running   0          1m
4redis-sentinel-1   1/1       Running   0          1m
5redis-sentinel-2   1/1       Running   0          1m

下面创建redis slave的Service和StatefulSet：

1kubectl create -f redis.statefulset.yaml
2service "redis" created
3statefulset "redis" created

注意上面的过程中，在创建redis master和slave的stateful set时可能需要一定的时间，因为涉及到PVC, PV, rbd image的创建，耐心等待。

因为redis-master这个StatefulSet的副本数为1，redis slave这个SatefulSet中的副本数为2，所以我们可以看到集群中创建了3个PVC，并创建了3个PV:

 1kubectl get pvc -n devops
 2NAME                                 STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
 3redis-master-volume-redis-master-0   Bound     pvc-fd2c30e3-7b14-11e7-ad4a-1866da8c6175   5Gi        RWO           default        10m
 4redis-volume-redis-0                 Bound     pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175   5Gi        RWO           default        7m
 5redis-volume-redis-1                 Bound     pvc-6a96951c-7b17-11e7-ad4a-1866da8c6175   5Gi        RWO           default        7m
 6
 7
 8kubectl get pv -n devops
 9NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                       STORAGECLASS   REASON    AGE
10pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175   5Gi        RWO           Delete          Bound     devops/redis-volume-redis-0                 default                  10m
11pvc-6a96951c-7b17-11e7-ad4a-1866da8c6175   5Gi        RWO           Delete          Bound     devops/redis-volume-redis-1                 default                  7m
12pvc-fd2c30e3-7b14-11e7-ad4a-1866da8c6175   5Gi        RWO           Delete          Bound     devops/redis-master-volume-redis-master-0   default                  7m

一定要确认STATUS的状态为Bound，如果不是可以通过kubectl describe pvc <pvc-name> -n <namespace>查看具体的事件。实际上rbd image的创建是由controller-manager调用rbd命令完成的，所以如果有问题也可以看一下controller-manager的日志。

下面的decribe pv命令详细打印出了这个PV已经使用Ceph RBD Image:

 1kubectl describe pv pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175 -n devops
 2Name:           pvc-65a2ba2c-7b17-11e7-ad4a-1866da8c6175
 3Labels:         <none>
 4Annotations:    pv.kubernetes.io/bound-by-controller=yes
 5                pv.kubernetes.io/provisioned-by=kubernetes.io/rbd
 6StorageClass:   default
 7Status:         Bound
 8Claim:          devops/redis-volume-redis-0
 9Reclaim Policy: Delete
10Access Modes:   RWO
11Capacity:       5Gi
12Message:
13Source:
14    Type:               RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
15    CephMonitors:       [192.168.61.3:6789 192.168.61.4:6879 192.168.61.5:6789]
16    RBDImage:           kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
17    FSType:
18    RBDPool:            kube
19    RadosUser:          kube
20    Keyring:            /etc/ceph/keyring
21    SecretRef:          &{ceph-secret-user}
22    ReadOnly:           false
23Events:                 <none>

另外可以在Ceph集群中查看创建的rbd image：

 1rbd list kube
 2kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
 3kubernetes-dynamic-pvc-8b1be6fc-7a7b-11e7-ac3c-1866da8c2fcd
 4kubernetes-dynamic-pvc-cce6429c-7a7a-11e7-ac3c-1866da8c2fcd
 5
 6rbd info -p kube --image kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd
 7rbd image 'kubernetes-dynamic-pvc-856cff45-7a7b-11e7-ac3c-1866da8c2fcd':
 8        size 5120 MB in 1280 objects
 9        order 22 (4096 kB objects)
10        block_name_prefix: rb.0.3e17b.238e1f29
11        format: 1

1kubectl get statefulset -n devops
2NAME             DESIRED   CURRENT   AGE
3redis            2         2         5m
4redis-master     1         1         10m
5redis-sentinel   3         3         7m

我们重点来看一下redis statefulset和redis service：

1kubectl get svc -l app="redis" -n devops
2NAME      CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
3redis     None         <none>        6379/TCP   7m
4
5
6kubectl get pod -l app="redis" -n devops
7NAME      READY     STATUS    RESTARTS   AGE
8redis-0   1/1       Running   0          7m
9redis-1   1/1       Running   0          7m

注意redis service的CLUSTER-IP为None，这是由有状态服务的特征决定的。有状态服务具有以下特征：

要求有稳定的网络身份，即唯一不变的hostname，并保存在DNS中。hostname是由statefulset的名字后边跟随"-序号"组成，这里是redis-1, redis-2。同时每个Pod的网络身份也是通过Service定义被创建出来了，根据Service的定义，通过ClusterIp:None指定，该Service将在DNS生成一条没有ClusterIP的记录。
要求有持久稳定的存储，通过PVC和PV提供。这里使用了Kubernetes的通过Dynamic Storage Provision特性，PV使用StorageClass来动态创建。
redis-0,redis-1这两个是reddis的slave节点。

最后我们来看一下k8s集群中redis节点：

1kubectl get statefulset -n devops
2NAME             DESIRED   CURRENT   AGE
3redis            2         2         8m
4redis-master     1         1         12m
5redis-sentinel   3         3         10m

我们以StatefulSet的形式部署了1个master, 2个slave, 3个sentinel。当其中master节点发生故障时，sentinel会从剩余redis节点中选举新的master并切换。 3个redis节点的数据都是保存在ceph rbd中。