使用helm在k8s上部署kafka
2019-07-08
1.配置helm chart repo #
kafka的helm chart还在孵化当中，使用前需要添加incubator的repo：helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator。
肉身在国内需要设置azure提供的镜像库地址:
1helm repo add stable http://mirror.azure.cn/kubernetes/charts
2helm repo add incubator http://mirror.azure.cn/kubernetes/charts-incubator
3
4helm repo list
5NAME URL
6stable http://mirror.azure.cn/kubernetes/charts
7local http://127.0.0.1:8879/charts
8incubator http://mirror.azure.cn/kubernetes/charts-incubator
2.创建Kafka和Zookeeper的Local PV #
2.1 创建Kafka的Local PV #
这里的部署环境是本地的测试环境，存储选择Local Persistence Volumes。首先，在k8s集群上创建本地存储的StorageClass local-storage.yaml：
1apiVersion: storage.k8s.io/v1
2kind: StorageClass
3metadata:
4 name: local-storage
5provisioner: kubernetes.io/no-provisioner
6volumeBindingMode: WaitForFirstConsumer
7reclaimPolicy: Retain
1kubectl apply -f local-storage.yaml
2storageclass.storage.k8s.io/local-storage created
这里要在node1、node2这两个k8s节点上部署3个kafka的broker节点，因此先在node1、node2上创建这3个kafka broker节点的Local PV kafka-local-pv.yaml:
1apiVersion: v1
2kind: PersistentVolume
3metadata:
4 name: datadir-kafka-0
5spec:
6 capacity:
7 storage: 5Gi
8 accessModes:
9 - ReadWriteOnce
10 persistentVolumeReclaimPolicy: Retain
11 storageClassName: local-storage
12 local:
13 path: /home/kafka/data-0
14 nodeAffinity:
15 required:
16 nodeSelectorTerms:
17 - matchExpressions:
18 - key: kubernetes.io/hostname
19 operator: In
20 values:
21 - node1
22---
23apiVersion: v1
24kind: PersistentVolume
25metadata:
26 name: datadir-kafka-1
27spec:
28 capacity:
29 storage: 5Gi
30 accessModes:
31 - ReadWriteOnce
32 persistentVolumeReclaimPolicy: Retain
33 storageClassName: local-storage
34 local:
35 path: /home/kafka/data-1
36 nodeAffinity:
37 required:
38 nodeSelectorTerms:
39 - matchExpressions:
40 - key: kubernetes.io/hostname
41 operator: In
42 values:
43 - node2
44---
45apiVersion: v1
46kind: PersistentVolume
47metadata:
48 name: datadir-kafka-2
49spec:
50 capacity:
51 storage: 5Gi
52 accessModes:
53 - ReadWriteOnce
54 persistentVolumeReclaimPolicy: Retain
55 storageClassName: local-storage
56 local:
57 path: /home/kafka/data-2
58 nodeAffinity:
59 required:
60 nodeSelectorTerms:
61 - matchExpressions:
62 - key: kubernetes.io/hostname
63 operator: In
64 values:
65 - node2
1kubectl apply -f kafka-local-pv.yaml
根据上面创建的local pv，在node1上创建目录/home/kafka/data-0，在node2上创建目录/home/kafka/data-1和/home/kafka/data-2。
1# node1
2mkdir -p /home/kafka/data-0
3
4# node2
5mkdir -p /home/kafka/data-1
6mkdir -p /home/kafka/data-2
2.2 创建Zookeeper的Local PV #
这里要在node1、node2这两个k8s节点上部署3个zookeeper节点，因此先在node1、node2上创建这3个zookeeper节点的Local PV zookeeper-local-pv.yaml:
1apiVersion: v1
2kind: PersistentVolume
3metadata:
4 name: data-kafka-zookeeper-0
5spec:
6 capacity:
7 storage: 5Gi
8 accessModes:
9 - ReadWriteOnce
10 persistentVolumeReclaimPolicy: Retain
11 storageClassName: local-storage
12 local:
13 path: /home/kafka/zkdata-0
14 nodeAffinity:
15 required:
16 nodeSelectorTerms:
17 - matchExpressions:
18 - key: kubernetes.io/hostname
19 operator: In
20 values:
21 - node1
22---
23apiVersion: v1
24kind: PersistentVolume
25metadata:
26 name: data-kafka-zookeeper-1
27spec:
28 capacity:
29 storage: 5Gi
30 accessModes:
31 - ReadWriteOnce
32 persistentVolumeReclaimPolicy: Retain
33 storageClassName: local-storage
34 local:
35 path: /home/kafka/zkdata-1
36 nodeAffinity:
37 required:
38 nodeSelectorTerms:
39 - matchExpressions:
40 - key: kubernetes.io/hostname
41 operator: In
42 values:
43 - node2
44---
45apiVersion: v1
46kind: PersistentVolume
47metadata:
48 name: data-kafka-zookeeper-2
49spec:
50 capacity:
51 storage: 5Gi
52 accessModes:
53 - ReadWriteOnce
54 persistentVolumeReclaimPolicy: Retain
55 storageClassName: local-storage
56 local:
57 path: /home/kafka/zkdata-2
58 nodeAffinity:
59 required:
60 nodeSelectorTerms:
61 - matchExpressions:
62 - key: kubernetes.io/hostname
63 operator: In
64 values:
65 - node2
1kubectl apply -f zookeeper-local-pv.yaml
根据上面创建的local pv，在node1上创建目录/home/kafka/zkdata-0，在node2上创建目录/home/kafka/zkdata-1和/home/kafka/zkdata-2。
1# node1
2mkdir -p /home/kafka/zkdata-0
3
4# node2
5mkdir -p /home/kafka/zkdata-1
6mkdir -p /home/kafka/zkdata-2
3.部署Kafka #
编写kafka chart的vaule文件kafka-values.yaml:
1replicas: 3
2tolerations:
3- key: node-role.kubernetes.io/master
4 operator: Exists
5 effect: NoSchedule
6- key: node-role.kubernetes.io/master
7 operator: Exists
8 effect: PreferNoSchedule
9persistence:
10 storageClass: local-storage
11 size: 5Gi
12zookeeper:
13 persistence:
14 enabled: true
15 storageClass: local-storage
16 size: 5Gi
17 replicaCount: 3
18 image:
19 repository: gcr.azk8s.cn/google_samples/k8szk
20 tolerations:
21 - key: node-role.kubernetes.io/master
22 operator: Exists
23 effect: NoSchedule
24 - key: node-role.kubernetes.io/master
25 operator: Exists
26 effect: PreferNoSchedule
- 安装过程需要使用到
gcr.io/google_samples/k8szk:v3等docker镜像，切换成使用azure的GCR Proxy Cache：gcr.azk8s.cn。
1helm install --name kafka --namespace kafka -f kafka-values.yaml incubator/kafka
最后需要确认所有的pod都处于running状态:
1kubectl get pod -n kafka -o wide
2NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
3kafka-0 1/1 Running 0 12m 10.244.0.61 node1 <none> <none>
4kafka-1 1/1 Running 0 6m3s 10.244.1.12 node2 <none> <none>
5kafka-2 1/1 Running 0 2m26s 10.244.1.13 node2 <none> <none>
6kafka-zookeeper-0 1/1 Running 0 12m 10.244.1.9 node2 <none> <none>
7kafka-zookeeper-1 1/1 Running 0 11m 10.244.1.10 node2 <none> <none>
8kafka-zookeeper-2 1/1 Running 0 11m 10.244.1.11 node2 <none> <none>
9
10kubectl get statefulset -n kafka
11NAME READY AGE
12kafka 3/3 22m
13kafka-zookeeper 3/3 22m
14
15kubectl get service -n kafka
16NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
17kafka ClusterIP 10.102.8.192 <none> 9092/TCP 31m
18kafka-headless ClusterIP None <none> 9092/TCP 31m
19kafka-zookeeper ClusterIP 10.110.43.203 <none> 2181/TCP 31m
20kafka-zookeeper-headless ClusterIP None <none> 2181/TCP,3888/TCP,2888/TCP 31m
可以看到当前kafka的helm chart，采用StatefulSet的形式部署了kafka和zookeeper，而我们通过Local PV的形式，将kafka-0调度到node1上，将kafka-1和kafka-2调度到node2上。
4.安装后的测试 #
在k8s集群内运行下面的客户端Pod，访问kafka broker进行测试：
1apiVersion: v1
2kind: Pod
3metadata:
4 name: testclient
5 namespace: kafka
6spec:
7 containers:
8 - name: kafka
9 image: confluentinc/cp-kafka:5.0.1
10 command:
11 - sh
12 - -c
13 - "exec tail -f /dev/null"
创建并进入testclient容器内：
1kubectl apply -f testclient.yaml
2kubectl -n kafka exec testclient -it sh
查看kafka相关命令：
1ls /usr/bin/ | grep kafka
2kafka-acls
3kafka-broker-api-versions
4kafka-configs
5kafka-console-consumer
6kafka-console-producer
7kafka-consumer-groups
8kafka-consumer-perf-test
9kafka-delegation-tokens
10kafka-delete-records
11kafka-dump-log
12kafka-log-dirs
13kafka-mirror-maker
14kafka-preferred-replica-election
15kafka-producer-perf-test
16kafka-reassign-partitions
17kafka-replica-verification
18kafka-run-class
19kafka-server-start
20kafka-server-stop
21kafka-streams-application-reset
22kafka-topics
23kafka-verifiable-consumer
24kafka-verifiable-producer
创建一个Topic test1：
1kafka-topics --zookeeper kafka-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1
查看的Topic：
1kafka-topics --zookeeper kafka-zookeeper:2181 --list
2test1
5.总结 #
当前基于Helm官方仓库的chartincubator/kafka在k8s上部署的kafka，使用的镜像是confluentinc/cp-kafka:5.0.1。
即部署的是Confluent公司提供的kafka版本。Confluent Platform Kafka(简称CP Kafka)提供了一些Apache Kafka没有的高级特性，例如跨数据中心备份、Schema注册中心以及集群监控工具等。CP Kafka目前分为免费版本和企业版两种，免费版除了Apache Kafka的标准组件外还包含Schema注册中心和Rest Proxy。
Confluent Platform and Apache Kafka Compatibility中给出了Confluent Kafka和Apache Kafka的版本对应关系，可以看出这里安装的cp 5.0.1对应Apache Kafka的2.0.x。
进入一个broker容器中，查看：
1ls /usr/share/java/kafka | grep kafka
2kafka-clients-2.0.1-cp1.jar
3kafka-log4j-appender-2.0.1-cp1.jar
4kafka-streams-2.0.1-cp1.jar
5kafka-streams-examples-2.0.1-cp1.jar
6kafka-streams-scala_2.11-2.0.1-cp1.jar
7kafka-streams-test-utils-2.0.1-cp1.jar
8kafka-tools-2.0.1-cp1.jar
9kafka.jar
10kafka_2.11-2.0.1-cp1-javadoc.jar
11kafka_2.11-2.0.1-cp1-scaladoc.jar
12kafka_2.11-2.0.1-cp1-sources.jar
13kafka_2.11-2.0.1-cp1-test-sources.jar
14kafka_2.11-2.0.1-cp1-test.jar
15kafka_2.11-2.0.1-cp1.jar
可以看到对应apache kafka的版本号是2.11-2.0.1，前面2.11是Scala编译器的版本，Kafka的服务器端代码是使用Scala语言开发的，后边2.0.1是Kafka的版本。
即CP Kafka 5.0.1是基于Apache Kafka 2.0.1的。