1.配置helm chart repo

kafka的helm chart还在孵化当中,使用前需要添加incubator的repo:helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

肉身在国内需要设置azure提供的镜像库地址:

1helm repo add stable http://mirror.azure.cn/kubernetes/charts
2helm repo add incubator http://mirror.azure.cn/kubernetes/charts-incubator
3
4helm repo list
5NAME     	URL                                               
6stable   	http://mirror.azure.cn/kubernetes/charts          
7local    	http://127.0.0.1:8879/charts                      
8incubator	http://mirror.azure.cn/kubernetes/charts-incubator

2.创建Kafka和Zookeeper的Local PV

2.1 创建Kafka的Local PV

这里的部署环境是本地的测试环境,存储选择Local Persistence Volumes。首先,在k8s集群上创建本地存储的StorageClass local-storage.yaml

1apiVersion: storage.k8s.io/v1
2kind: StorageClass
3metadata:
4  name: local-storage
5provisioner: kubernetes.io/no-provisioner
6volumeBindingMode: WaitForFirstConsumer
7reclaimPolicy: Retain
1kubectl apply -f local-storage.yaml 
2storageclass.storage.k8s.io/local-storage created

这里要在node1、node2这两个k8s节点上部署3个kafka的broker节点,因此先在node1、node2上创建这3个kafka broker节点的Local PV kafka-local-pv.yaml:

 1apiVersion: v1
 2kind: PersistentVolume
 3metadata:
 4  name: datadir-kafka-0
 5spec:
 6  capacity:
 7    storage: 5Gi 
 8  accessModes:
 9  - ReadWriteOnce
10  persistentVolumeReclaimPolicy: Retain
11  storageClassName: local-storage
12  local:
13    path: /home/kafka/data-0
14  nodeAffinity:
15    required:
16      nodeSelectorTerms:
17      - matchExpressions:
18        - key: kubernetes.io/hostname
19          operator: In
20          values:
21          - node1
22---
23apiVersion: v1
24kind: PersistentVolume
25metadata:
26  name: datadir-kafka-1
27spec:
28  capacity:
29    storage: 5Gi 
30  accessModes:
31  - ReadWriteOnce
32  persistentVolumeReclaimPolicy: Retain
33  storageClassName: local-storage
34  local:
35    path: /home/kafka/data-1
36  nodeAffinity:
37    required:
38      nodeSelectorTerms:
39      - matchExpressions:
40        - key: kubernetes.io/hostname
41          operator: In
42          values:
43          - node2
44---
45apiVersion: v1
46kind: PersistentVolume
47metadata:
48  name: datadir-kafka-2
49spec:
50  capacity:
51    storage: 5Gi 
52  accessModes:
53  - ReadWriteOnce
54  persistentVolumeReclaimPolicy: Retain
55  storageClassName: local-storage
56  local:
57    path: /home/kafka/data-2
58  nodeAffinity:
59    required:
60      nodeSelectorTerms:
61      - matchExpressions:
62        - key: kubernetes.io/hostname
63          operator: In
64          values:
65          - node2
1kubectl apply -f kafka-local-pv.yaml

根据上面创建的local pv,在node1上创建目录/home/kafka/data-0,在node2上创建目录/home/kafka/data-1/home/kafka/data-2

1# node1
2mkdir -p /home/kafka/data-0
3
4# node2
5mkdir -p /home/kafka/data-1
6mkdir -p /home/kafka/data-2

2.2 创建Zookeeper的Local PV

这里要在node1、node2这两个k8s节点上部署3个zookeeper节点,因此先在node1、node2上创建这3个zookeeper节点的Local PV zookeeper-local-pv.yaml:

 1apiVersion: v1
 2kind: PersistentVolume
 3metadata:
 4  name: data-kafka-zookeeper-0
 5spec:
 6  capacity:
 7    storage: 5Gi 
 8  accessModes:
 9  - ReadWriteOnce
10  persistentVolumeReclaimPolicy: Retain
11  storageClassName: local-storage
12  local:
13    path: /home/kafka/zkdata-0
14  nodeAffinity:
15    required:
16      nodeSelectorTerms:
17      - matchExpressions:
18        - key: kubernetes.io/hostname
19          operator: In
20          values:
21          - node1
22---
23apiVersion: v1
24kind: PersistentVolume
25metadata:
26  name: data-kafka-zookeeper-1
27spec:
28  capacity:
29    storage: 5Gi 
30  accessModes:
31  - ReadWriteOnce
32  persistentVolumeReclaimPolicy: Retain
33  storageClassName: local-storage
34  local:
35    path: /home/kafka/zkdata-1
36  nodeAffinity:
37    required:
38      nodeSelectorTerms:
39      - matchExpressions:
40        - key: kubernetes.io/hostname
41          operator: In
42          values:
43          - node2
44---
45apiVersion: v1
46kind: PersistentVolume
47metadata:
48  name: data-kafka-zookeeper-2
49spec:
50  capacity:
51    storage: 5Gi 
52  accessModes:
53  - ReadWriteOnce
54  persistentVolumeReclaimPolicy: Retain
55  storageClassName: local-storage
56  local:
57    path: /home/kafka/zkdata-2
58  nodeAffinity:
59    required:
60      nodeSelectorTerms:
61      - matchExpressions:
62        - key: kubernetes.io/hostname
63          operator: In
64          values:
65          - node2
1kubectl apply -f zookeeper-local-pv.yaml

根据上面创建的local pv,在node1上创建目录/home/kafka/zkdata-0,在node2上创建目录/home/kafka/zkdata-1/home/kafka/zkdata-2

1# node1
2mkdir -p /home/kafka/zkdata-0
3
4# node2
5mkdir -p /home/kafka/zkdata-1
6mkdir -p /home/kafka/zkdata-2

3.部署Kafka

编写kafka chart的vaule文件kafka-values.yaml:

 1replicas: 3
 2tolerations:
 3- key: node-role.kubernetes.io/master
 4  operator: Exists
 5  effect: NoSchedule
 6- key: node-role.kubernetes.io/master
 7  operator: Exists
 8  effect: PreferNoSchedule
 9persistence:
10  storageClass: local-storage
11  size: 5Gi
12zookeeper:
13  persistence:
14    enabled: true
15    storageClass: local-storage
16    size: 5Gi
17  replicaCount: 3
18  image:
19    repository: gcr.azk8s.cn/google_samples/k8szk
20  tolerations:
21  - key: node-role.kubernetes.io/master
22    operator: Exists
23    effect: NoSchedule
24  - key: node-role.kubernetes.io/master
25    operator: Exists
26    effect: PreferNoSchedule
  • 安装过程需要使用到gcr.io/google_samples/k8szk:v3等docker镜像,切换成使用azure的GCR Proxy Cache:gcr.azk8s.cn
1helm install --name kafka --namespace kafka -f kafka-values.yaml incubator/kafka 

最后需要确认所有的pod都处于running状态:

 1kubectl get pod -n kafka -o wide
 2NAME                READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
 3kafka-0             1/1     Running   0          12m     10.244.0.61   node1   <none>           <none>
 4kafka-1             1/1     Running   0          6m3s    10.244.1.12   node2   <none>           <none>
 5kafka-2             1/1     Running   0          2m26s   10.244.1.13   node2   <none>           <none>
 6kafka-zookeeper-0   1/1     Running   0          12m     10.244.1.9    node2   <none>           <none>
 7kafka-zookeeper-1   1/1     Running   0          11m     10.244.1.10   node2   <none>           <none>
 8kafka-zookeeper-2   1/1     Running   0          11m     10.244.1.11   node2   <none>           <none>
 9
10kubectl get statefulset -n kafka
11NAME              READY   AGE
12kafka             3/3     22m
13kafka-zookeeper   3/3     22m
14
15kubectl get service -n kafka
16NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
17kafka                      ClusterIP   10.102.8.192    <none>        9092/TCP                     31m
18kafka-headless             ClusterIP   None            <none>        9092/TCP                     31m
19kafka-zookeeper            ClusterIP   10.110.43.203   <none>        2181/TCP                     31m
20kafka-zookeeper-headless   ClusterIP   None            <none>        2181/TCP,3888/TCP,2888/TCP   31m

可以看到当前kafka的helm chart,采用StatefulSet的形式部署了kafka和zookeeper,而我们通过Local PV的形式,将kafka-0调度到node1上,将kafka-1kafka-2调度到node2上。

4.安装后的测试

在k8s集群内运行下面的客户端Pod,访问kafka broker进行测试:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: testclient
 5  namespace: kafka
 6spec:
 7  containers:
 8  - name: kafka
 9    image: confluentinc/cp-kafka:5.0.1
10    command:
11    - sh
12    - -c
13    - "exec tail -f /dev/null"

创建并进入testclient容器内:

1kubectl apply -f testclient.yaml
2kubectl -n kafka exec testclient -it sh

查看kafka相关命令:

 1ls /usr/bin/ | grep kafka
 2kafka-acls
 3kafka-broker-api-versions
 4kafka-configs
 5kafka-console-consumer
 6kafka-console-producer
 7kafka-consumer-groups
 8kafka-consumer-perf-test
 9kafka-delegation-tokens
10kafka-delete-records
11kafka-dump-log
12kafka-log-dirs
13kafka-mirror-maker
14kafka-preferred-replica-election
15kafka-producer-perf-test
16kafka-reassign-partitions
17kafka-replica-verification
18kafka-run-class
19kafka-server-start
20kafka-server-stop
21kafka-streams-application-reset
22kafka-topics
23kafka-verifiable-consumer
24kafka-verifiable-producer

创建一个Topic test1:

1kafka-topics --zookeeper kafka-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1

查看的Topic:

1kafka-topics --zookeeper kafka-zookeeper:2181 --list
2test1

5.总结

当前基于Helm官方仓库的chartincubator/kafka在k8s上部署的kafka,使用的镜像是confluentinc/cp-kafka:5.0.1。 即部署的是Confluent公司提供的kafka版本。Confluent Platform Kafka(简称CP Kafka)提供了一些Apache Kafka没有的高级特性,例如跨数据中心备份、Schema注册中心以及集群监控工具等。CP Kafka目前分为免费版本和企业版两种,免费版除了Apache Kafka的标准组件外还包含Schema注册中心和Rest Proxy。

Confluent Platform and Apache Kafka Compatibility中给出了Confluent Kafka和Apache Kafka的版本对应关系,可以看出这里安装的cp 5.0.1对应Apache Kafka的2.0.x。

进入一个broker容器中,查看:

 1ls /usr/share/java/kafka | grep kafka
 2kafka-clients-2.0.1-cp1.jar
 3kafka-log4j-appender-2.0.1-cp1.jar
 4kafka-streams-2.0.1-cp1.jar
 5kafka-streams-examples-2.0.1-cp1.jar
 6kafka-streams-scala_2.11-2.0.1-cp1.jar
 7kafka-streams-test-utils-2.0.1-cp1.jar
 8kafka-tools-2.0.1-cp1.jar
 9kafka.jar
10kafka_2.11-2.0.1-cp1-javadoc.jar
11kafka_2.11-2.0.1-cp1-scaladoc.jar
12kafka_2.11-2.0.1-cp1-sources.jar
13kafka_2.11-2.0.1-cp1-test-sources.jar
14kafka_2.11-2.0.1-cp1-test.jar
15kafka_2.11-2.0.1-cp1.jar

可以看到对应apache kafka的版本号是2.11-2.0.1,前面2.11是Scala编译器的版本,Kafka的服务器端代码是使用Scala语言开发的,后边2.0.1是Kafka的版本。 即CP Kafka 5.0.1是基于Apache Kafka 2.0.1的。

参考