使用Ansible部署Kubernetes 1.6高可用集群

我们已经用ansible在新的环境中部署了etcd和docker，接下来使用ansible部署Kubernetes 1.6集群。由于对ansible的使用已经没有什么问题了，因此本篇记录的侧重点是白话描述一下部署的具体步骤，以及部署过程踩的一些坑。因为前段日子写过一篇《Kubernetes 1.6 高可用集群部署》，本次在编写ansible部署Kubernetes的roles时主要是一这篇文章里手动部署的过程为参考。

环境准备

系统配置

Kubernetes集群各节点禁用SELINUX，同时在各节点创建/etc/sysctl.d/k8s.conf文件添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

{% raw %}
- name: copy sysctl k8s.conf
  copy:
    src: k8s.conf
    dest: /etc/sysctl.d

- name: sysctl k8s.conf
  sysctl: 
    name: "{{ item }}"
    value: 1
    sysctl_file: /etc/sysctl.d/k8s.conf
  with_items:
    - net.bridge.bridge-nf-call-iptables
    - net.bridge.bridge-nf-call-ip6tables

- name: config disable selinux
  lineinfile: 
    path: /etc/selinux/config
    regexp: '^SELINUX='
    line: 'SELINUX=enforcing'
    
- name: diable selinux
  selinux: 
    state: disabled
{% endraw %}

部署etcd高可用集群

前面准备好了，直接复用《使用Ansible部署etcd 3.2高可用集群》。

各节点安装Docker

主要坑在了这块，之前的手动部署的那套环境(Kubernetes 1.6 高可用集群部署)按照官方文档的建议按照的是Docker 1.12：

Kubernetes 1.6还没有针对docker 1.13和最新的docker 17.03上做测试和验证，Kubernetes官方推荐的Docker 1.12版本

而这次部署这套新的环境前面已经使用Ansible安装Docker CE 17.03，按照的是最新的Docker CE 17.03。结果写完整套部署Kubernetes的ansible role和playbook后，使用ansilbe-playbook运行，整个安装过程一气呵成，但在测试时却出现了问题，不同Node上的Pod之间无法通信，网络不通。

最后在这两个issue中找到了问题的原因：

Docker从1.13版本开始调整了默认的防火墙规则，禁用了iptables filter表中FOWARD链，这样就引起了Kubernetes集群中跨Node的Pod无法通信。解决的方法有以下两种：

在所有节点上执行sudo iptables -P FORWARD ACCEPT
通过修改dockerd选项配置文件/etc/docker/daemon.json中的--iptables-false

这里使用的是第一种方法，在部署Kubernetes Node的roles中加入了下面的task：

{% raw %}
- name: config filter FORWARD chain for pod networks
  iptables: 
    table: filter
    chain: FORWARD
    policy: ACCEPT
{% endraw %}

用户,证书和二进制包安装

之前的手动部署的那套环境(Kubernetes 1.6 高可用集群部署)中是以root用户运行Kubernetes各个核心组件的，现在搭建的这套环境相对比较正式，因此需要创建运行这些核心的组件的系统用户和用户组。同时创建Kubernetes的配置文件目录和日志目录。

下一步生成和分发集群所需的SSL证书和秘钥到各个节点：

kube_ca_cert_file：CA证书
kube_ca_key_file：CA私钥
kube_apiserver_cert_file：APIServer的证书
kube_apiserver_key_file：APIServer的私钥
kube_admin_cert_file：kubernetes-admin客户端用户的证书
kube_admin_key_file：kubernetes-admin客户端用户的私钥
kube_controller_manager_cert_file：controller manager客户端的证书
kube_controller_manager_key_file：controller mangager客户端的私钥
kube_scheduler_cert_file：scheduler客户端的证书
kube_scheduler_key_file：scheduler客户端的私钥

{% raw %}
- name: gen certs on the first master server
  command:
    "{{ kube_cert_dir }}/make-ca-cert.sh"
  args:
    creates: "{{ kube_cert_dir }}/ca.key"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  environment:
    NODE_IPS: "{% for host in groups['k8s-master'] %}{{ hostvars[host]['k8s_master_address'] }}{% if not loop.last %},{% endif %}{% endfor %}"
    NODE_DNS: "{{ groups['k8s-master']|join(',') }}"
    CERT_DIR: "{{ kube_cert_dir }}"
    CERT_GROUP: kube
    
- name: slurp kube certs
  slurp:
    src: "{{ item }}"
  register: pki_certs
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  with_items: 
    - "{{ kube_ca_cert_file }}"
    - "{{ kube_ca_key_file }}"
    - "{{ kube_admin_cert_file }}"
    - "{{ kube_admin_key_file }}"
    - "{{ kube_apiserver_cert_file }}"
    - "{{ kube_apiserver_key_file }}"
    - "{{ kube_controller_manager_cert_file }}"
    - "{{ kube_controller_manager_key_file }}"
    - "{{ kube_scheduler_cert_file }}"
    - "{{ kube_scheduler_key_file }}"
    
- name: copy kube certs to other node servers
  copy:
    dest: "{{ item.item }}"
    content: "{{ item.content | b64decode }}"
    owner: kube
    group: kube
    mode: 0400
  with_items: "{{ pki_certs.results }}"
  when: inventory_hostname != groups['k8s-master'][0]
{% endraw %}

接下来要在各个节点下载和安装Kubernetes各组件二进制可执行文件：

kube-apiserver
kube-controller-manager
kube-scheduler
kubelet
kube-proxy
kubectl

Master集群部署

Master集群由三个Master节点组成，每个节点上部署kube-apiserver,kube-controller-manager,kube-scheduler三个核心组件。 kube-apiserver的3个实例同时提供服务，在其前端部署一个高可用的负载均衡器作为kube-apiserver的地址。 kube-controller-manager和kube-scheduler也是各自3个实例，在同一时刻只能有1个实例工作，这个实例通过选举产生。

apiserver

apiserver的部署十分简单，因为我们的ETCD集群启用SSL，所以apiserver参数需要指定etcd客户端相关的证书。下面只列出 systemd unit模板文件：

{% raw %}
Unit]
Description=kube-apiserver
After=network.target
After=etcd.service

[Service]
User=kube
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart={{ kube_bin_dir }}/kube-apiserver \
	    --logtostderr=true \
	    --v=0 \
	    --advertise-address={{ k8s_master_address }} \
	    --bind-address={{ k8s_master_address }} \
	    --secure-port=6443 \
	    --insecure-port=0 \
	    --allow-privileged=true \
	    --etcd-servers={{ kube_etcd_servers }} \
	    --etcd-cafile={{ kube_etcd_ca_file }} \
	    --etcd-certfile={{ kube_etcd_cert_file }} \
	    --etcd-keyfile={{ kube_etcd_key_file }} \
	    --storage-backend=etcd3 \
	    --service-cluster-ip-range=10.96.0.0/12 \
	    --tls-cert-file={{ kube_apiserver_cert_file }} \
	    --tls-private-key-file={{ kube_apiserver_key_file }} \
	    --client-ca-file={{ kube_ca_cert_file }} \
	    --service-account-key-file={{ kube_ca_key_file }} \
	    --experimental-bootstrap-token-auth=true \
	    --apiserver-count=3 \
	    --enable-swagger-ui=true \
	    --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds \
	    --authorization-mode=RBAC \
	    --audit-log-maxage=30 \
	    --audit-log-maxbackup=3 \
	    --audit-log-maxsize=100 \
	    --audit-log-path={{ kube_log_dir }}/audit.log
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
{% endraw %}

controller-manager

controller-manager部署也很简单，下面只列出 systemd unit模板文件：

{% raw %}
[Unit]
Description=kube-controller-manager
After=network.target
After=kube-apiserver.service

[Service]
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart={{ kube_bin_dir }}/kube-controller-manager \
	    --logtostderr=true \
	    --v=0 \
	    --master={{ kube_apiserver_lb_address }} \
	    --kubeconfig={{ kube_controller_manager_kubeconfig_file }} \
	    --cluster-name=kubernetes \
	    --cluster-signing-cert-file={{ kube_ca_cert_file }} \
	    --cluster-signing-key-file={{ kube_ca_key_file }} \
	    --service-account-private-key-file={{ kube_ca_key_file }} \
	    --root-ca-file={{ kube_ca_cert_file }} \
	    --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers \
	    --use-service-account-credentials=true \
	    --service-cluster-ip-range=10.96.0.0/12 \
	    --cluster-cidr=10.244.0.0/16 \
	    --allocate-node-cidrs=true \
	    --leader-elect=true \
	    --controllers=*,bootstrapsigner,tokencleaner
Restart=on-failure
Type=simple
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
{% endraw %}

其中--kubeconfig使用controller manager客户端相关的证书生成。

scheduler

controller-manager部署也很简单，下面只列出 systemd unit模板文件：

{% raw %}
[Unit]
Description=kube-scheduler
After=network.target
After=kube-apiserver.service

[Service]
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart={{ kube_bin_dir }}/kube-scheduler \
	    --logtostderr=true \
	    --v=0 \
	    --master={{ kube_apiserver_lb_address }} \
	    --kubeconfig={{ kube_scheduler_kubeconfig_file }} \
	    --leader-elect=true
Restart=on-failure
Type=simple
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
{% endraw %}

其中--kubeconfig使用scheduler客户端相关的证书生成。

Node节点部署

cni

{% raw %}
---
    
- name: create cni download dir
  file: 
    path: "{{ kube_cni_download_dir }}"
    state: directory
  delegate_to: "{{ groups['k8s-node'][0] }}"
  run_once: true
  
- name: check whether cni downloaded on the first node
  stat: 
    path: "{{ kube_cni_download_dir }}/{{ kube_cni_release }}"
  register: kube_cni_downloaded_check
  delegate_to: "{{ groups['k8s-node'][0] }}"
  run_once: true
  
- name: download cni on the first node
  get_url:
    url: "{{ kube_cni_download_url }}"
    dest: "{{ kube_cni_download_dir }}"
    validate_certs: no
    timeout: 20
  register: download_cni
  delegate_to: "{{ groups['k8s-node'][0] }}"
  run_once: true
  when: not kube_cni_downloaded_check.stat.exists
  
- name: check whether cni tar extracted on the first node
  stat: 
    path: "{{ kube_cni_download_dir }}/cnitool"
  register: kube_cni_release_tar_check
  delegate_to: "{{ groups['k8s-node'][0] }}"
  run_once: true
  
- name: extract cni tar file
  unarchive:
    src: "{{ kube_cni_download_dir }}/{{ kube_cni_release }}"
    dest: "{{ kube_cni_download_dir }}"
    remote_src: yes
  run_once: true
  delegate_to: "{{ groups['k8s-node'][0] }}"
  when: not kube_cni_release_tar_check.stat.exists
  
- name: fetch cni binary from the first node
  fetch:
    src: "{{ kube_cni_download_dir }}/{{ item }}"
    dest: "tmp/k8s-cni/{{ item }}"
    flat: yes
  run_once: true
  delegate_to: "{{ groups['k8s-node'][0] }}"
  with_items:
    - bridge
    - cnitool
    - dhcp
    - flannel
    - host-local
    - ipvlan
    - loopback
    - macvlan
    - noop
    - ptp
    - tuning
  
- name: create cni bin and conf dir
  file: 
    path: "{{ item }}"
    state: directory
    owner: kube
    group: kube
    mode: 0751
    recurse: yes
  with_items:
    - "{{ kube_cni_conf_dir }}"
    - "{{ kube_cni_bin_dir }}"
    
- name: copy cni binary
  copy:
    src: "tmp/k8s-cni/{{ item }}"
    dest: "{{ kube_cni_bin_dir }}"
    owner: kube
    group: kube
    mode: 0751
  with_items:
    - bridge
    - cnitool
    - dhcp
    - flannel
    - host-local
    - ipvlan
    - loopback
    - macvlan
    - noop
    - ptp
    - tuning
{% endraw %}

kubelet

kubelet部署也很简单，下面只列出 systemd unit模板文件：

{% raw %}
[Unit]
Description=kubelet
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory={{ kube_kubelet_data_dir }}
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart={{ kube_bin_dir }}/kubelet \
        --logtostderr=true \
        --v=0 \
        --address={{ k8s_node_address }} \
        --api-servers={{ kube_apiserver_lb_address }} \
        --cluster-dns=10.96.0.10 \
        --cluster-domain=cluster.local \
        --kubeconfig={{ kube_kubelet_kubeconfig_file }} \
        --require-kubeconfig=true \
        --pod-manifest-path={{ kube_pod_manifest_dir }} \
        --allow-privileged=true \
        --authorization-mode=Webhook \
        --client-ca-file={{ kube_ca_cert_file }} \
        --network-plugin=cni \
        --cni-conf-dir={{ kube_cni_conf_dir }} \
        --cni-bin-dir={{ kube_cni_bin_dir }}
Restart=on-failure

[Install]
WantedBy=multi-user.target
{% endraw %}

其中--kubeconfig使用kubelet客户端相关的证书生成。

kube-proxy

kube-proxy部署也很简单，下面只列出 systemd unit模板文件：

{% raw %}
[Unit]
Description=kubelet
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory={{ kube_kubelet_data_dir }}
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart={{ kube_bin_dir }}/kubelet \
        --logtostderr=true \
        --v=0 \
        --address={{ k8s_node_address }} \
        --api-servers={{ kube_apiserver_lb_address }} \
        --cluster-dns=10.96.0.10 \
        --cluster-domain=cluster.local \
        --kubeconfig={{ kube_kubelet_kubeconfig_file }} \
        --require-kubeconfig=true \
        --pod-manifest-path={{ kube_pod_manifest_dir }} \
        --allow-privileged=true \
        --authorization-mode=Webhook \
        --client-ca-file={{ kube_ca_cert_file }} \
        --network-plugin=cni \
        --cni-conf-dir={{ kube_cni_conf_dir }} \
        --cni-bin-dir={{ kube_cni_bin_dir }}
Restart=on-failure

[Install]
WantedBy=multi-user.target
{% endraw %}

PodNetWork插件flannel

flannel以DaemonSet的形式运行在Kubernetes集群中。由于我们的etcd集群启用了TLS认证，为了从flannel容器中能访问etcd，我们先把etcd的TLS证书信息保存到Kubernetes的Secret中。

{% raw %}
- name: delete etcd client cert secret
  command: "{{ kube_bin_dir }}/kubectl delete secret etcd-tls-secret \
   -n kube-system"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"

- name: create etcd client cert secret
  command: "{{ kube_bin_dir }}/kubectl create secret generic etcd-tls-secret \
   --from-file={{ kube_etcd_cert_file }} \
   --from-file={{ kube_etcd_key_file }}  \
   --from-file={{ kube_etcd_ca_file }} \
   -n kube-system"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  
- name: copy kube-flannel-rbac.yml
  copy:
    src: kube-flannel-rbac.yml
    dest: "{{ ansible_temp_dir }}"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  

- name: apply kube-flannel-rbac.yml
  command: "{{ kube_bin_dir }}/kubectl apply -f {{ ansible_temp_dir }}/kube-flannel-rbac.yml"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  
- name: create kube-flannel.yml
  template: 
    src: kube-flannel.yml.j2
    dest: "{{ ansible_temp_dir }}/kube-flannel.yml"
    
- name: apply kube-flannel.yml
  command: "{{ kube_bin_dir }}/kubectl apply -f {{ ansible_temp_dir }}/kube-flannel.yml"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
{% endraw %}

注意我们已经开始使用ansible的command moudle调用kubectl来在Kubernetes集群中创建资源，而没有使用ansible的kubernetes module，这是因为我们启用了Kubernetes ApiServer的SSL双向认证， ansible的kubernetes module当前还不支持以这种方式访问ApiServer。

flannel安装完成后，需要先确认每个节点上的flannel的DaemonSet的Pod都处于Running状态。这个时候可以部署一个三个副本的nginx到集群中测试一下跨节点Pod之间的网络通信已经OK。

kubectl run nginx --replicas=3 --image=nginx  --port=80

在一个节点上curl另外两个节点上nginx的Pod IP确保通信正常。

kube-dns

kube-dns插件也是跑在Kubernetes集群上的。

{% raw %}
---

- name: copy kube dns yml
  copy:
    src: "kube-dns/{{ item }}"
    dest: "{{ ansible_temp_dir }}"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  with_items:
    - kubedns-cm.yaml
    - kubedns-sa.yaml
    - kubedns-controller.yaml
    - kubedns-svc.yaml
    
- name: apply kube dns yml
  command: "{{ kube_bin_dir }}/kubectl apply -f {{ ansible_temp_dir }}/{{ item }}"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  with_items:
    - kubedns-cm.yaml
    - kubedns-sa.yaml
    - kubedns-controller.yaml
    - kubedns-svc.yaml
    
- name: scale kube dns
  command: "{{ kube_bin_dir }}/kubectl --namespace=kube-system scale deployment kube-dns --replicas=3"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
{% endraw %}

yml文件可以参考Kubernetes 1.6 高可用集群部署中的4.5章节。

kube-dns的Pod跑起来之后需要测试一下dns是否好用：

kubectl run curl --image=radial/busyboxplus:curl -i --tty

nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

dashboard和heapster插件

{% raw %}
---

- name: copy dashboard and heapster yml
  copy:
    src: "dashboard/{{ item }}"
    dest: "{{ ansible_temp_dir }}"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  with_items:
    - kubernetes-dashboard.yaml
    - heapster-rbac.yaml
    - heapster.yaml
    - influxdb.yaml
    - grafana.yaml

- name: apply dashboard and heapster yml
  command: "{{ kube_bin_dir }}/kubectl apply -f {{ ansible_temp_dir }}/{{ item }}"
  run_once: true
  delegate_to: "{{ groups['k8s-master'][0] }}"
  with_items:
    - kubernetes-dashboard.yaml
    - heapster-rbac.yaml
    - heapster.yaml
    - influxdb.yaml
    - grafana.yaml
{% endraw %}