使用kubeadm安装Kubernetes 1.7

2017-07-04 2017-08-03 阅读: Kubernetes

kubeadm是Kubernetes官方提供的快速安装和初始化Kubernetes集群的工具,目前的还处于孵化开发状态,伴随Kubernetes每个版本的发布都会同步更新。 当然,目前的kubeadm是不能用于生产环境的。 但伴随着Kubernetes每次版本升级,kubeadm都会对集群配置方面的一些实践做调整,通过实验kubeadm我们可以学习到Kubernetes官方在集群配置上一些新的最佳实践。 例如从Kubernetes 1.6开始kubeadm的目标就是可以快速初始化一个更加安全的Kubernetes集群,Kubernetes 1.6进入beat的RBAC,在这版的kubeadm就有了集成。 目前,我们一套Kubernetes 1.6环境的部署,就是先根据kubeadm中的配置实践整理,再编写成ansible role部署的:

下面回到正题中来,Kubernetes 1.7已经发布了,为了跟上官方更新的脚步,接下来体验一下Kubernetes 1.7中的kubeadm。

0.更新

  • 2017/07/04 首次使用kubeadm 1.7.0遇到问题,向集群中添加新的Node时报Token不合法的问题(issue 335PR 48480),按照issue 335中描述的通过手动创建role和rolebinding解决。
  • 2017/07/15 重新使用kubeadm 1.7.1体验安装流程,向集群中添加新的Node时报[preflight] WARNING: hostname "" could not be reached,无法完成安装。 stackoverflow上有人遇到了相同的问题
  • 2017/07/25 kubernetes 1.7.2已经发布了,这回体验kubeadm 1.7.2的安装流程,以上问题依旧,无法完成安装。 github上有这个issuekubeadm cannot work with machines that cannot resolve their own host。看来需要继续等kubeadm 1.7.3了。
  • 2017/08/03 kubernetes 1.7.3已经发布,这回体验kubeadm 1.7.3的安装流程。

1.准备

1.1系统配置

在安装之前,需要先做如下准备。两台CentOS 7主机如下:

cat /etc/hosts
192.168.61.91 k8s-node1
192.168.61.92 k8s-node2

如果各个主机启用了防火墙,需要开放Kubernetes各个组件所需要的端口,可以查看Installing kubeadm中的”Check required ports”一节。 这里简单起见在各节点禁用防火墙:

systemctl stop firewalld
systemctl disable firewalld

创建/etc/sysctl.d/k8s.conf文件,添加如下内容:

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

禁用SELINUX:

setenforce 0
vi /etc/selinux/config
SELINUX=disabled

1.2安装Docker

Kubernetes官方还没有针对docker 1.13和docker 17.03以上版本做测试和验证,但由于我们线上环境的Kubernetes 1.6集群中已经使用了docker 17.03,所以这里决定体验一下Docker的最新版本。 接下来在各节点安装最新版本的Docker。

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

查看最新的Docker版本:

yum list docker-ce.x86_64  --showduplicates |sort -r
docker-ce.x86_64            17.06.0.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.03.2.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.03.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.03.0.ce-1.el7.centos             docker-ce-stable

可以看出最新版本已经是docker 17.06了,安装这个版本:

yum makecache fast
yum install docker-ce-17.06.0.ce-1.el7.centos

systemctl start docker
systemctl enable docker

Docker从1.13版本开始调整了默认的防火墙规则,禁用了iptables filter表中FOWARD链,这样会引起Kubernetes集群中跨Node的Pod无法通信,在各个Docker节点执行下面的命令:

iptables -P FORWARD ACCEPT

2.安装kubeadm和kubelet

下面在各节点安装kubeadm和kubelet:

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

测试地址https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64是否可用,如果不可用需要科学上网。

curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

注意kubernetes yum repo的地址对比使用kubeadm安装Kubernetes 1.6时发生了变化

查看kubeadm, kubelet, kubectl, kubernets-cni的最新版本:

yum list kubeadm  --showduplicates |sort -r
kubeadm.x86_64                        1.7.3-1                        kubernetes
kubeadm.x86_64                        1.7.2-0                        kubernetes
kubeadm.x86_64                        1.7.2-0                        @kubernetes
kubeadm.x86_64                        1.7.1-0                        kubernetes
kubeadm.x86_64                        1.7.0-0                        kubernetes
kubeadm.x86_64                        1.6.7-0                        kubernetes
...
 

yum list kubelet  --showduplicates |sort -r
kubelet.x86_64                        1.7.3-1                        kubernetes
kubelet.x86_64                        1.7.2-0                        kubernetes
kubelet.x86_64                        1.7.2-0                        @kubernetes
kubelet.x86_64                        1.7.1-0                        kubernetes
kubelet.x86_64                        1.7.0-0                        kubernetes
kubelet.x86_64                        1.6.7-0                        kubernetes
...


yum list kubectl  --showduplicates |sort -r
kubectl.x86_64                        1.7.3-1                        kubernetes
kubectl.x86_64                        1.7.2-0                        kubernetes
kubectl.x86_64                        1.7.2-0                        @kubernetes
kubectl.x86_64                        1.7.1-0                        kubernetes
kubectl.x86_64                        1.7.0-0                        kubernetes
kubectl.x86_64                        1.6.7-0                        kubernetes
...

kubeadm, kubelet, kubectl已经是1.7.3,就是我们要安装的版本,直接安装即可:

yum makecache fast
yum install -y kubelet kubeadm kubectl

... 
Installed:
  kubeadm.x86_64 0:1.7.3-1            kubectl.x86_64 0:1.7.3-1            kubelet.x86_64 0:1.7.3-1

Dependency Installed:
  kubernetes-cni.x86_64 0:0.5.1-0                             socat.x86_64 0:1.7.2.2-5.el7
  • 从安装结果可以看出还安装了kubernetes-cni和socat两个依赖:

在各节点开机启动kubelet服务:

systemctl enable kubelet.service

3.使用kubeadm init初始化集群

接下来使用kubeadm初始化集群,选择k8s-node1作为Master Node,在k8s-node1上执行下面的命令:

kubeadm init \
  --kubernetes-version=v1.7.3 \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=192.168.61.91

因为我们选择flannel作为Pod网络插件,所以上面的命令指定–pod-network-cidr=10.244.0.0/16。

第一次运行上面的init命令输出卡在了:

...
[apiclient] Created API client, waiting for the control plane to become ready

通过查看/var/log/message发现如下日志:

error: failed to run Kubelet: failed to create kubelet: 
misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

使用docker info打印docker信息:

docker info
......
Server Version: 17.06.0-ce
......
Cgroup Driver: cgroupfs

可以看出docker 17.06使用的Cgroup Driver为cgroupfs。 查看kubelet的 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf文件,其中包含如下内容:

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

查看了一下kubelet的启动参数

  --cgroup-driver string Driver that the kubelet uses to manipulate cgroups on the host.
		Possible values: 'cgroupfs', 'systemd' (default "cgroupfs")
 默认值为cgroupfs,但是我们注意到yum安装kubelet,kubeadm时生成10-kubeadm.conf文件中将这个参数值改成了systemd。

于是修改各节点docker的cgroup driver使其和kubelet一致,即修改或创建/etc/docker/daemon.json,加入下面的内容:

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

重启docker:

systemctl restart docker
systemctl status docker

前面集群初始化遇到问题,使用下面的命令进行清理:

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

清理完成后,再重新初始化:

kubeadm init \
>   --kubernetes-version=v1.7.3 \
>   --pod-network-cidr=10.244.0.0/16 \
>   --apiserver-advertise-address=192.168.61.91
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.3
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 1.12
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [k8s-node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.91]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 490.016471 seconds
[token] Using token: 78bf97.8b83ab4d9933076f
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 78bf97.8b83ab4d9933076f 192.168.61.91:6443

这次初始化成功了,上面记录了完成的初始化输出的内容。

其中由以下关键内容:

  • kubeadm 1.7当前还处于beta状态,还不能用于生产环境。目前来看这东西安装的etcd和apiserver都是单节点,当然不能用于生产环境。
  • kubeadm 1.7仍然启用了集群安全的RBAC机制
  • 输出内容中还给我们报了Docker版本过高的警告,当然这里只是体验。Kubernetes官方还没有针对docker 1.13和docker 17.03以上版本做测试和验证,所以还是应该官方推荐的Docker 1.12版本。ps:我们目前有一套环境使用的docker 17.03。
  • 接下来是生成证书和相关的kubeconfig文件,这个目前我们在Kubernetes 1.6 高可用集群部署也是这么做的,目前没看出有什么新东西
  • 生成token记录下来,后边使用kubeadm join往集群中添加节点时会用到
  • 另外注意kubeadm还报了starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)的警告
  • 下面的命令是配置常规用户如何使用kubectl访问集群:
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
  • 最后给出了将节点加入集群的命令kubeadm join --token d3c777.2556db2a398bc913 192.168.61.91:6443

查看一下集群状态:

kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}

确认个组件都处于healthy状态。

4.安装Pod Network

接下来安装flannel network add-on:

mkdir -p ~/k8s/
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl create -f kube-flannel-rbac.yml
kubectl apply -f  kube-flannel.yml
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created

这里注意flannel 0.8.0已经发布,kube-flannel.yml这个文件里的flannel的镜像是0.8.0,quay.io/coreos/flannel:v0.8.0-amd64

如果Node有多个网卡的话,参考flannel issues 39701,目前需要在kube-flannel.yml中使用--iface参数指定集群主机内网网卡的名称,否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地,flanneld启动参数加上--iface=<iface-name>

......
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
......
containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.8.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth1" ]
......

使用kubectl get pod --all-namespaces -o wide确保所有的Pod都处于Running状态。

kubectl get pod --all-namespaces -o wide

5.master node参与工作负载

使用kubeadm初始化的集群,出于安全考虑Pod不会被调度到Master Node上,也就是说Master Node不参与工作负载。

这里搭建的是测试环境可以使用下面的命令使Master Node参与工作负载:

kubectl taint nodes k8s-node1 node-role.kubernetes.io/master-
node "k8s-node1" untainted

6.测试DNS

kubectl run curl --image=radial/busyboxplus:curl -i --tty
If you don't see a command prompt, try pressing enter.
[ root@curl-2716574283-xr8zd:/ ]$

进入后执行nslookup kubernetes.default确认解析正常:

nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

7.向Kubernetes集群添加Node

下面我们将k8s-node2这个主机添加到Kubernetes集群中,在k8s-node2上执行:

kubeadm join --token 78bf97.8b83ab4d9933076f 192.168.61.91:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 1.12
[discovery] Trying to connect to API Server "192.168.61.91:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.61.91:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://192.168.61.91:6443"
[discovery] Successfully established connection with API Server "192.168.61.91:6443"
[bootstrap] Detected server version: v1.7.3
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

这次很是顺利,下面在master节点上执行命令查看集群中的节点:

kubectl get nodes
NAME        STATUS    AGE       VERSION
k8s-node1   Ready     16m       v1.7.3
k8s-node2   Ready     5m        v1.7.3

如何从集群中移除Node

如果需要从集群中移除k8s-node2这个Node执行下面的命令:

在master节点上执行:

kubectl drain k8s-node2 --delete-local-data --force --ignore-daemonsets
kubectl delete node k8s-node2

在k8s-node2上执行:

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

8.dashboard插件部署

mkdir -p ~/k8s/
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard.yaml

注意当前dashboard的版本已经是1.6.3了

9.heapster插件部署

下面安装Heapster为集群添加使用统计和监控功能,为Dashboard添加仪表盘。 使用InfluxDB做为Heapster的后端存储,开始部署:

mkdir -p ~/k8s/heapster
cd ~/k8s/heapster
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml

kubectl create -f ./

最后确认所有的pod都处于running状态,打开Dashboard,集群的使用统计会以仪表盘的形式显示出来。

本次安装涉及到的Docker镜像:

gcr.io/google_containers/kube-apiserver-amd64:v1.7.3
gcr.io/google_containers/kube-controller-manager-amd64:v1.7.3
gcr.io/google_containers/kube-scheduler-amd64:v1.7.3
gcr.io/google_containers/kube-proxy-amd64:v1.7.3
quay.io/coreos/flannel:v0.8.0-amd64
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4
gcr.io/google_containers/etcd-amd64:3.0.17
gcr.io/google_containers/pause-amd64:3.0
gcr.io/google_containers/kube-proxy-amd64:v1.7.3
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3
gcr.io/google_containers/heapster-amd64:v1.3.0
gcr.io/google_containers/heapster-grafana-amd64:v4.4.1
gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1

参考

标题:使用kubeadm安装Kubernetes 1.7
本文链接:https://blog.frognew.com/2017/07/kubeadm-install-kubernetes-1.7.html
转载请注明出处。

目录