Kubernetes集群安全：Api Server认证
2017-01-19
Kubernetes提供了三种级别的客户端认证方式：
- HTTPS证书认证，是基于CA根证书签名的双向数字证书认证方式，是最严格的认证
- HTTP Token认证，通过Token识别每个合法的用户
- HTTP Basic认证
HTTP Token认证和Http Basic认证是相对简单的认证方式，Kubernetes的各组件与Api Server的通信方式仍然是HTTPS，但不再使用CA数字证书。
基于CA证书的双向认证方式 #
kube-apiserver证书配置 #
使用kubeadm初始化的Kubernetes集群中，kube-apiserver是以静态Pod的形式运行在Master Node上。
可以在Master Node上找到其定义文件/etc/kubernetes/manifests/kube-apiserver.json，其中启动命令参数部分如下：
1 "containers": [
2 {
3 "name": "kube-apiserver",
4 "image": "gcr.io/google_containers/kube-apiserver-amd64:v1.5.2",
5 "command": [
6 "kube-apiserver",
7 "--insecure-bind-address=127.0.0.1",
8 "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",
9 "--service-cluster-ip-range=10.96.0.0/12",
10 "--service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem",
11 "--client-ca-file=/etc/kubernetes/pki/ca.pem",
12 "--tls-cert-file=/etc/kubernetes/pki/apiserver.pem",
13 "--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
14 "--token-auth-file=/etc/kubernetes/pki/tokens.csv",
15 "--secure-port=6443",
16 "--allow-privileged",
17 "--advertise-address=192.168.61.100",
18 "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
19 "--anonymous-auth=false",
20 "--etcd-servers=http://127.0.0.1:2379"
21 ],
我们注意到有如下三个启动参数：
--client-ca-file: 指定CA根证书文件为/etc/kubernetes/pki/ca.pem，内置CA公钥用于验证某证书是否是CA签发的证书--tls-private-key-file: 指定ApiServer私钥文件为/etc/kubernetes/pki/apiserver-key.pem--tls-cert-file：指定ApiServer证书文件为/etc/kubernetes/pki/apiserver.pem
说明Api Server已经启动了HTTPS证书认证，此时如果在集群外部使用浏览器访问https://:6443/api会提示Unauthorized。
1curl -k https://192.168.61.100:6443/api
2Unauthorized
在Master Node上进入/etc/kubernetes/pki/目录：
1cd /etc/kubernetes/pki/
2ls
3apiserver-key.pem apiserver-pub.pem ca.pem sa-key.pem tokens.csv
4apiserver.pem ca-key.pem ca-pub.pem sa-pub.pem
查看CA根证书ca.pem：
1openssl x509 -noout -text -in ca.pem
2Certificate:
3 Data:
4 Version: 3 (0x2)
5 Serial Number: 0 (0x0)
6 Signature Algorithm: sha256WithRSAEncryption
7 Issuer: CN=kubernetes
8 Validity
9 Not Before: Jan 12 04:52:08 2017 GMT
10 Not After : Jan 12 04:52:08 2027 GMT
11 Subject: CN=kubernetes
12 Subject Public Key Info:
13
查看ApiServer的证书apiserver.pem：
1openssl x509 -noout -text -in apiserver.pem
2Certificate:
3 Data:
4 Version: 3 (0x2)
5 Serial Number: 5846608255968714821 (0x5123533b6de1a045)
6 Signature Algorithm: sha256WithRSAEncryption
7 Issuer: CN=kubernetes
8 Validity
9 Not Before: Jan 12 04:52:08 2017 GMT
10 Not After : Jan 12 04:52:08 2018 GMT
11 Subject: CN=kube-apiserver
12 Subject Public Key Info:
13 Public Key Algorithm: rsaEncryption
14......
15 X509v3 extensions:
16 X509v3 Key Usage: critical
17 Digital Signature, Key Encipherment
18 X509v3 Extended Key Usage:
19 TLS Web Server Authentication, TLS Web Client Authentication
20 X509v3 Subject Alternative Name:
21 DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:192.168.61.100
22 Signature Algorithm: sha256WithRSAEncryption
23......
验证apiserver.pem由ca.pem签发：
1openssl verify -CAfile ca.pem apiserver.pem
2apiserver.pem: OK
生成客户端私钥和证书 #
客户端要通过HTTPS证书双向认证的形式访问Api Server需要生成客户端的私钥和证书，其中客户端证书的在生成时-CA参数要指定为Api Server的CA根证书文件/etc/kubernetes/pki/ca.pem，-CAkey参数要指定为Api Server的CA key /etc/kubernetes/pki/ca-key.pem。
具体操作可参考Creating Certificates
下面生成客户端私钥和证书：
1cd /etc/kubernetes/pki/
2openssl genrsa -out client.key 2048
3openssl req -new -key client.key -subj "/CN=192.168.61.100" -out client.csr
4openssl x509 -req -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client.crt -days 3650
其中/CN设置客户端所在Node的IP地址
查看生成的证书：
1openssl x509 -noout -text -in client.crt
2
3openssl verify -CAfile ca.pem client.crt
4client.crt: OK
kubectl使用生成的客户端私钥和证书访问ApiServer:
1cd /etc/kubernetes/pki/
2
3
4kubectl --server=https://192.168.61.100:6443 \
5--certificate-authority=ca.pem \
6--client-certificate=client.crt \
7--client-key=client.key \
8get nodes
9
10NAME STATUS AGE
11cent0 Ready,master 7d
12cent1 Ready 7d
13cent2 Ready 7d
master node核心组件与ApiServer的认证方式 #
接下来我们来看一下master node上其他核心组件与ApiServer通信的认证方式。
/etc/kubernetes/manifests下的kube-controller-manager.json和kube-scheduler.json说明Controller Manager和Scheduler都是以静态Pod的形式运行在Master Node上，注意到这两个文件里的启动参数--master=127.0.0.1:8080，说明它们直接通过insecure-port 8080和ApiServer通信。
而前面ApiServer的--insecure-bind-address=127.0.0.1，因此他们之间无需走secure-port。
HTTP Token认证 #
我们继续注意一下Master Node下的/etc/kubernetes/manifests/kube-apiserver.json文件：
1 "command": [
2 "kube-apiserver",
3 "--insecure-bind-address=127.0.0.1",
4 "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",
5 "--service-cluster-ip-range=10.96.0.0/12",
6 "--service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem",
7 "--client-ca-file=/etc/kubernetes/pki/ca.pem",
8 "--tls-cert-file=/etc/kubernetes/pki/apiserver.pem",
9 "--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
10 "--token-auth-file=/etc/kubernetes/pki/tokens.csv",
11 "--secure-port=6443",
12 "--allow-privileged",
13 "--advertise-address=192.168.61.100",
14 "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
15 "--anonymous-auth=false",
16 "--etcd-servers=http://127.0.0.1:2379"
17 ],
--token-auth-file=/etc/kubernetes/pki/tokens.csv指定了静态Token文件，说明已经开启了Http Token认证。
这个文件的格式是token,user,uid,"group1,group2,group3"
1cat /etc/kubernetes/pki/tokens.csv
2792c62a1b5f2b07b,kubeadm-node-csr,ab47c6cb-f403-11e6-95a3-0800279704c8,system:kubelet-bootstrap
请求Api时只要在Authorization头中加入Bearer Token即可：
1curl -k --header "Authorization: Bearer 792c62a1b5f2b07b" https://192.168.61.100:6443/api
2{
3 "kind": "APIVersions",
4 "versions": [
5 "v1"
6 ],
7 "serverAddressByClientCIDRs": [
8 {
9 "clientCIDR": "0.0.0.0/0",
10 "serverAddress": "192.168.61.100:6443"
11 }
12 ]
13}
kubectl使用Bearer访问Api Server:
1kubectl --server=https://192.168.61.100:6443 \
2--token=792c62a1b5f2b07b \
3--insecure-skip-tls-verify=true \
4cluster-info
Http Basic认证 #
kubeadm初始化的集群没有开启Http Basic认证。实践中不建议使用，这里简单体验一下。
在Master Node上创建/etc/kubernetes/baisc_auth文件，文件中每行的格式为password,user,uid,"group1,group2,group3"。
这里简单写入如下内容：
11234,admin,1
/etc/kubernetes/manifests/kube-apiserver.json文件Container的command中加入：
1--basic_auth_file=/etc/kubernetes/basic_auth
因为静态Pod直接被kubelet管理，所以我们需要重启一下kubelet使上面的配置生效：
1systemctl restart kubelet
请求时使用请求头Authorization Basic BASE64ENCODED(USER:PASSWORD)：
1curl -k --header "Authorization:Basic YWRtaW46MTIzNA==" https://192.168.61.100:6443/api
2{
3 "kind": "APIVersions",
4 "versions": [
5 "v1"
6 ],
7 "serverAddressByClientCIDRs": [
8 {
9 "clientCIDR": "0.0.0.0/0",
10 "serverAddress": "192.168.61.100:6443"
11 }
12 ]
13}
1kubectl --server=https://192.168.61.100:6443 \
2--username=admin \
3--password=1234 \
4--insecure-skip-tls-verify=true \
5cluster-info