Kubernetes Ingress实战(一)：在Kubernetes集群中部署NGINX Ingress Controller

2018-06-06

去年整理过一篇《Kubernetes Ingress实战》，经过这一年的发展Kubernetes的Ingress发生了很大的变化，原来的文章很多地方都不适用了。因此决定结合我们目前的使用情况重新写几篇Kubernetes Ingress相关的分享，内容是比较入门和初级的实操，请高手勿喷。

Ingress是一种Kubernetes资源，借助Nginx、Haproxy或云厂商的负载均衡器将Kubernetes集群内的Service暴露到集群外。因为我们的Kubernetes集群是采以Bare-metal部署的，所以这个系列将主要介绍ingress-nginx。

下面将ingress-nginx部署到用于测试的Kubernetes集群中，这里使用的Kubernetes的版本是1.9.8。

1.创建namespace #

首先创建名称为ingress-nginx的namespace，ingress-nginx的相关组件的都将在这个namespace下。

namespace.yaml的内容如下：

1---
2
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: ingress-nginx

1kubectl create -f namespace.yaml

2.部署default backend #

下面来创建default-http-backend的Deployment和Service.

default-backend.yaml的内容如下：

 1---
 2
 3apiVersion: extensions/v1beta1
 4kind: Deployment
 5metadata:
 6  name: default-http-backend
 7  labels:
 8    app: default-http-backend
 9  namespace: ingress-nginx
10spec:
11  replicas: 1
12  selector:
13    matchLabels:
14      app: default-http-backend
15  template:
16    metadata:
17      labels:
18        app: default-http-backend
19    spec:
20      terminationGracePeriodSeconds: 60
21      containers:
22      - name: default-http-backend
23        # Any image is permissible as long as:
24        # 1. It serves a 404 page at /
25        # 2. It serves 200 on a /healthz endpoint
26        image: gcr.io/google_containers/defaultbackend:1.4
27        livenessProbe:
28          httpGet:
29            path: /healthz
30            port: 8080
31            scheme: HTTP
32          initialDelaySeconds: 30
33          timeoutSeconds: 5
34        ports:
35        - containerPort: 8080
36        resources:
37          limits:
38            cpu: 10m
39            memory: 20Mi
40          requests:
41            cpu: 10m
42            memory: 20Mi
43---
44
45apiVersion: v1
46kind: Service
47metadata:
48  name: default-http-backend
49  namespace: ingress-nginx
50  labels:
51    app: default-http-backend
52spec:
53  ports:
54  - port: 80
55    targetPort: 8080
56  selector:
57    app: default-http-backend

需要注意gcr.io/google_containers/defaultbackend:1.4这个镜像在gcr上，由于某些已知的原因，最好将此镜像放到本地Kubernetes集群的私有镜像仓库中

1kubectl apply -f default-backend.yaml
2deployment "default-http-backend" created
3service "default-http-backend" created

确认default-http-backend的Service和Pod被创建，并且Pod处于Running状态。

1kubectl get svc,deploy,pod -n ingress-nginx
2NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
3svc/default-http-backend   ClusterIP   10.111.47.113   <none>        80/TCP    1m
4
5NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
6deploy/default-http-backend   1         1         1            1           1m
7
8NAME                                       READY     STATUS    RESTARTS   AGE
9po/default-http-backend-64985b4bcb-9pn7s   1/1       Running   0          1m

default-http-backend从名称上来看即默认的后端，当集群外部的请求通过ingress进入到集群内部时，如果无法负载到相关后端的Service上，这种未知的请求将会被负载到这个默认的后端上。

3.创建ConfigMap #

接下来创建nginx-configuration、tcp-services、udp-services三个ConfigMap：

 1kubectl get deployment,pod -n ingress-nginx
 2NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
 3deploy/default-http-backend       1         1         1            1           1h
 4deploy/nginx-ingress-controller   2         2         2            2           2m
 5
 6NAME                                           READY     STATUS    RESTARTS   AGE
 7po/default-http-backend-64985b4bcb-9pn7s       1/1       Running   0          1h
 8po/nginx-ingress-controller-7f9d776bff-5k28n   1/1       Running   0          2m
 9po/nginx-ingress-controller-7f9d776bff-rwls2   1/1       Running   0          2m
10---
11
12kind: ConfigMap
13apiVersion: v1
14metadata:
15  name: nginx-configuration
16  namespace: ingress-nginx
17  labels:
18    app: ingress-nginx

1---
2
3kind: ConfigMap
4apiVersion: v1
5metadata:
6  name: tcp-services
7  namespace: ingress-nginx

1---
2
3kind: ConfigMap
4apiVersion: v1
5metadata:
6  name: udp-services
7  namespace: ingress-nginx

1kubectl apply -f configmap.yaml
2kubectl apply -f tcp-services-configmap.yaml
3kubectl apply -f udp-services-configmap.yaml
4
5kubectl get cm -n ingress-nginx
6NAME                  DATA      AGE
7nginx-configuration   0         59s
8tcp-services          0         51s
9udp-services          0         45s

4.创建ServiceAccount并进行授权 #

接下来创建ServiceAccount nginx-ingress-serviceaccount，并创建相关的ClusterRole, Role, ClusterRoleBinding, RoleBinding以对其进行授权：

  1---
  2
  3apiVersion: v1
  4kind: ServiceAccount
  5metadata:
  6  name: nginx-ingress-serviceaccount
  7  namespace: ingress-nginx
  8
  9---
 10
 11apiVersion: rbac.authorization.k8s.io/v1beta1
 12kind: ClusterRole
 13metadata:
 14  name: nginx-ingress-clusterrole
 15rules:
 16  - apiGroups:
 17      - ""
 18    resources:
 19      - configmaps
 20      - endpoints
 21      - nodes
 22      - pods
 23      - secrets
 24    verbs:
 25      - list
 26      - watch
 27  - apiGroups:
 28      - ""
 29    resources:
 30      - nodes
 31    verbs:
 32      - get
 33  - apiGroups:
 34      - ""
 35    resources:
 36      - services
 37    verbs:
 38      - get
 39      - list
 40      - watch
 41  - apiGroups:
 42      - "extensions"
 43    resources:
 44      - ingresses
 45    verbs:
 46      - get
 47      - list
 48      - watch
 49  - apiGroups:
 50      - ""
 51    resources:
 52        - events
 53    verbs:
 54        - create
 55        - patch
 56  - apiGroups:
 57      - "extensions"
 58    resources:
 59      - ingresses/status
 60    verbs:
 61      - update
 62
 63---
 64
 65apiVersion: rbac.authorization.k8s.io/v1beta1
 66kind: Role
 67metadata:
 68  name: nginx-ingress-role
 69  namespace: ingress-nginx
 70rules:
 71  - apiGroups:
 72      - ""
 73    resources:
 74      - configmaps
 75      - pods
 76      - secrets
 77      - namespaces
 78    verbs:
 79      - get
 80  - apiGroups:
 81      - ""
 82    resources:
 83      - configmaps
 84    resourceNames:
 85      # Defaults to "<election-id>-<ingress-class>"
 86      # Here: "<ingress-controller-leader>-<nginx>"
 87      # This has to be adapted if you change either parameter
 88      # when launching the nginx-ingress-controller.
 89      - "ingress-controller-leader-nginx"
 90    verbs:
 91      - get
 92      - update
 93  - apiGroups:
 94      - ""
 95    resources:
 96      - configmaps
 97    verbs:
 98      - create
 99  - apiGroups:
100      - ""
101    resources:
102      - endpoints
103    verbs:
104      - get
105
106---
107
108apiVersion: rbac.authorization.k8s.io/v1beta1
109kind: RoleBinding
110metadata:
111  name: nginx-ingress-role-nisa-binding
112  namespace: ingress-nginx
113roleRef:
114  apiGroup: rbac.authorization.k8s.io
115  kind: Role
116  name: nginx-ingress-role
117subjects:
118  - kind: ServiceAccount
119    name: nginx-ingress-serviceaccount
120    namespace: ingress-nginx
121
122---
123
124apiVersion: rbac.authorization.k8s.io/v1beta1
125kind: ClusterRoleBinding
126metadata:
127  name: nginx-ingress-clusterrole-nisa-binding
128roleRef:
129  apiGroup: rbac.authorization.k8s.io
130  kind: ClusterRole
131  name: nginx-ingress-clusterrole
132subjects:
133  - kind: ServiceAccount
134    name: nginx-ingress-serviceaccount
135    namespace: ingress-nginx

1kubectl apply -f rbac.yaml
2serviceaccount "nginx-ingress-serviceaccount" created
3clusterrole "nginx-ingress-clusterrole" created
4role "nginx-ingress-role" created
5rolebinding "nginx-ingress-role-nisa-binding" created
6clusterrolebinding "nginx-ingress-clusterrole-nisa-binding" created

5.部署nginx-ingress-controller #

下面部署关键组件nginx-ingress-controller这个ingress controller，前面提到Ingress是一种Kubernetes资源，借助Nginx、Haproxy或云厂商的负载均衡器将Kubernetes集群内的Service暴露到集群外，nginx-ingress-controller就是这个负载均衡器。

 1---
 2
 3apiVersion: extensions/v1beta1
 4kind: Deployment
 5metadata:
 6  name: nginx-ingress-controller
 7  namespace: ingress-nginx 
 8spec:
 9  replicas: 2
10  selector:
11    matchLabels:
12      app: ingress-nginx
13  template:
14    metadata:
15      labels:
16        app: ingress-nginx
17      annotations:
18        prometheus.io/port: '10254'
19        prometheus.io/scrape: 'true'
20    spec:
21      serviceAccountName: nginx-ingress-serviceaccount
22      affinity:
23        nodeAffinity:
24          requiredDuringSchedulingIgnoredDuringExecution:
25            nodeSelectorTerms:
26            - matchExpressions:
27              - key: kubernetes.io/hostname
28                operator: In
29                values:
30                - node1
31                - node2
32      containers:
33        - name: nginx-ingress-controller
34          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
35          args:
36            - /nginx-ingress-controller
37            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
38            - --configmap=$(POD_NAMESPACE)/nginx-configuration
39            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
40            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
41            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
42            - --annotations-prefix=nginx.ingress.kubernetes.io
43          env:
44            - name: POD_NAME
45              valueFrom:
46                fieldRef:
47                  fieldPath: metadata.name
48            - name: POD_NAMESPACE
49              valueFrom:
50                fieldRef:
51                  fieldPath: metadata.namespace
52          ports:
53          - name: http
54            containerPort: 80
55          - name: https
56            containerPort: 443
57          livenessProbe:
58            failureThreshold: 3
59            httpGet:
60              path: /healthz
61              port: 10254
62              scheme: HTTP
63            initialDelaySeconds: 10
64            periodSeconds: 10
65            successThreshold: 1
66            timeoutSeconds: 1
67          readinessProbe:
68            failureThreshold: 3
69            httpGet:
70              path: /healthz
71              port: 10254
72              scheme: HTTP
73            periodSeconds: 10
74            successThreshold: 1
75            timeoutSeconds: 1
76          securityContext:
77            runAsNonRoot: false

1kubectl apply -f nginx-ingress-controller.yaml
2deployment "nginx-ingress-controller" created

注意这里我们使用Kubernetes Pod调度的Node亲和性特性，将nginx-ingress-controller的两个Pod实例固定调度到node1和node2两个节点上。

6.将nginx-ingress-controller暴露到Kubernetes集群外 #

再看前面提到的Ingress是一种Kubernetes资源，借助Nginx、Haproxy或云厂商的负载均衡器将Kubernetes集群内的Service暴露到集群外，nginx-ingress-controller就是这个负载均衡器。所以我们还需要创建为nginx-ingress-controller创建一个Service并暴露到集群外边(ps: 这个有点先有鸡还是先有蛋的意思)。因为我们的Kubernetes集群是采以Bare-metal部署的，这里使用ExternalIP的形式创建一个ingress-nginx的Service，ingress-nginx.svc.yaml如下：

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  name: ingress-nginx
 5  namespace: ingress-nginx
 6spec:
 7  externalIPs:
 8  - 192.168.1.101
 9  - 192.168.1.102
10  ports:
11  - name: http
12    port: 80
13    targetPort: 80
14    protocol: TCP
15  - name: https
16    port: 443
17    targetPort: 443
18    protocol: TCP
19  selector:
20    app: ingress-nginx

其中192.168.1.101, 192.168.1.102分别对应node1和node2两个k8s node的ip。

1kubectl apply -f ingress-nginx.svc.yaml
2service "ingress-nginx" created

1kubectl get svc -n ingress-nginx
2NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP               PORT(S)              AGE
3default-http-backend   ClusterIP   10.111.47.113   <none>                    80/TCP               1h
4ingress-nginx          ClusterIP   10.96.122.13    192.168.1.101,192.168.1.102   80/TCP,443/TCP   55s

可以看出通过192.168.1.101,192.168.1.102:80/443将nginx-ingress-controller的nginx的80和443端口暴露到集群外边。

分别请求这两个ExternalIP:

1curl 192.168.1.101:80
2default backend - 404
3
4curl 192.168.1.102:80
5default backend - 404

因为我们还没有在Kubernetes集群中创建ingress资源，所以直接对这两个ExternalIP的请求被负载到了default backend上。

分别在node1和node2上查看：

1netstat -nltp | grep kube-proxy | grep 80
2tcp        0      0 192.168.1.101:80          0.0.0.0:*               LISTEN      2748840/kube-proxy
3
4netstat -nltp | grep kube-proxy | grep 80
5tcp        0      0 192.168.1.102:80          0.0.0.0:*               LISTEN      2989745/kube-proxy

可以看到ExternalIP的Service也是通过kube-proxy对外暴露的。这里的192.168.1.101和192.168.1.102是两个内网ip。实际中需要将边缘路由器或全局统一接入层的负载均衡器将到达公网ip的外网流量转发到内网ip上，外部通过域名访问集群中将会以ingress暴露的所有服务。至于如何创建ingress，将会在下一篇中介绍。

到此为止，我们已经将nginx-ingress-controller部署到了Kubernetes集群中。

参考 #

ingress-nginx