青蛙小白
博客 / 2018/06

Kubernetes Ingress实战(一)：在Kubernetes集群中部署NGINX Ingress Controller

2018/06/06 · — 字 · 阅读约 — 分钟 ·
#kubernetes
目录

去年整理过一篇《Kubernetes Ingress实战》，经过这一年的发展Kubernetes的Ingress发生了很大的变化，原来的文章很多地方都不适用了。因此决定结合我们目前的使用情况重新写几篇Kubernetes Ingress相关的分享，内容是比较入门和初级的实操，请高手勿喷。

Ingress是一种Kubernetes资源，借助Nginx、Haproxy或云厂商的负载均衡器将Kubernetes集群内的Service暴露到集群外。因为我们的Kubernetes集群是采以Bare-metal部署的，所以这个系列将主要介绍ingress-nginx。

下面将ingress-nginx部署到用于测试的Kubernetes集群中，这里使用的Kubernetes的版本是1.9.8。

1.创建namespace

首先创建名称为ingress-nginx的namespace，ingress-nginx的相关组件的都将在这个namespace下。

namespace.yaml的内容如下：

---

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
kubectl create -f namespace.yaml

2.部署default backend

下面来创建default-http-backend的Deployment和Service.

default-backend.yaml的内容如下：

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend

需要注意gcr.io/google_containers/defaultbackend:1.4这个镜像在gcr上，由于某些已知的原因，最好将此镜像放到本地Kubernetes集群的私有镜像仓库中

kubectl apply -f default-backend.yaml
deployment "default-http-backend" created
service "default-http-backend" created

确认default-http-backend的Service和Pod被创建，并且Pod处于Running状态。

kubectl get svc,deploy,pod -n ingress-nginx
NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
svc/default-http-backend   ClusterIP   10.111.47.113   <none>        80/TCP    1m

NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/default-http-backend   1         1         1            1           1m

NAME                                       READY     STATUS    RESTARTS   AGE
po/default-http-backend-64985b4bcb-9pn7s   1/1       Running   0          1m

default-http-backend从名称上来看即默认的后端，当集群外部的请求通过ingress进入到集群内部时，如果无法负载到相关后端的Service上，这种未知的请求将会被负载到这个默认的后端上。

3.创建ConfigMap

接下来创建nginx-configuration、tcp-services、udp-services三个ConfigMap：

kubectl get deployment,pod -n ingress-nginx
NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/default-http-backend       1         1         1            1           1h
deploy/nginx-ingress-controller   2         2         2            2           2m

NAME                                           READY     STATUS    RESTARTS   AGE
po/default-http-backend-64985b4bcb-9pn7s       1/1       Running   0          1h
po/nginx-ingress-controller-7f9d776bff-5k28n   1/1       Running   0          2m
po/nginx-ingress-controller-7f9d776bff-rwls2   1/1       Running   0          2m
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
kubectl apply -f configmap.yaml
kubectl apply -f tcp-services-configmap.yaml
kubectl apply -f udp-services-configmap.yaml

kubectl get cm -n ingress-nginx
NAME                  DATA      AGE
nginx-configuration   0         59s
tcp-services          0         51s
udp-services          0         45s

4.创建ServiceAccount并进行授权

接下来创建ServiceAccount nginx-ingress-serviceaccount，并创建相关的ClusterRole, Role, ClusterRoleBinding, RoleBinding以对其进行授权：

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
kubectl apply -f rbac.yaml
serviceaccount "nginx-ingress-serviceaccount" created
clusterrole "nginx-ingress-clusterrole" created
role "nginx-ingress-role" created
rolebinding "nginx-ingress-role-nisa-binding" created
clusterrolebinding "nginx-ingress-clusterrole-nisa-binding" created

5.部署nginx-ingress-controller

下面部署关键组件nginx-ingress-controller这个ingress controller，前面提到Ingress是一种Kubernetes资源，借助Nginx、Haproxy或云厂商的负载均衡器将Kubernetes集群内的Service暴露到集群外，nginx-ingress-controller就是这个负载均衡器。

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - node1
                - node2
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          securityContext:
            runAsNonRoot: false
kubectl apply -f nginx-ingress-controller.yaml
deployment "nginx-ingress-controller" created

注意这里我们使用Kubernetes Pod调度的Node亲和性特性，将nginx-ingress-controller的两个Pod实例固定调度到node1和node2两个节点上。

6.将nginx-ingress-controller暴露到Kubernetes集群外

再看前面提到的Ingress是一种Kubernetes资源，借助Nginx、Haproxy或云厂商的负载均衡器将Kubernetes集群内的Service暴露到集群外，nginx-ingress-controller就是这个负载均衡器。所以我们还需要创建为nginx-ingress-controller创建一个Service并暴露到集群外边(ps: 这个有点先有鸡还是先有蛋的意思)。 因为我们的Kubernetes集群是采以Bare-metal部署的，这里使用ExternalIP的形式创建一个ingress-nginx的Service，ingress-nginx.svc.yaml如下：

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  externalIPs:
  - 192.168.1.101
  - 192.168.1.102
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx

其中192.168.1.101, 192.168.1.102分别对应node1和node2两个k8s node的ip。

kubectl apply -f ingress-nginx.svc.yaml
service "ingress-nginx" created
kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP               PORT(S)              AGE
default-http-backend   ClusterIP   10.111.47.113   <none>                    80/TCP               1h
ingress-nginx          ClusterIP   10.96.122.13    192.168.1.101,192.168.1.102   80/TCP,443/TCP   55s

可以看出通过192.168.1.101,192.168.1.102:80/443将nginx-ingress-controller的nginx的80和443端口暴露到集群外边。

分别请求这两个ExternalIP:

curl 192.168.1.101:80
default backend - 404

curl 192.168.1.102:80
default backend - 404

因为我们还没有在Kubernetes集群中创建ingress资源，所以直接对这两个ExternalIP的请求被负载到了default backend上。

分别在node1和node2上查看：

netstat -nltp | grep kube-proxy | grep 80
tcp        0      0 192.168.1.101:80          0.0.0.0:*               LISTEN      2748840/kube-proxy

netstat -nltp | grep kube-proxy | grep 80
tcp        0      0 192.168.1.102:80          0.0.0.0:*               LISTEN      2989745/kube-proxy

可以看到ExternalIP的Service也是通过kube-proxy对外暴露的。这里的192.168.1.101和192.168.1.102是两个内网ip。 实际中需要将边缘路由器或全局统一接入层的负载均衡器将到达公网ip的外网流量转发到内网ip上，外部通过域名访问集群中将会以ingress暴露的所有服务。 至于如何创建ingress，将会在下一篇中介绍。

到此为止，我们已经将nginx-ingress-controller部署到了Kubernetes集群中。

参考

TAGS # kubernetes
评论