kubernetes 1.11 配置使用 nginx ingress

[TOC]

1. 实验环境说明

lab1: etcd master haproxy keepalived 192.168.105.92
lab2: etcd master haproxy keepalived 192.168.105.93
lab3: etcd master haproxy keepalived 192.168.105.94
lab4: node  192.168.105.95
lab4: node  192.168.105.96

vip(loadblancer ip): 192.168.105.99

2. 暴露服务的方式

由于 Pod 和 Service 是 kubernetes 集群范围内的虚拟概念，所以集群外的客户端系统无法通过 Pod 的 IP 地址或者 Service 的虚拟 IP 地址和虚拟端口号访问到它们。为了让外部客户端能够访问到这些服务，可以将 Pod 或 Service 的端口号映射到宿主机。

将容器应用的端口号映射到物理机
1）设置容器级别的hostPort，将容器应用的端口号映射到物理机
2）设置 Pod 级别的hostNetwork=true，该 Pod 中所有容器的端口号都将被直接映射到物理机上
将 Service 的端口号映射到物理机
1）设置nodePort映射到物理机，同时设置 Service 的类型为NodePort
2）设置 LoadBalancer 映射到云服务商提供的 LoadBalancer 地址

如果设置了 Service 的nodePort，那么集群会在每一个节点都监听设置的nodePort，外部客户端可以通过任意 nodeIP:Port的方式对集群服务进行访问。但是当集群中服务较多，那么需要管理的端口也会比较多，各个端口之间不能冲突，比较麻烦；另外，因为方式访问形式为nodeIP:Port的方式，那么对于一些 HTTP 服务，这种方式是无法做到根据 URL 路径进行转发的。

ingress 是 kubernetes V1.1 版本之后新增的资源对象，用于实现 HTTP 层业务路由机制。
实现 ingress 路由机制主要包括 3 个组件：
1）ingress 是 kubernetes 的一个资源对象，用于编写定义规则。
2）反向代理负载均衡器，通常以 Service 的 Port 方式运行，接收并按照 ingress 定义的规则进行转发，通常为 nginx，haproxy，traefik 等，本文使用 nginx。
3）ingress-controller，监听 apiserver，获取服务新增，删除等变化，并结合 ingress 规则动态更新到反向代理负载均衡器上，并重载配置使其生效。

3. 安装 nginx ingress

一般是以DaemonSet形式在每个 node 上创建一个资源，然后以hostNetwork的方式开放一个端口。

下面是根据官方只修改了 image 地址和添加了hostNetwork: true的 ingress-control 的 yaml 文件。

---
#---
#
#apiVersion: v1
#kind: Namespace
#metadata:
#  name: default
#---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: default-http-backend
          # Any image is permissible as long as:
          # 1. It serves a 404 page at /
          # 2. It serves 200 on a /healthz endpoint
          image: registry.cn-hangzhou.aliyuncs.com/google-containers/defaultbackend:1.4
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 10m
              memory: 20Mi
            requests:
              cpu: 10m
              memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    app: default-http-backend
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: default-http-backend
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: default
  labels:
    app: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: default
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - 'ingress-controller-leader-nginx'
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true # 添加这行
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            #- --apiserver-host=https://192.168.105.99:8443
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            #- --report-node-internal-ip-address=true
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---
## Expose Nginx Ingress controller
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx-ingress-controller
#  namespace: default
#spec:
#  type: LoadBalancer
#  ports:
#    - port: 80
#    #  nodePort: 30000
#      name: http
#    #- port: 18080
#    #  nodePort: 32000
#    #  name: http-mgmt
#  selector:
#    app: nginx-ingress-controller

4. 使用`DaemonSet`将 ingress control 固定 pod

DaemonSet 能够让所有（或者特定）的节点运行同一个 pod。

当节点加入到 K8S 集群中，pod 会被（DaemonSet）调度到该节点上运行，当节点从 K8S 集群中被移除，被 DaemonSet 调度的 pod 会被移除，如果删除 DaemonSet，所有跟这个 DaemonSet 相关的 pods 都会被删除。

在某种程度上，DaemonSet 承担了 RC 的部分功能，它也能保证相关 pods 持续运行，如果一个 DaemonSet 的 Pod 被杀死、停止、或者崩溃，那么 DaemonSet 将会重新创建一个新的副本在这台计算节点上。

一般应用于日志收集、监控采集、分布式存储守护进程、ingress 等。

Deployment 和 DaemonSet 的区别
Deployment 部署的副本 Pod 会分布在各个 Node 上，每个 Node 都可能运行好几个副本。DaemonSet 的不同之处在于：每个 Node 上最多只能运行一个副本。

主要区别：

The scalability is much better when using a Deployment, because you will have a Single-Pod-per-Node model when using the DaemonSet.
It is possible to exclusively run a Service on a dedicated set of machines using taints and tolerations with a DaemonSet.
On the other hand the DaemonSet allows you to access any Node directly on Port 80 and 443, where you have to setup a Service object with a Deployment.

以下是只使用标签方式，将 ingress control 固定 node。

将要使用的 node 节点打上标签

[root@lab1 nginx-ingress]# kubectl label nodes lab4 LB=ingress
node/lab4 labeled

修改 ingress control 的Deployment为DaemonSet，并根据标签使用nodeSelector。

---
#---
#
## 有需要创建namespace,取消该段注释
#apiVersion: v1
#kind: Namespace
#metadata:
#  name: default
#---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: default-http-backend
          # Any image is permissible as long as:
          # 1. It serves a 404 page at /
          # 2. It serves 200 on a /healthz endpoint
          image: registry.cn-hangzhou.aliyuncs.com/google-containers/defaultbackend:1.4
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 10m
              memory: 20Mi
            requests:
              cpu: 10m
              memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    app: default-http-backend
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: default-http-backend
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: default
  labels:
    app: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: default
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - 'ingress-controller-leader-nginx'
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default
---
apiVersion: extensions/v1beta1
kind: DaemonSet # Deployment修改成Daemonset
metadata:
  name: nginx-ingress-controller
  namespace: default
spec:
  #replicas: 2  # 注释该段
  #selector:
  #  matchLabels:
  #    app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      nodeSelector: # 节点标签
        LB: ingress
      #tolerations:  # 声明可接受的污点
      #- key: "LB"
      #  operator: "Equal"
      #  value: "NIC"
      #  effect: "NoSchedule"
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true # 添加这行，绑定到主机
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            #- --apiserver-host=https://192.168.105.99:8443
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            #- --report-node-internal-ip-address=true
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---
## Expose Nginx Ingress controller
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx-ingress-controller
#  namespace: default
#spec:
#  type: LoadBalancer
#  ports:
#    - port: 80
#    #  nodePort: 30000
#      name: http
#    #- port: 18080
#    #  nodePort: 32000
#    #  name: http-mgmt
#  selector:
#    app: nginx-ingress-controller

5. 配置 nginx url 转发

以下是配置 nginx 域名ingress.test.com的 url 转发规则。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-nginx-ingress
  namespace: default
spec:
  rules:
    - host: ingress.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nginx
              servicePort: 80

  #    - path: /nginx
  #      backend:
  #        serviceName: nginx
  #        servicePort: 80

配置好后，kubectl get ing查看结果。

外部通过域名绑定 Nginx Ingress controller 所在的节点的 IP 即可访问。

参数资料：
[1] https://kubernetes.io/docs/concepts/services-networking/ingress/
[2] https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
[3] https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
[4] https://blog.csdn.net/waltonwang/article/details/77587003
[5] https://blog.csdn.net/tiger435/article/details/73650174
[6] http://blog.51cto.com/newfly/2067531