kubernetes 1.11配置使用nginx ingress

2018年8月24日

ingresskubernetesnginx

kubernetes 1.11 配置使用 nginx ingress

1. 实验环境说明

lab1: etcd master haproxy keepalived 192.168.105.92
lab2: etcd master haproxy keepalived 192.168.105.93
lab3: etcd master haproxy keepalived 192.168.105.94
lab4: node  192.168.105.95
lab4: node  192.168.105.96

vip(loadblancer ip): 192.168.105.99

2. 暴露服务的方式

由于 Pod 和 Service 是 kubernetes 集群范围内的虚拟概念，所以集群外的客户端系统无法通过 Pod 的 IP 地址或者 Service 的虚拟 IP 地址和虚拟端口号访问到它们。为了让外部客户端能够访问到这些服务，可以将 Pod 或 Service 的端口号映射到宿主机。

将容器应用的端口号映射到物理机
1）设置容器级别的hostPort，将容器应用的端口号映射到物理机
2）设置 Pod 级别的hostNetwork=true，该 Pod 中所有容器的端口号都将被直接映射到物理机上
将 Service 的端口号映射到物理机
1）设置nodePort映射到物理机，同时设置 Service 的类型为NodePort
2）设置 LoadBalancer 映射到云服务商提供的 LoadBalancer 地址

如果设置了 Service 的nodePort，那么集群会在每一个节点都监听设置的nodePort，外部客户端可以通过任意 nodeIP:Port的方式对集群服务进行访问。但是当集群中服务较多，那么需要管理的端口也会比较多，各个端口之间不能冲突，比较麻烦；另外，因为方式访问形式为nodeIP:Port的方式，那么对于一些 HTTP 服务，这种方式是无法做到根据 URL 路径进行转发的。

ingress 是 kubernetes V1.1 版本之后新增的资源对象，用于实现 HTTP 层业务路由机制。
实现 ingress 路由机制主要包括 3 个组件：
1）ingress 是 kubernetes 的一个资源对象，用于编写定义规则。
2）反向代理负载均衡器，通常以 Service 的 Port 方式运行，接收并按照 ingress 定义的规则进行转发，通常为 nginx，haproxy，traefik 等，本文使用 nginx。
3）ingress-controller，监听 apiserver，获取服务新增，删除等变化，并结合 ingress 规则动态更新到反向代理负载均衡器上，并重载配置使其生效。

3. 安装 nginx ingress

一般是以DaemonSet形式在每个 node 上创建一个资源，然后以hostNetwork的方式开放一个端口。

下面是根据官方只修改了 image 地址和添加了hostNetwork: true的 ingress-control 的 yaml 文件。

---
#---
#
#apiVersion: v1
#kind: Namespace
#metadata:
#  name: default
#---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: default-http-backend
          # Any image is permissible as long as:
          # 1. It serves a 404 page at /
          # 2. It serves 200 on a /healthz endpoint
          image: registry.cn-hangzhou.aliyuncs.com/google-containers/defaultbackend:1.4
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 10m
              memory: 20Mi
            requests:
              cpu: 10m
              memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    app: default-http-backend
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: default-http-backend
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: default
  labels:
    app: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: default
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - 'ingress-controller-leader-nginx'
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true # 添加这行
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            #- --apiserver-host=https://192.168.105.99:8443
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            #- --report-node-internal-ip-address=true
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---
## Expose Nginx Ingress controller
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx-ingress-controller
#  namespace: default
#spec:
#  type: LoadBalancer
#  ports:
#    - port: 80
#    #  nodePort: 30000
#      name: http
#    #- port: 18080
#    #  nodePort: 32000
#    #  name: http-mgmt
#  selector:
#    app: nginx-ingress-controller

4. 使用`DaemonSet`将 ingress control 固定 pod

DaemonSet 能够让所有（或者特定）的节点运行同一个 pod。

当节点加入到 K8S 集群中，pod 会被（DaemonSet）调度到该节点上运行，当节点从 K8S 集群中被移除，被 DaemonSet 调度的 pod 会被移除，如果删除 DaemonSet，所有跟这个 DaemonSet 相关的 pods 都会被删除。

在某种程度上，DaemonSet 承担了 RC 的部分功能，它也能保证相关 pods 持续运行，如果一个 DaemonSet 的 Pod 被杀死、停止、或者崩溃，那么 DaemonSet 将会重新创建一个新的副本在这台计算节点上。

一般应用于日志收集、监控采集、分布式存储守护进程、ingress 等。

Deployment 和 DaemonSet 的区别
Deployment 部署的副本 Pod 会分布在各个 Node 上，每个 Node 都可能运行好几个副本。DaemonSet 的不同之处在于：每个 Node 上最多只能运行一个副本。

主要区别：

The scalability is much better when using a Deployment, because you will have a Single-Pod-per-Node model when using the DaemonSet.
It is possible to exclusively run a Service on a dedicated set of machines using taints and tolerations with a DaemonSet.
On the other hand the DaemonSet allows you to access any Node directly on Port 80 and 443, where you have to setup a Service object with a Deployment.

以下是只使用标签方式，将 ingress control 固定 node。

将要使用的 node 节点打上标签

[root@lab1 nginx-ingress]# kubectl label nodes lab4 LB=ingress
node/lab4 labeled

修改 ingress control 的Deployment为DaemonSet，并根据标签使用nodeSelector。

---
#---
#
## 有需要创建namespace,取消该段注释
#apiVersion: v1
#kind: Namespace
#metadata:
#  name: default
#---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: default-http-backend
          # Any image is permissible as long as:
          # 1. It serves a 404 page at /
          # 2. It serves 200 on a /healthz endpoint
          image: registry.cn-hangzhou.aliyuncs.com/google-containers/defaultbackend:1.4
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 10m
              memory: 20Mi
            requests:
              cpu: 10m
              memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: default
  labels:
    app: default-http-backend
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: default-http-backend
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: default
  labels:
    app: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: default
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - 'extensions'
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - 'ingress-controller-leader-nginx'
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: default
---
apiVersion: extensions/v1beta1
kind: DaemonSet # Deployment修改成Daemonset
metadata:
  name: nginx-ingress-controller
  namespace: default
spec:
  #replicas: 2  # 注释该段
  #selector:
  #  matchLabels:
  #    app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      nodeSelector: # 节点标签
        LB: ingress
      #tolerations:  # 声明可接受的污点
      #- key: "LB"
      #  operator: "Equal"
      #  value: "NIC"
      #  effect: "NoSchedule"
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true # 添加这行，绑定到主机
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.17.1
          args:
            - /nginx-ingress-controller
            #- --apiserver-host=https://192.168.105.99:8443
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            #- --report-node-internal-ip-address=true
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---
## Expose Nginx Ingress controller
#apiVersion: v1
#kind: Service
#metadata:
#  name: nginx-ingress-controller
#  namespace: default
#spec:
#  type: LoadBalancer
#  ports:
#    - port: 80
#    #  nodePort: 30000
#      name: http
#    #- port: 18080
#    #  nodePort: 32000
#    #  name: http-mgmt
#  selector:
#    app: nginx-ingress-controller

5. 配置 nginx url 转发

以下是配置 nginx 域名ingress.test.com的 url 转发规则。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-nginx-ingress
  namespace: default
spec:
  rules:
    - host: ingress.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nginx
              servicePort: 80

  #    - path: /nginx
  #      backend:
  #        serviceName: nginx
  #        servicePort: 80

配置好后，kubectl get ing查看结果。

外部通过域名绑定 Nginx Ingress controller 所在的节点的 IP 即可访问。

参数资料：
[1] https://kubernetes.io/docs/concepts/services-networking/ingress/
[2] https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
[3] https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
[4] https://blog.csdn.net/waltonwang/article/details/77587003
[5] https://blog.csdn.net/tiger435/article/details/73650174
[6] http://blog.51cto.com/newfly/2067531

kubernetes 1.11配置使用nginx ingress

kubernetes 1.11 配置使用 nginx ingress

目录

1. 实验环境说明

2. 暴露服务的方式

3. 安装 nginx ingress

4. 使用`DaemonSet`将 ingress control 固定 pod

5. 配置 nginx url 转发

评论

kubernetes 1.11 配置使用 nginx ingress

目录

1. 实验环境说明

2. 暴露服务的方式

3. 安装 nginx ingress

4. 使用`DaemonSet`将 ingress control 固定 pod

5. 配置 nginx url 转发

kubernetes 1.11配置使用nginx ingress

kubernetes 1.11 配置使用 nginx ingress

目录

1. 实验环境说明

2. 暴露服务的方式

3. 安装 nginx ingress

4. 使用DaemonSet将 ingress control 固定 pod

5. 配置 nginx url 转发

评论

kubernetes 1.11 配置使用 nginx ingress

目录

1. 实验环境说明

2. 暴露服务的方式

3. 安装 nginx ingress

4. 使用DaemonSet将 ingress control 固定 pod

5. 配置 nginx url 转发

4. 使用`DaemonSet`将 ingress control 固定 pod

4. 使用`DaemonSet`将 ingress control 固定 pod