Kubernetes Nginx Ingress Controller on NodePort

Question

I am deploying the nginx based ingress controller on Kubernetes cluster managed by RKE. ( I have also tried the same directly without RKE ).

In both the cases , it tries to use/bind to Ports 80 , and 443 on the host, and it fails because in the pod security policy for all service accounts I am not allowing host ports.

In fact I don't need to access the ingress directly on the hosts, but I want to access the ingress controller as a Service on the NodePort from external LoadBalancer.

Is there way to deploy Nginx ingress controller not to use any hostPort.

Answer 1

Done by disabling hostNetwork , and remove unnecessary privileges and capabilities:

C02W84XMHTD5:Downloads iahmad$ kubectl get deployments -n ingress-nginx -o yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    annotations:
      deployment.kubernetes.io/revision: "1"

    labels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
    name: nginx-ingress-controller
    namespace: ingress-nginx
    resourceVersion: "68427"
    selfLink: /apis/extensions/v1beta1/namespaces/ingress-nginx/deployments/nginx-ingress-controller
    uid: 0b92b556-12fa-11ea-9d82-08002762a3c5
  spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    strategy:
      rollingUpdate:
        maxSurge: 25%
        maxUnavailable: 25%
      type: RollingUpdate
    template:
      metadata:
        annotations:
          prometheus.io/port: "10254"
          prometheus.io/scrape: "true"
        creationTimestamp: null
        labels:
          app.kubernetes.io/name: ingress-nginx
          app.kubernetes.io/part-of: ingress-nginx
      spec:
        containers:
        - args:
          - /nginx-ingress-controller
          - --configmap=$(POD_NAMESPACE)/nginx-configuration
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          - --publish-service=$(POD_NAMESPACE)/ingress-nginx
          - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                - /wait-shutdown
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          name: nginx-ingress-controller
          ports:
          - containerPort: 80
            name: http
            protocol: TCP
          - containerPort: 443
            name: https
            protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          resources: {}
          securityContext:
            runAsUser: 33
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        serviceAccount: nginx-ingress-serviceaccount
        serviceAccountName: nginx-ingress-serviceaccount
        terminationGracePeriodSeconds: 300
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: 2019-11-29T22:46:59Z
      lastUpdateTime: 2019-11-29T22:46:59Z
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    - lastTransitionTime: 2019-11-29T22:46:13Z
      lastUpdateTime: 2019-11-29T22:46:59Z
      message: ReplicaSet "nginx-ingress-controller-84758fb96c" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
    observedGeneration: 1
    readyReplicas: 1
    replicas: 1
    updatedReplicas: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

and then creating a nodeport service pointing to the ingress controller ports:

C02W84XMHTD5:Downloads iahmad$ kubectl get svc -n ingress-nginx -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:

    labels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
    name: ingress-nginx
    namespace: ingress-nginx
    resourceVersion: "68063"
    selfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx
    uid: 7aa425a4-12f9-11ea-9d82-08002762a3c5
  spec:
    clusterIP: 10.97.110.93
    externalTrafficPolicy: Cluster
    ports:
    - name: http
      nodePort: 30864
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      nodePort: 30716
      port: 443
      protocol: TCP
      targetPort: 443
    selector:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
    sessionAffinity: None
    type: NodePort
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
C02W84XMHTD5:Downloads iahmad$ 

Answer 2

In the documentation about NodePort, you can find that this type can allocate ports from range 30000-32767. However there is a workaround. If you will add a special flag --service-node-port-range with requested range, admission controller allow you to create NodePort with Ports 80 and 443.

You will need to go to /etc/kubernetes/manifests/, edit kube-apiserver.yaml with sudo and add entry - --service-node-port-range=1-32767. After that you need to save it.

Now you will need to create service. To do that you need to edit this yaml and in ports add node port to spec.ports

Before:

 ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP

After:

  ports:
  - name: http
    nodePort: 80
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 443
    port: 443
    protocol: TCP
    targetPort: 443

After thoses changes you can edit again kube-apiserver.yaml in /etc/kubernetes/manifests/ and comment it using # in the same line as - --service-node-port-range.

Then you will be able to curl this NodePort address and Node address.

EDIT: After clarification

Ingress can be deployed in two ways. The first one is deploy Nginx as Deamonset which is requiring hostPort inside configuration file. However there is another option, you can deploy Nginx as Deployment.

NodeIP and Known Port: Pods in the DaemonSet can use a hostPort, so that the pods are reachable via the node IPs. Clients know the list of node IPs somehow, and know the port by convention.

However in the bottom of the page you can find:

DaemonSets are similar to Deployments in that they both create Pods, and those Pods have processes which are not expected to terminate (e.g. web servers, storage servers).

Use a Deployment for stateless services, like frontends, where scaling up and down the number of replicas and rolling out updates are more important than controlling exactly which host the Pod runs on. Use a DaemonSet when it is important that a copy of a Pod always run on all or certain hosts, and when it needs to start before other Pods.

You need to deploy Ingress as Deployment and not as Deamonset.

Example of Nginx Deployment can be found here. As Deployment is not requiring hostPort you will be able to create pods without this parameter.