Nginx Ingress Controller on Bare Metal

After many hours of reading, trial-&-error, and general frustration… I have collected a few helpful bits WRT configuring the nginx ingress controller for a bare metal configuration.

The Challenge:
Deploy a k8s environment to host a small collection of apps, which sits behind a single routable IP address. Each app must:

• manage its own SSL certificate from Lets Encrypt
• be addressed with name-based routing and also support rewrites correctly. (www.example.com, if it redirects to /login must not send to wrong back-end service)
• whenever possible, leverage a fully automated end-to-end deployment pipeline, all in-house, within the cluster (Jenkins, private repos, etc. - will be in a different post)

The Journey:

Initially struggled with the nginx ingress controller because of some of the default parameters and a general bias the controller has for running in a cloud providers IaaS - such as AWS / GCE. This also led to a general lack of examples and documentation for the scenario I was trying to solve. Specifically:

• The controller defaults to forwarding http to https.  I had to turn this off to be able to test http only services.  I did this by adding 'ssl-redirect: false’ to the ingress controller’s configmap in the data section.
• It also implements a strict HSTS configuration. This caused the temp certs created during setup to become “stuck" in my browsers and lead me down 'troubleshooting rabbit holes' which were not relevant or fruitful.  To correct, I had to set these values in the ingress controller configmap:
    data:  
      hsts: "true"  
      hsts-include-subdomains: "true"
      hsts-max-age: "0"  
      hsts-preload: "false"
• Using the helm chart for the ingress controller installation did not work as desired.  I wound up installing manually from yml files which I massaged from the nginx ingress controller repo and examples.  All told, I wound up with a series of 6 scripts which I installed sequentially (I'll publish these later - time permitting):
• 01.default-backend.yml
• 02.default-backend-svc.yml
• 03.ingress-controller-configmap.yml
• 04.ingress-controller-svc.yml
• 05.ingress-controller-rbac-roles.yml 
• 06.ingress-controller-deploy-rbac.yml
  
• Configuring the kube-lego package was also a challenge, as getting the cert validation step to work required the site to be routable before it was secured. It also exposed the temp self-signed cert which led me to the issues above with HSTS. Ultimately, I learned I needed to use these parameters when installing the lego chart:
  
  helm install stable/kube-lego --namespace <my_namespace> --name <my_deploy_name> --set config.LEGO_EMAIL=<me_email>,config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory,LEGO_SUPPORTED_INGRESS_CLASS=nginx,LEGO_SUPPORTED_INGRESS_PROVIDER=nginx,LEGO_DEFAULT_INGRESS_CLASS=nginx,image.pullPolicy=Always,rbac.create=true,rbac.serviceAccountName=<my_sa_name>
  
  I hope it goes without saying that the <> parts are variable which I put appropriate inputs in myself for. You need to do the same.
  
• I needed something other than the default backend running to tell when I got the correct settings for the ingress. The easiest thing to use wound up being the http-svc described as a prerequisite in the nginx ingress controller repo:   https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md

As a side note - besides the documentation for each of the projects involved, and the k8s docs, I found this site to be VERY helpful:  http://stytex.de/blog/2017/01/25/deploy-kubernetes-to-bare-metal-with-nginx/

The morale of the story is this - routing in k8s is complex enough, but the examples readily available for those of us trying to now apply this to an on-premise and/or bare metal deployment have a lot of gaps. That leaves us to hunt and search to find the materials we need. Keep good notes and share with all, as the troubleshooting will be critical to us everyone getting better with Kubernetes. I am just trying to do my part.