Nginx Ingress Controller on Bare Metal

After many hours of reading, trial-&-error, and general frustration… I have collected a few helpful bits WRT configuring the nginx ingress controller for a bare metal configuration.

The Challenge:
Deploy a k8s environment to host a small collection of apps, which sits behind a single routable IP address. Each app must:

• manage its own SSL certificate from Lets Encrypt
• be addressed with name-based routing and also support rewrites correctly. (www.example.com, if it redirects to /login must not send to wrong back-end service)
• whenever possible, leverage a fully automated end-to-end deployment pipeline, all in-house, within the cluster (Jenkins, private repos, etc. - will be in a different post)

The Journey:

Initially struggled with the nginx ingress controller because of some of the default parameters and a general bias the controller has for running in a cloud providers IaaS - such as AWS / GCE. This also led to a general lack of examples and documentation for the scenario I was trying to solve. Specifically:

• The controller defaults to forwarding http to https.  I had to turn this off to be able to test http only services.  I did this by adding 'ssl-redirect: false’ to the ingress controller’s configmap in the data section.
• It also implements a strict HSTS configuration. This caused the temp certs created during setup to become “stuck" in my browsers and lead me down 'troubleshooting rabbit holes' which were not relevant or fruitful.  To correct, I had to set these values in the ingress controller configmap:
    data:  
      hsts: "true"  
      hsts-include-subdomains: "true"
      hsts-max-age: "0"  
      hsts-preload: "false"
• Using the helm chart for the ingress controller installation did not work as desired.  I wound up installing manually from yml files which I massaged from the nginx ingress controller repo and examples.  All told, I wound up with a series of 6 scripts which I installed sequentially (I'll publish these later - time permitting):
• 01.default-backend.yml
• 02.default-backend-svc.yml
• 03.ingress-controller-configmap.yml
• 04.ingress-controller-svc.yml
• 05.ingress-controller-rbac-roles.yml 
• 06.ingress-controller-deploy-rbac.yml
  
• Configuring the kube-lego package was also a challenge, as getting the cert validation step to work required the site to be routable before it was secured. It also exposed the temp self-signed cert which led me to the issues above with HSTS. Ultimately, I learned I needed to use these parameters when installing the lego chart:
  
  helm install stable/kube-lego --namespace <my_namespace> --name <my_deploy_name> --set config.LEGO_EMAIL=<me_email>,config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory,LEGO_SUPPORTED_INGRESS_CLASS=nginx,LEGO_SUPPORTED_INGRESS_PROVIDER=nginx,LEGO_DEFAULT_INGRESS_CLASS=nginx,image.pullPolicy=Always,rbac.create=true,rbac.serviceAccountName=<my_sa_name>
  
  I hope it goes without saying that the <> parts are variable which I put appropriate inputs in myself for. You need to do the same.
  
• I needed something other than the default backend running to tell when I got the correct settings for the ingress. The easiest thing to use wound up being the http-svc described as a prerequisite in the nginx ingress controller repo:   https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md

As a side note - besides the documentation for each of the projects involved, and the k8s docs, I found this site to be VERY helpful:  http://stytex.de/blog/2017/01/25/deploy-kubernetes-to-bare-metal-with-nginx/

The morale of the story is this - routing in k8s is complex enough, but the examples readily available for those of us trying to now apply this to an on-premise and/or bare metal deployment have a lot of gaps. That leaves us to hunt and search to find the materials we need. Keep good notes and share with all, as the troubleshooting will be critical to us everyone getting better with Kubernetes. I am just trying to do my part.

Ingress Routing for ICP

I have spent time this week trying to play through a few more customer scenarios with IBM Cloud Private.  The issue under test this week was related to a set of services which all want to use the same port, but need to be mapped to either different URLs OR be mapped to different paths on the same URL.  We know Kubernetes is capable of this, but is there anything different in doing this with ICP, which already has some opinions about things like which ingress controller to use?

I started with the basics and deployed a simple, containerized application into my ICP environment.  This blog post outlines some of the basics for exposing an application which is running in ICP.  In particular, I used it to help with deploying a container and the basics of exposing an app.  (For a more complete experience, I also ran through this with a node.js app which my firm wrote, built it with Jenkins and placed the image into the ICP repo.  When deploying from the ICP Docker repo, you must specify the image with the repo address like this: mycluster.icp:8500/default/helloworld:latest where the values are - <internal_cluster_url>:8500/<namespace>/<image>:<tag>)

With an application deployed, and a couple of instances running, I set out to work through the ingress configuration to make the site available in a fan-out pattern to start with.  For this exercise, I was using the apache web server, so I called the app static.  The path on which the app was to be exposed was <proxy-address>/static/.

I created an Ingress by navigating to the Applications link in the menu, and selecting the settings icon on the right side of the screen corresponding to the application I deployed.

The form to configure the ingress looked like this:

I realized there was no easy way to define path based routing.  I attempted to add the path to the url, adjusted settings in numerous fields, until ultimately I concluded this cannot happen through the ICP form interface... NO PROBLEM!  I switched on JSON mode and we were off to the races.

I navigated to my ingress record, selected Edit, and clicked the toggle at the top to manipulate my entry.

{

  "kind": "Ingress",

  "apiVersion": "extensions/v1beta1",

  "metadata": {

    "name": "default-static-demo",

    "namespace": "default",

    "selfLink": "/apis/extensions/v1beta1/namespaces/default/ingresses/default-static-demo",

    "uid": "ac450ec0-aa0d-11e7-9443-0800279d0b70",

    "resourceVersion": "359332",

    "generation": 4,

    "creationTimestamp": "2017-10-05T20:42:15Z",

    "annotations": {

      "ingress.kubernetes.io/rewrite-target": "/"

    }

  },

  "spec": {

    "rules": [

      {

        "http": {

          "paths": [

            {

              "path": "/static/",

              "backend": {

                "serviceName": "default-static-demo",

                "servicePort": 80

              }

            }

          ]

        }

      }

    ]

  }

}

By appending the necessary bits and clicking submit, ICP stored my new configuration, incremented my generation number, and triggered the steps to update the configuration of my cluster to serve up my application on <host>/static/.  The rewrite directive handled relative URL rewriting and everything.

Next steps for me will be to convert this into a yaml template so I can bundle this into my next helm chart!

To make the application work for a different host (static.<host>) for example, I was able to use the form method, and place the desired URL in the hostname filed as depicted above.

Jenkins CI / CD in ICP

As we continue our exploration of IBM Cloud Private (ICP), we wanted to be able to grab some of our existing CI patterns and port them into ICP as opposed to our own Docker Swarm based technology stack.  Someone shared this link with me as a starting point:

https://medium.com/ibm-cloud/running-jenkins-ci-cd-deployments-in-an-ibm-cloud-private-environment-525d5eff33dc

It was a good place to begin, and describes the steps to get started, there were some gaps from this set of instruction because they were created with an older version of what is now ICP.

Tips from my experiences:

1. Do NOT configure Jenkins executors.  The ICP Jenkins chart will automatically spawn disposable slaves to build apps and images.  Be patient while the jobs sit in a pending state until the slave is built, communicates with the master, ... well, for all the magic stuff that makes it work to happen.  If you configured your build step correctly, it will eventually get there.
2. If building a Docker container, the image repository on a default ICP installation has a different address than the one in the instructions.  The correct default repo address is:
   https://mycluster:8500
3. When building your registry credential entry in Jenkins, the name of the variable is flexible.  You can make it something that is meaningful to you, but remember that the username and password are the ICP admin user information you specified at installation time, unless you created your own set of credentials in the namespace.
4. Building and replacing the image in the repo does not update the running container on its own.  Writing a chart and leveraging the Kubernetes native way to manage release lifecycle would be the best native way to handle releases without needing to write it yourself.

This is just the tip of the iceberg, as we explore patterns of end-to-end CI/CD with the ICP platform and the tooling included, and available within the platform.  For Jenkins itself, there are many opportunities for optimizing the pattern, such as triggering deployments from SCM as opposed to needlessly polling the source management system are apparent, but not in the scope of this post as it is merely an addendum to the starter pattern already found available.

Cheers!