How to configure Digital Ocean Kubernetes with cert-manager and external-dns

This is a guide that combines the instructions from several different sources to allow you to install the tools you need to get our Helm StarterKit going on Digital Ocean Kubernetes and benefit from automated DNS and the nginx load balancer.

If you found this article looking for how to install these tools but don't have any interest in the StarterKit that's ok too! You can just use it to help get everything installed and then do your own thing.

Prerequisites:

• A managed Digital Ocean Managed Kubernetes cluster
• The kubectl command-line tool installed on your local machine and configured to connect to your cluster.
• The DigitalOcean command-line client, doctl, installed on your machine. See How To Use doctl for more information on using doctl.
• A DigitalOcean API key (Personal Access Token) with read and write permissions. To create one, visit How to Create a Personal Access Token.
• The helm command-line tool installed and configured on your machine
• A private docker registry (we recommend DO for easy linking)
• Our Helm StarterKit repo cloned to your machine

Installing the Nginx Ingress Controller

In this section, we will install the Kubernetes-maintained Nginx Ingress Controller. This controller manages traffic routing for backend services based on rules defined in Ingress Resources. These rules can be path-based (e.g., /web1) or host-based (e.g., web1.your_domain.com).

For DigitalOcean Kubernetes clusters, the Nginx Ingress Controller will automatically provision a DigitalOcean Load Balancer to handle external traffic. This Load Balancer directs traffic to the Nginx Ingress Controller Pod, which then forwards it to the appropriate backend services.

Workaround for DigitalOcean Kubernetes

Before proceeding, we need to enable Pod-to-Pod communication through the Nginx Ingress load balancer. This is necessary for cert-manager's self-check when provisioning certificates from Let's Encrypt.

1. Create an A record in your DNS management service for workaround.example.com pointing to the external IP of your DigitalOcean Load Balancer. You can find this IP in the DigitalOcean interface or by looking up the ingress-nginx service in kubectl.
2. Add the Nginx Ingress Controller repository to Helm:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

3. Update your Helm repositories:

helm repo update

4. Install the Nginx Ingress Controller, replacing workaround.example.com with your configured domain:

helm install nginx-ingress ingress-nginx/ingress-nginx \
--set controller.publishService.enabled=true \
--set-string controller.config.use-forward-headers=true,controller.config.compute-full-forward-for=true,controller.config.use-proxy-protocol=true  \
--set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-enable-proxy-protocol=true" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-hostname=workaround\.example\.com"

After these steps, the Nginx Ingress Controller will be installed and configured on your DigitalOcean Kubernetes cluster. After you install it, the output should contain a command to verify the installation if you'd like.

Installing and configuring cert-manager

In this section, we will install cert-manager, a Kubernetes add-on that automates the provisioning and management of TLS certificates from Let's Encrypt and other certificate authorities (CAs). By annotating Ingress Resources and configuring Issuers or ClusterIssuers, certificates can be automatically requested and applied to your applications.

Install cert-manager:

Follow the instructions in the official cert-manager documentation for Helm installations: https://cert-manager.io/docs/installation/helm/

Create Staging and Production Issuers:

1. Before issuing certificates, we'll need to create an Issuer, which specifies the certificate authority cert-manager uses to issue new certs. We are going to use Let's Encrypt, a free certificate authority. Let's Encrypt provides both a staging server for testing and a production server for live certificates.
2. In the Polar Labs Standard Helm Deployment repository, you'll find two templates under _support-templates/cert-manager/: staging-issuer.yaml and production-issuer.yaml
3. Replace the placeholders in these files (your email address) with the appropriate values.
4. Begin by creating the staging ClusterIssuer using the staging-issuer.yaml file. This ClusterIssuer, named letsencrypt-staging, will use Let's Encrypt's staging server for testing your certificate configuration without incurring rate limits.

kubectl create -f _support-templates/cert-manager/staging-issuer.yaml

5. Replace the placeholders in these files (your email address) with the appropriate values.
6. Begin by creating the staging ClusterIssuer using the staging-issuer.yaml file. This ClusterIssuer, named letsencrypt-staging, will use Let's Encrypt's staging server for testing your certificate configuration without incurring rate limits.

kubectl create -f _support-templates/cert-manager/staging-issuer.yaml

7. Once your staging configuration is created successfully, create the production ClusterIssuer using the production-issuer.yaml file. This ClusterIssuer, named letsencrypt-production, will use Let's Encrypt's production server to issue verifiable TLS certificates for your applications.

kubectl create -f _support-templates/cert-manager/production-issuer.yaml

With these steps, you have successfully installed cert-manager and configured the staging and production ClusterIssuers to obtain certificates from Let's Encrypt. Certificates will be issued when you annotate and update your Ingress resources, which will be covered in subsequent sections.

Implementing external-dns for Automated DNS Management

Managing DNS records manually for Kubernetes services and ingresses can be time-consuming and error-prone, especially as your applications grow. To streamline this process, we'll deploy external-dns, a tool that automatically manages external DNS records from within your Kubernetes cluster.

external-dns seamlessly integrates with your DNS provider, eliminating the need for manual configuration. It automatically updates DNS records whenever a service or ingress is created, modified, or deleted, ensuring your applications remain accessible without manual intervention.

1. Configure external-dns:
• In the Polar Labs Standard Helm Deployment repository, locate the Values.yaml file under _support-templates/external-dns/.
• Modify the values in this file to match your specific setup, paying close attention to the apiToken and other relevant parameters. Refer to the external-dns documentation for detailed explanations of these values.
2. Add the Bitnami Chart Repository:

helm repo add bitnami https://charts.bitnami.com/bitnami

3. Update Helm's Cache:

helm repo update

4. Install external-dns:

helm install external-dns bitnami/external-dns -f _support-templates/external-dns/Values.yaml

5. Verify Installation:

kubectl --namespace=default get pods -l "app.kubernetes.io/name=external-dns,app.kubernetes.io/instance=external-dns"

6. You should see a pod named external-dns in the Running state.
The verify command might change since this article was published. When you install external-dns it will tell you the command to verify.

Important considerations:

• Disabling external-dns: We have found that sometimes we don't want external-dns to manage our records for us, but there was no native way to selectively disable a specific ingress. We instead added our own little hack to our ingress template to allow for disabling external-dns for specific ingresses. In order to use it, simply add disableExternalDNS: true to your application's values.yaml file when using our Helm StarterKit. Refer to the pl-standard-chart/templates/_nginx-cert-manager-ingress.yaml file for annotations we used to achieve this if you are doing your own thing and want to know how we did it for your own implementation.
• Conflict with cert-manager: Be aware of a potential conflict between external-dns and cert-manager where external-dns attempts to create records for hosts defined in internal cert-manager resources and fails to do so, stopping the external-dns process early. To address this, our external-dns implementation is configured to only operate on ingresses that include the cert-manager annotation cert-manager.io/cluster-issuer. This is not present on internal cert-manager resources - it is only present on our own ingresses that need cert-manager's functionality. This hack ensures that external-dns only manages DNS records for ingresses you have explicitly configured and requested certificates for. This does however mean you must request a cert for an ingress to get external-dns support. This happens by default with our ingress file though. You can see the in the values.yaml file we used to install external-dns and you can see the annotation itself in our ingress template.

With external-dns in place, you can deploy and manage Kubernetes applications without worrying about manual DNS updates, simplifying your workflow and reducing the risk of human error.

Installing our Test Chart for verification

To ensure your Kubernetes cluster and all installed components are functioning correctly, we've included a test chart in the StarterKit repository. This chart deploys a simple "hello world" application that you can access in your browser to verify your configuration.

Before you begin:

1. Ensure that kubectl is connected to your Kubernetes cluster. You can use doctl to write your config file for you with doctl kubernetes cluster kubeconfig save use_your_cluster_name.
2. Verify that Helm can access your kubectl installation.

Installation Steps:

1. Open the values.yaml file located in the test-chart folder.
2. Edit the following values:
• registrySecretName: Replace with the name of your Docker registry secret.
• hosts: Update the domain names to match your own. You can use any subdomains you prefer.
3. Install the test chart using Helm:

helm install test-chart ./test-chart

Once installed, you should be able to access the "hello world" application in your browser by navigating to the domain names you specified in the hosts section of your values.yaml file. If the application loads successfully, it indicates that your cluster, ingress controller, cert-manager, and external-dns are working as expected.

And that's it! If everything went successfully, you can now use nginx-ingress-controller, cert-manager, and external-dns in your Digital Ocean Kubernetes cluster. You can also use our Helm StarterKit right out of our public repo https://helm.polarlabs.ca without even needing to fork the repo.

Happy Helming!

References

• https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes 
• https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-on-digitalocean-kubernetes-using-helm 
• https://www.shebanglabs.io/set-up-nginx-ingress-proxy-protocol-on-digitalocean/
• https://cert-manager.io/docs/installation/helm/
• https://www.digitalocean.com/community/tutorials/how-to-automatically-manage-dns-records-from-digitalocean-kubernetes-using-externaldns