Create a Development Cluster with k3d + k3s + rancher + kong gateway

Introduction

This document details the steps to setup k3s and rancher on a single node machine. Note that this is purely for development/testing purpose and should not be implemented in a production setting.

I have chose to include k3d (k3s on docker) here so as to give a more realistic development setup environment. k3d enables us to run multiple k3s cluster on the same machine, where a docker container is used to emulate a k3s node.

Instead of k3s’ default traefik ingress controller, kong is used. This is purely a matter of personal opinion. Kong API Gateway offers a richer set of plugins such as authentication, authorization, rate limiter etc, which I find to be more powerful and customization as compared to other available gateways.

On top of this, an existing standalone Nginx is also utilized because I am hosting other non-container applications on my home network.

A simple diagram of my setup is as follows:

Installation Steps:

1. Install k3d
   At this point of writing, v4.4.4 is the latest version. You can also check for other available version at the following url: https://github.com/rancher/k3d/releases

wget -q -O — https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG=v4.4.4 bash

2. Create your first cluster

sudo k3d cluster create hades-cluster — k3s-server-arg ‘ — no-deploy=traefik’

3. Install kong

sudo helm repo add kong https://charts.konghq.com 
sudo helm repo update
sudo helm install kong kong/kong — set ingressController.installCRDs=false — create-namespace -n kong

4. Install cert-manager

sudo helm repo add jetstack https://charts.jetstack.io
sudo helm repo update
sudo kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.crds.yaml
sudo helm install cert-manager jetstack/cert-manager — create-namespace — namespace cert-manager — version v1.3.1

5. Install rancher

sudo helm repo add rancher-latest https://releases.rancher.com/server-charts/latest 
sudo helm repo update
sudo helm install rancher rancher-latest/rancher — create-namespace — namespace cattle-system — set hostname=rancher.cchienhs.co — set ingress.extraAnnotations.’kubernetes\.io/ingress\.class’=kong

Additional Setup on Standalone Nginx

server { 
  listen 443 ssl http2; 
  listen [::]:443 ssl http2;
  
  # replace server_name with your own value
  server_name rancher.xxx.xxx;
  ssl_certificate   "path_to_ssl_certificate";
  ssl_certificate_key "path_to_ssl_certificate_key";
  client_max_body_size 1000M;
  location / {
    # redirect all http traffic to virtual host
    # Replace following ip and hostname with your own values      
    proxy_pass https://172.20.0.2:31893;
    proxy_set_header Host rancher.xxx.xxx;
    proxy_set_header X-Forwarded-For $remote_addr;
    # Websocket support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }
}