Setting up Kubernetes for local development using k3d and Docker

Kubernetes has become the de facto choice for running workloads for most microservices. Hence, keeping parity with production, it’s wise to use Kubernetes for local development. This article uses k3d distributions of Kubernetes since it’s lightweight and runs on Docker. If you are confident running Kubernetes, there are other options like minikube and Docker Desktop.

I initially wrote this article for K3D version 3.x. The later version supports the local registry out of the box. If you want to set up the registry outside of K3D, follow along or skip to the K3D cluster setup.

Pre-requisites

Have the following software installed

• Docker for Mac / docker-ce
• k3d
• kubectl

Setting up local Docker registry

• On Mac, go to Docker for Mac preferences, go to the Docker Engine tab to add the following. If on Debian/Ubuntu, edit the /etc/docker/daemon.json file to do the same

{
	"debug": true,
	"experimental": false,
	"insecure-registries": [
		"registry.localhost:5000"
	],
	"registry-mirrors": [
		"https://registry-1.docker.io"
	]
}

Note: The Above enabled insecure registry, so you don’t have to have a signed certificate to access your local registry. If an image doesn’t exist in the local registry, then the request is forwarded to Docker Hub

• Update the host file

# Added for Kubernetes development
127.0.0.1 registry.localhost
::1       registry.localhost
#End of section

• Restart the Docker service for the above to take effect

• Create a local Docker registry

# Create a folder to hold all the images for the local registry, so if you move or update the registry, you don't have to build all the images 
$ mkdir -p $HOME/registry

# Create Docker registry
$ docker run -d --name registry.localhost --restart=always -v $HOME/registry:/var/lib/registry -p 5000:5000 registry

Setting up the k3d cluster

• As the Docker registry is outside of the k8s cluster, we need to make the k3d aware of the registry. Make K3D aware of the registry bypassing the details during cluster creation. Create the following file and save it at $HOME/k3d-registries.yaml

$ cat << EOF > $HOME/k3d-registries.yaml
mirrors:
	"registry.localhost:5000":
	   endpoint:
		 -  "http://registry.localhost:5000"
EOF

• Create the cluster with the name dev as the name

$ k3d cluster create dev --port 9000:80@loadbalancer -v $HOME/k3d-registries.yaml:/etc/rancher/k3s/registries.yaml 

Note: Do not use the relative paths for volume

The above command creates a node cluster named dev and adds a loadbalancer and an ingress controller, traefik. Loadbalancer sits outside of the cluster as a Docker container and routes traffic through to each node, in this case, just to one node. The above command also should add the k8s context to your kubeconfig. If not, follow the command outputs to do the same.

• Connect the local Docker registry network to k8s network so that Kubernetes can pull images from it. Look at network names using docker network ls

$ docker network connect k3d-dev registry.localhost

Note: Skip this step if the Docker registry is hosted externally and can be reached with the URL mentioned in $HOME/k3d-registries.yaml

Test the cluster and setup

Test your setup to prove you can successfully push to the registry, and your k8s cluster can pull from it.

# Pull the latest nginx image 
docker pull nginx:latest

# Tag the nginx image 
docker tag nginx:latest registry.localhost:5000/nginx:k8s

# Push the tagged image to your local Docker registry
docker push registry.localhost:5000/nginx:k8s

# Run a container on your local k8s cluster pulled from local docker registry
kubectl run nginx --image=registry.localhost:5000/nginx:k8s