Maximise Your Productivity: Harness Hot Reloading in Kubernetes

Episode #37: Accelerate Your Kubernetes Workflow with Hot Reloading. Master Fast Feedback Loops Using Tilt, K3d, and ttl.sh!

Hot reloading a K8s application. Image generated with Flux.pro AI model

What is hot reloading?

Developers can use hot reloading to instantly see changes in their code reflected in the running application without needing to restart or rebuild. This speeds up the development process and boosts productivity.

If you have ever programmed in React or any other front-end framework, you might have used hot-reloading before.

Hot reloading is less common in compiled programming languages like Golang, and it is definitively hard to achieve without Tilt when running applications in Kubernetes.

You might be asking:

Why would I want hot-reloading in Kubernetes when I can test my application locally without the extra complexity?

With the advent of cloud-native technologies, micro-services must interact with many other services like databases, APIs, distributed caches, etc.

If you want to develop in such a complex environment, you have a couple of options:

• use Testcontainers in your functional tests.

• use mocks and unit tests. This solution requires quite a large amount of code, and it is not for everyone.

• "develop in production" with hot-reloading. Among the other solutions, this is the only one that doesn't require to write extra code.

Beware, I'm not suggesting you shouldn't write functional or unit tests with mocks.

Hot reloading is exclusively for those instances where you want to test something quickly and don't want to mock the rest of the system architecture to test your changes.

This article will explain how you can achieve hot reloading of a Golang application running in Kubernetes with Tilt.

You can apply the insights you'll gain from this article to any compiled or interpreted programming language.

Also, you can extend this solution to using Kubernetes on any cloud provider, remote Docker registry and more complex applications with lots of different Kubernetes resources.

We have split this article into the following sections:

• K3d and ttl.sh

• What is Tilt?

• A Golang sample application

• Hot reloading with Tilt

  • Golang compilation

  • Docker build

  • Kubernetes apply and port-forwarding

K3d and ttl.sh

This article is primarily about Tilt, but I would like to introduce two other tools that can contribute to achieving a fast feedback loop: K3d and ttl.sh.

K3d is the easiest and fastest way to create a Kubernetes cluster on your machine. It's a more lightweight solution both in terms of CPU and memory consumption when compared to Minikube or Kind since it uses SQLite instead of Etcd to store the state of your Kubernetes cluster.

A lightweight Kubernetes cluster with K3d means I can fit more containers in my laptop memory.

I have already mentioned K3d in a past article Exploring Kubernetes Dev Environments in 2023.

Exploring Kubernetes Dev Environments in 2023

The other tool in this article is ttl.sh, dubbed on their website as an Anonymous & ephemeral Docker image registry.

From their website, we can read:

Anonymous: No login required. Image names provide the initial secrecy for access. Add a UUID to your image name to reduce discoverability.

Ephemeral: Image tags provide the time limit. The default is 24 hours, and the max is 24 hours (valid time tags :5m, :1600s, :4h, :1d)

I prefer ttl.sh to the default Docker registry when developing with Tilt. It avoids the extra complexity of the Docker login, and there is not need to delete intermediate images created by Tilt. Given its ephemeral nature, ttl.sh will delete those images automatically after 24h.

What is Tilt?

A great description of Tilt is provided on their official website.

Tilt automates all the steps from a code change to a new process: watching files, building container images, and bringing your environment up-to-date. Think docker build && kubectl apply or docker-compose up.

You might never hear about Tilt, but Tilt was acquired by Docker in May 2022.

Tilt implements its logic in a file called Tiltfile using syntax from a programming language called Starlark, a simplified version of Python designed for configuration files.

You don't have to be a Python programmer to create a Tiltfile. However, if you want to master more advanced features, it helps if you know Python programming basics.

To learn more about Tilt, check their documentation.

A Golang sample application

I have created a straightforward Golang application to show you how to implement hot reloading in Kubernetes.

Here is the Golang application.

package main

import (
	"os"

	"github.com/gin-gonic/gin"
)

func main() {
	r := gin.Default()

	r.GET("/", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Hello, World!",
		})
	})

	r.Run(":" + os.Getenv("PORT"))
}

If you want to skip ahead and look at the final solution, check out the GitHub repository at tilt-go-sample-app.

There is much more in that repository, which I am not explaining here. In particular, I use two tools: Taskfile and Devenv (UI for Nix).

If you want to learn more about those tools, make sure to check out the articles below:

Supercharge Your Automation Workflows with Taskfile: Say Goodbye to Makefile

Effortless Python Development with Nix

Hot reloading with Tilt

Below is the full code snippet of the Tiltfile I used to implement hot reloading in my application:

# -*- mode: Python -*-

load('ext://restart_process', 'docker_build_with_restart')

local_resource(
  'compile',
  'CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o build/hello cmd/hello.go',
  ignore=['build'],
  deps=['cmd/hello.go', 'go.mod', 'go.sum'])

docker_build_with_restart(
  'ttl.sh/tilt-go-sample-app',
  '.',
  entrypoint=['/app/hello'],
  dockerfile='deploy/docker/Dockerfile',
  only=[
    './build'
  ],
  live_update=[
    sync('./build', '/app')
  ]
)

k8s_yaml('deploy/kubernetes.dev/pod.yaml')

k8s_resource('tilt-go-sample-app', port_forwards=8085,
             resource_deps=['compile'])

When trying to replicate the results in this article, make sure you have the above Tiltfile and all the resources mentioned in this article from tilt-go-sample-app.

Then, you only need to run tilt up from the root folder where you have your Tiltfile.

There are three stages in this Tiltfile:

1. Golang compilation

2. Docker build

3. Kubernetes apply and Port forwarding

Let's analyse each of the stages in detail.

Golang compilation

Here it is the snippet that we are analysing in this section:

local_resource(
  'compile',
  'CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o build/hello cmd/hello.go',
  ignore=['build'],
  deps=['cmd/hello.go', 'go.mod', 'go.sum'])

To build the Golang application, you must define a local resource and assign it a name and a command to run.

Here, we used the name compile and the command CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o build/hello cmd/hello.go.

The command is executed by default from the folder where the Tiltfile is located.

We are compiling the Golang code to run on linux/amd64, the architecture we chose for our Docker base image.

Any changes to the file or directory paths specified in the deps array will kick off a rerun of the command. We only have one Golang file cmd/hello.go, and its dependencies are defined in go.mod and go.sum.

Docker build

Before explaining the Tiltfile command to build the Docker image, let's analyse the content of the Dockerfile.

FROM cgr.dev/chainguard/wolfi-base:latest@sha256:3490ac41510e17846b30c9ebfc4a323dfdecbd9a35e7b0e4e745a8f496a18f25

RUN mkdir /app

COPY build/ /app

CMD ["/app/hello"]

Here, we use a distroless container from Chainguard called wolfi-base, a minimal image with very little in it.

We have pinned both the tag and the digest of the base image to fully replicate the build in the future.

The rest is straightforward. We are copying the previous stage's output from the folder build to the path /app. We will use the entrypoint /app/hello to run our application from inside the container.

load('ext://restart_process', 'docker_build_with_restart')

docker_build_with_restart(
  'ttl.sh/tilt-go-sample-app',
  '.', # context = current directory
  entrypoint=['/app/hello'],
  dockerfile='deploy/docker/Dockerfile',
  only=[
    './build'
  ],
  live_update=[
    sync('./build', '/app')
  ]
)

The command docker_build_with_restart is loaded as an extension to the docker_build command. It provides a crucial part of the hot reloading functionality.

When the compilation stage changes something in the build folder (specified in the only parameter), this command will:

1. build a docker image called tilt-go-sample-app using the Dockerfile deploy/docker/Dockerfile with the entrypoint /app/hello.

2. copy that container image to the ttl.sh container registry at the location ttl.sh/tilt-go-sample-app

3. at any subsequent change to build, Tilt will skip creating that container image if it doesn't detect any changes in the Dockerfile. Instead, it will just live update the content of the ./build/ folder to /app/ inside that container. As you can imagine, this logic dramatically speeds up the development since it happens almost instantaneously for such a simple application.

Remember that the container image name ttl.sh/tilt-go-sample-app needs to match the container name in the Kubernetes manifest, explained in the next section.

Kubernetes apply and port-forwarding

Finally, the last stage in this Tiltfile involves deploying our Kubernetes resource and port-forwarding the port 8085 from the container to the developer machine.

The following snippet from the Tiltfile achieves exactly those things.

k8s_yaml('deploy/kubernetes.dev/pod.yaml')

k8s_resource('tilt-go-sample-app', port_forwards=8085,
             resource_deps=['compile'])

Remember that tilt-go-sample-app in the command k8s_resource is not the container name but the name of the Kubernetes resource. I have fallen in this trap quite a few times before figuring out what was wrong with my Tiltfile.

You can see that in the pod.yaml definition at metadata.name

apiVersion: v1
data:
  HOSTNAME: 0.0.0.0
  PORT: "8085"
kind: ConfigMap
metadata:
  name: app
  namespace: default
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: tilt-go-sample-app
  name: tilt-go-sample-app
spec:
  containers:
  - image: ttl.sh/tilt-go-sample-app:latest
    name: tilt-go-sample-app
    envFrom:
    - configMapRef:
        name: app
    ports:
    - containerPort: 8085
  dnsPolicy: ClusterFirst
  restartPolicy: Always

As you can see from above, the image name ttl.sh/tilt-go-sample-app matches the first parameter of docker_build_with_restart from the previous section.

This is the most basic Kubernetes manifest that you could create. Just a pod with a single container and a ConfigMap to pass the port to expose to the application.

As you can recall from the beginning of this article, this application takes an environment variable to configure the port at which the REST application is listening.

---

Are you ready to take your skills to new heights? 🚀

🚢 Let's embark on this journey together! 

👣 Follow me on LinkedIn and Twitter to receive valuable content on AI, Kubernetes, System Design, Elasticsearch, and much more. 

🎓 If you want personalised guidance, I am here to support you. Book a mentoring session with me at https://mentors.to/gsantoro on MentorCruise, and let's work together to unlock your full potential. 

♻️ Remember, sharing is caring! If this content has helped you, please re-share it with others so they can benefit from it. 

🤩 Let's inspire and empower each other to reach new heights!