Over the last articles, we prepared everything a cluster needs. We have something to orchestrate (a containerized GitHub runner), a homelab with three ThinkCentre machines, a router that isolates them in their own network, Ubuntu Server installed on each of them, and Ansible to configure them all from one place. Time to turn these three machines into our first orchestrated cluster.

This is also what I do every day as the “DevOps guy” at work: deploying tools and applications for the development team, such as Jenkins, GitHub runners, a LAVA web instance, and dashboard or monitoring tools like Grafana and Prometheus.

If you have access to multiple machines or VMs, Kubernetes is often the right tool for the job. And if your needs are not too specific, its lightweight distribution, K3s, is usually a great fit.

Why K3s specifically? It keeps the Kubernetes experience with a smaller footprint and a very simple install process. It also bundles useful components out of the box, which makes it ideal for homelabs like ours, edge environments, and small to medium internal platforms such as a company lab.

What we will end up with:

• the key Kubernetes concepts, explained in a few lines each,
• a K3s cluster on our three homelab nodes: one control plane and two agents,
• a handful of kubectl commands to find our way around the cluster,
• the whole setup as an Ansible playbook, versioned with the rest of our infrastructure.

What is Kubernetes in a nutshell?

Instead of running an application directly on a bare machine, in the dev world we usually use a container, which creates a “fake computer environment” isolated from our system.

We do that for several reasons:

• Isolation (containing the app inside a “box” to limit contamination, corruption or destruction of our system. Containers share the host kernel though, so they are not a strong security boundary on their own).
• Disposability (we can trash this “box” and make a new one on the spot).
• Scalability (we can deploy one or many identical “boxes” on demand).
• Consistency (we can consistently reproduce an identical dev environment in that “box” on any machine, anywhere, anytime).

For that, we usually use Docker. Then comes Docker Compose which allows us to define, run and manage several containers together on a single machine, making an “app cluster” locally, connecting them through networks, volumes, or devices.

But when you want to change scale and do that with multiple machines at the same time, here comes Kubernetes. A useful way to picture it is as an evolution from the same ideas as Docker Compose, but at a larger scale, with orchestration, scheduling, recovery, and networking built in.

In one sentence: Kubernetes is an automatic app babysitter and traffic manager that keeps apps running, healthy, and scalable across many computers.

Key concepts of Kubernetes

Here are some of the main concepts to know in Kubernetes:

On infrastructure:

• Cluster: A fleet of computers working together.
• Node: One computer in that fleet.
• Control Plane: The brain of Kubernetes, a set of components (API server, scheduler, controllers, datastore) that manages deployment and orchestration. In K3s, it runs on a server node, which also runs workloads by default; the other nodes are agents.

On resources:

• Namespace: a logical partition inside a cluster used to organize and isolate resources.
• Pod: the smallest deployable unit, a group of one or more containers that share the same network and storage (usually one app per Pod), deployed on nodes.
• Service: an abstraction that exposes one or more Pods as a stable network endpoint.
• Secret: a Kubernetes object used to store sensitive data such as tokens, passwords, or keys.
• Ingress: an object that manages external HTTP/HTTPS access and routes traffic to Services.
• ConfigMap: a Kubernetes object used to store non-sensitive configuration values.
• PersistentVolume (PV): a storage resource in the cluster used to keep data beyond Pod lifecycle.

On operation:

• Deployment: an object that declares how many identical Pods of an app should run, and manages their lifecycle (through ReplicaSets), including creation, updates, and scaling.
• Desired State: The target configuration declared by the user that Kubernetes continuously tries to maintain.
• Self-healing: Kubernetes’ ability to automatically replace or restart failed containers and pods to keep applications running.
• Scaling: The process of adjusting the number of running application instances up or down based on demand.
• Rolling Update: A strategy that gradually replaces old application versions with new ones to minimize downtime.

Real-life workflow example: I deploy an app as a Deployment, expose it internally with a Service, store credentials in a Secret, inject app settings with a ConfigMap, keep data with a PersistentVolume, and publish it externally through an Ingress.

Small architecture diagram

flowchart LR
    User((User / CI)) --> Ingress[Ingress]
    Ingress --> Service[Service]
    subgraph Cluster[K3s Cluster]
        CP[Control Plane]
        subgraph W1[Worker Node 1]
            Pod1[Pod A]
        end
        subgraph W2[Worker Node 2]
            Pod2[Pod B]
        end
    end
    Service --> Pod1
    Service --> Pod2
    CP -. schedules .-> Pod1
    CP -. schedules .-> Pod2

K3s cluster setup

To set up a K3s cluster, we need to list the machines we want to add to the cluster and have proper access to them (login and admin rights). We’ll need to set one (or more) of them as a control plane (the strategic center of the cluster) to centralize information and manage the other machines set as nodes.

On all machines, we need to run:

# Install dependencies
sudo apt install -y curl vim gnupg lsb-release ca-certificates
# Disable all active swap spaces (recommended: it keeps the memory behavior of the nodes predictable)
sudo swapoff -a
# Disable swap at boot
sudo sed -i '/ swap / s/^/#/' /etc/fstab

On the control plane:

# Install k3s
curl -sfL https://get.k3s.io | sh -s - server --write-kubeconfig-mode 644
# Verify with k3s command to get the state of nodes
sudo k3s kubectl get nodes
# Get the control plane token NODE_TOKEN
sudo cat /var/lib/rancher/k3s/server/node-token

We also need the control plane IP, MASTER_IP: the address the worker nodes use to reach it. Be careful with hostname -I, it lists all the addresses of the machine, including the ones of Docker (172.17.0.1) and, once the cluster runs, of the K3s internal network. Use the address of the machine on your LAN, ideally a fixed one (a static DHCP lease on your router, for instance).

On worker nodes:

# Set MASTER_IP and NODE_TOKEN with the values from the control plane
export MASTER_IP=<control plane IP>
export NODE_TOKEN=<control plane token>
# Install the K3s agent
curl -sfL https://get.k3s.io | K3S_URL="https://${MASTER_IP}:6443" K3S_TOKEN="${NODE_TOKEN}" sh -s - agent

Useful commands

Now that the cluster is up and running, here are a few useful commands to navigate in K3s. First, kubectl is the main command-line tool used to interact with the cluster from the control plane.

Some common actions are:

• get: list one or more Kubernetes objects.
• describe: show detailed information about a specific object.
• create: create a new object.
• delete: delete an existing object.

Here are a few useful examples:

• get nodes: kubectl get nodes
• get namespaces: kubectl get namespaces
• create a namespace: kubectl create namespace <mynamespace>
• create a secret in a namespace from a .env file: kubectl create secret generic <mysecret> --from-env-file=<.env> -n <mynamespace>
• describe a Pod: kubectl describe pod <mypod> -n <mynamespace>
• open an interactive shell inside a Pod: kubectl exec -it -n <mynamespace> <mypod> -- bash
• list Pods in a namespace with their status: kubectl get pods -n <mynamespace> -o wide
• run a debug Pod: kubectl run <myapp-debug> -n <mynamespace> --rm -it --image=<mydockerimage> --command -- sh
• view Pod logs: kubectl logs -n <mynamespace> <mypod>
• check node CPU and memory usage: kubectl top nodes

Ansible alternative: the full playbook

Typing the same commands on every machine works for a first try, but it’s exactly the kind of manual change our Ansible article tells us to avoid. Here is the whole setup as a single playbook, reusing the inventory of that article: the controlplane group gets the K3s server, the nodes group gets the agents, and linux contains them all.

playbook/install_k3s.yaml:

---
- name: Prepare the nodes for K3s
  hosts: linux
  become: true
  tags: prep
  tasks:
    - name: Install curl
      ansible.builtin.apt:
        name: curl
        state: present
        update_cache: true

    - name: Disable active swap
      ansible.builtin.command: swapoff -a
      when: ansible_swaptotal_mb > 0

    - name: Disable swap at boot
      ansible.builtin.replace:
        path: /etc/fstab
        regexp: '^([^#].*\sswap\s.*)$'
        replace: '# \1'

- name: Install the K3s server
  hosts: controlplane
  become: true
  tasks:
    - name: Install K3s server
      ansible.builtin.shell: curl -sfL https://get.k3s.io | sh -s - server --write-kubeconfig-mode 644
      args:
        creates: /usr/local/bin/k3s

    - name: Read the node token
      ansible.builtin.slurp:
        src: /var/lib/rancher/k3s/server/node-token
      register: node_token

- name: Join the agents
  hosts: nodes
  become: true
  vars:
    k3s_server: "{{ groups['controlplane'][0] }}"
  tasks:
    - name: Install K3s agent
      ansible.builtin.shell: curl -sfL https://get.k3s.io | sh -s - agent
      environment:
        K3S_URL: "https://{{ k3s_server }}:6443"
        K3S_TOKEN: "{{ hostvars[k3s_server].node_token.content | b64decode | trim }}"
      args:
        creates: /usr/local/bin/k3s
      no_log: true

A few things worth noticing:

• Swap is only turned off when the node actually has some (ansible_swaptotal_mb is a fact gathered by Ansible), and the replace module comments out the swap line of /etc/fstab only once, so a second run reports changed=0.
• The token never leaves Ansible’s memory: slurp reads it on the control plane, and the agents get it from hostvars. no_log: true keeps it out of the terminal output.
• The control plane IP comes from the inventory (groups['controlplane'][0]), so there is no address to look up by hand.
• The install scripts are run with shell and creates, with the same limits as the Docker install of the Ansible article: Ansible knows K3s is installed, not which version.

Then, as usual, a dry run first and the real run after:

ansible-playbook playbook/install_k3s.yaml --ask-vault-pass --check --diff --tags prep
ansible-playbook playbook/install_k3s.yaml --ask-vault-pass
ansible controlplane -a "k3s kubectl get nodes" --ask-vault-pass

Next steps

This post was an introduction to K3s core concepts and I hope it was useful. In the next blog articles I am going to talk more about actual deployment use cases, such as GitHub runners over a fleet, with some additional tools like ArgoCD. Stay tuned! 🚀

Where this post fits

This post, its tags, and everything they connect to. Open the full graph →

posttag

+−reset

cd ../blog