Our homelab is racked and our router gives the nodes an isolated network, with all their traffic going through the VPS. The machines now need to be configured: timezone, time synchronization, packages, container runtime, and more to come.

We could SSH into each node and type the same commands three times. It works on day one. A few months later, one node got a package the others don’t have, another has a config file edited by hand during a late-night debugging session, and nobody remembers what changed or why. Every problem starts with some archaeology.

Infrastructure as code solves this: we describe the state we want for our machines in text files, keep those files in git, and let a tool apply them. Every change is reviewed, versioned and repeatable. The tool we will use is Ansible.

What we will end up with:

• Ansible installed on the admin laptop, the only machine allowed to configure the nodes,
• a dedicated SSH key for Ansible,
• an inventory listing our nodes and their groups,
• the sudo password stored in an encrypted vault,
• two playbooks: one for time synchronization, one to install Docker,
• everything in a git repository.

Why Ansible?

There are many infrastructure as code tools. Ansible fits a small homelab well:

• Agentless: nothing to install on the nodes. Ansible only needs SSH and Python, which every Debian or Ubuntu server already has.
• Push model: we run a command from our laptop, and the changes are applied right away. No server to host, no daemon polling for changes.
• Readable: playbooks are YAML files that read almost like a checklist.
• Huge module library: packages, files, services, users, firewalls, and thousands more in community collections.

The key idea behind it is idempotence: running a playbook twice must give the same result as running it once. If a node is already in the desired state, Ansible changes nothing. This is what makes it safe to re-run a playbook on the whole fleet at any time.

The big picture

flowchart LR
    subgraph Admin[admin network 10.1.20.0/24]
        Laptop[Admin laptop<br/>Ansible + git repo]
    end
    Router[OpenWrt router]
    subgraph DMZ[dmz network 10.1.40.0/24]
        N1[node1<br/>10.1.40.140]
        N2[node2<br/>10.1.40.141]
        N3[node3<br/>10.1.40.142]
    end

    Laptop -- SSH --> Router
    Router -- SSH --> N1
    Router -- SSH --> N2
    Router -- SSH --> N3

The router’s firewall already allows the admin zone to reach the dmz zone, and not the other way around. That’s exactly what we need: the admin laptop is the control node, it pushes configuration to the nodes, and a compromised node can’t reach back to it.

The nodes have no internet access except through the WireGuard tunnel, so every package they download goes through the VPS. Ansible itself doesn’t care: it only needs SSH from the laptop.

Step 1: Install Ansible on the admin laptop

On a Debian or Ubuntu laptop:

sudo apt install -y ansible
ansible --version

The ansible package contains both ansible-core (the engine and the ansible.builtin modules) and the community collections, including community.general, which we will use for the timezone.

Step 2: A dedicated SSH key

We create a key used only by Ansible. It’s easy to revoke without touching our personal key, and its purpose is clear when we read a node’s authorized_keys:

ssh-keygen -t ed25519 -f ~/.ssh/ansible_ed25519 -C "ansible@homelab"

Then we copy it to each node:

for ip in 10.1.40.140 10.1.40.141 10.1.40.142; do
  ssh-copy-id -i ~/.ssh/ansible_ed25519.pub admin@$ip
done

ssh -i ~/.ssh/ansible_ed25519 admin@10.1.40.140 should now log in without a password.

Step 3: The repository layout

A minimal Ansible project fits in a handful of files, versioned in its own repository:

ansible/
  ansible.cfg                  default settings for this project
  inventory.ini                the machines and their groups
  group_vars/
    linux/
      vault.yaml               encrypted secrets for the "linux" group
  playbook/
    setup_timesyncd.yaml       timezone and NTP
    install_docker.yaml        Docker engine
    templates/
      timesyncd.conf.j2        config file template

ansible.cfg

Ansible reads ansible.cfg from the current directory, so every command run from ansible/ picks up these settings:

[defaults]
inventory = ./inventory.ini
private_key_file = ~/.ssh/ansible_ed25519
gathering = smart
callback_result_format = yaml

• inventory: no need to pass -i inventory.ini on every command.
• private_key_file: the dedicated key from Step 2.
• gathering = smart: Ansible collects facts about each host (OS, architecture, IP addresses…) once, and reuses them within the run instead of collecting them again for each play.
• callback_result_format = yaml: errors and results are printed as YAML instead of a long single line of JSON. Much easier to read.

inventory.ini

The inventory lists the machines Ansible manages, and sorts them into groups:

[controlplane]
10.1.40.140

[nodes]
10.1.40.141
10.1.40.142

[linux:children]
controlplane
nodes

[linux:vars]
ansible_user=admin
ansible_become=true

• controlplane and nodes split the machines by role. They don’t matter yet, but our cluster will need a node that manages the others.
• linux is a group of groups (:children): it contains all our machines. Playbooks that apply to the whole fleet target linux.
• [linux:vars] sets variables for every host in the group: the SSH user, and ansible_become=true to run tasks with sudo.

The IP addresses are the fixed ones the router hands out to each node through its static DHCP leases.

Step 4: Keep the sudo password in a vault

To use sudo, Ansible needs the user’s password. We don’t want to type it at each run, and we certainly don’t want it in plain text in git.

Ansible Vault encrypts a file with a password (AES-256). The encrypted file can be committed safely, and Ansible decrypts it in memory when it runs:

ansible-vault create group_vars/linux/vault.yaml

This opens an editor. We add one line:

ansible_become_password: <the sudo password of the admin user>

Once saved, the file on disk looks like this:

$ANSIBLE_VAULT;1.1;AES256
64623266343665633533346236393465...

Ansible automatically loads every file in group_vars/<group>/ for the hosts of that group, so all hosts in linux get ansible_become_password without any extra configuration.

To view or change it later:

ansible-vault view group_vars/linux/vault.yaml
ansible-vault edit group_vars/linux/vault.yaml

Step 5: First contact

Before writing any playbook, we check that Ansible can reach every node, with an ad hoc command (a single module run directly from the command line):

cd ansible/
ansible linux -m ping --ask-vault-pass

10.1.40.140 | SUCCESS => {"changed": false, "ping": "pong"}
10.1.40.141 | SUCCESS => {"changed": false, "ping": "pong"}
10.1.40.142 | SUCCESS => {"changed": false, "ping": "pong"}

The ping module is not an ICMP ping: it connects over SSH, runs Python on the node, and returns pong. A success means SSH, the key and Python all work.

Ad hoc commands are also handy to look at the whole fleet at once:

ansible linux -a "uptime" --ask-vault-pass
ansible linux -m ansible.builtin.setup -a "filter=ansible_distribution*" --ask-vault-pass

Step 6: A first playbook, time synchronization

A playbook describes the desired state of a group of machines, as an ordered list of tasks. Each task calls a module with parameters.

Clocks matter more than they seem in a cluster: TLS certificates, logs from several machines, and distributed systems that elect a leader all assume that nodes agree on the time. Our first playbook sets the timezone and configures systemd-timesyncd, the lightweight NTP client of systemd.

playbook/setup_timesyncd.yaml:

---
- name: Configure systemd-timesyncd
  hosts: linux
  become: true

  vars:
    timezone: Europe/Paris
    ntp_servers:
      - 0.pool.ntp.org
      - 1.pool.ntp.org

  tasks:

    - name: Set timezone
      community.general.timezone:
        name: "{{ timezone }}"

    - name: Install systemd-timesyncd
      ansible.builtin.apt:
        name: systemd-timesyncd
        state: present
        update_cache: true

    - name: Configure timesyncd
      ansible.builtin.template:
        src: timesyncd.conf.j2
        dest: /etc/systemd/timesyncd.conf
        owner: root
        group: root
        mode: "0644"
      notify: Restart timesyncd

    - name: Enable and start timesyncd
      ansible.builtin.systemd:
        name: systemd-timesyncd
        enabled: true
        state: started

  handlers:

    - name: Restart timesyncd
      ansible.builtin.systemd:
        name: systemd-timesyncd
        state: restarted

Notice that no task says how to do something. apt with state: present doesn’t mean “run apt install”, it means “this package must be installed”. If it already is, the task reports ok and does nothing.

Templates

The template module renders a Jinja2 file with our variables, and copies the result to the node. playbook/templates/timesyncd.conf.j2:

# Managed by Ansible

[Time]
NTP={% for server in ntp_servers %}{{ server }} {% endfor %}
FallbackNTP=2.pool.ntp.org 3.pool.ntp.org
RootDistanceMaxSec=5
PollIntervalMinSec=32
PollIntervalMaxSec=2048

The # Managed by Ansible header is a small but useful habit: anyone opening the file on a node knows that a manual edit will be overwritten on the next run, and where to make the change instead.

Handlers

A handler is a task that only runs when another task notifies it, and only if that task actually changed something. Here, timesyncd is restarted when its config file changes, and left alone otherwise. Handlers run once at the end of the play, even if several tasks notify them.

Run it

First a dry run. --check reports what would change without changing anything, and --diff shows the content of the changes:

ansible-playbook playbook/setup_timesyncd.yaml --ask-vault-pass --check --diff

Then for real:

ansible-playbook playbook/setup_timesyncd.yaml --ask-vault-pass

PLAY RECAP ****************************************************************
10.1.40.140 : ok=6  changed=3  unreachable=0  failed=0  skipped=0
10.1.40.141 : ok=6  changed=3  unreachable=0  failed=0  skipped=0
10.1.40.142 : ok=6  changed=3  unreachable=0  failed=0  skipped=0

And now the most important test, we run it a second time:

10.1.40.140 : ok=5  changed=0  unreachable=0  failed=0  skipped=0
10.1.40.141 : ok=5  changed=0  unreachable=0  failed=0  skipped=0
10.1.40.142 : ok=5  changed=0  unreachable=0  failed=0  skipped=0

changed=0: the nodes were already in the desired state, so nothing was touched, and the handler didn’t run. That’s idempotence.

On a node, timedatectl timesync-status shows the server in use and the clock offset.

Step 7: Installing Docker, and the trap of shell commands

Our cluster will run containers, so every node needs a container runtime. The quickest way to install Docker is its convenience script. Ansible can run it with the shell module:

playbook/install_docker.yaml:

- name: Install Docker
  hosts: linux
  become: true
  tasks:
    - name: Install prerequisites
      ansible.builtin.package:
        name: [curl, ca-certificates]
        state: present

    - name: Install Docker with the convenience script
      ansible.builtin.shell: curl -fsSL https://get.docker.com | sh
      args:
        creates: /usr/bin/docker

shell and command are the escape hatches of Ansible: they run anything, but Ansible has no idea what the command does, so it can’t know whether the machine is already in the right state. Without help, it would download and run the script on every run, and always report changed.

The creates argument gives it that help: if /usr/bin/docker already exists, the task is skipped. It makes the task idempotent, but only partly. Ansible knows the binary exists, not which version it is, nor whether the Docker repository is configured correctly.

ansible-playbook playbook/install_docker.yaml --ask-vault-pass
ansible linux -a "docker --version" --ask-vault-pass

Step 8: Commit it

The whole configuration is now plain text, so it goes into git like any other code:

git add ansible/
git status        # vault.yaml must be the encrypted version
git commit -m "ansible: inventory, vault, timesyncd and docker playbooks"

From now on, the rule is simple: no manual change on the nodes. If a node needs something, we write it in a playbook, review the diff, commit, and run it on the whole fleet. The git history becomes the history of our infrastructure.

The final checklist

• Ansible installed on the admin laptop
• Dedicated ansible_ed25519 key copied to every node
• ansible linux -m ping returns pong for every node
• Sudo password in an encrypted vault, the vault password in a password manager
• Timezone and NTP configured on every node, second run reports changed=0
• Docker installed on every node
• Everything committed in git, with no secret in plain text

Our three nodes are now configured identically, from one place, and we can rebuild that configuration on a fresh machine with a single command. In the next articles, we will use the same approach to turn them into our first orchestrated cluster.

Where this post fits

This post, its tags, and everything they connect to. Open the full graph →

posttag

+−reset

cd ../blog