The homelab is a few second-hand machines in my apartment that I use to learn platform engineering. It runs apps I actually use: static knowledge sites, small web apps with their own databases, and a few GPU model servers, which the ML orchestration writeup covers.
Apps ship through GitOps. CI builds an image, a controller commits the new image tag to a git repository, and Argo CD applies that repository to a k3s cluster. The cluster went up in June as a way to learn GitOps, Helm and Kubernetes, and the lab doesn't strictly need Kubernetes. The apps started out on Dokploy, a Docker Swarm PaaS, and most of them have since moved to the cluster. The migration gets its own writeup.
The shape of it
Three machines hold the lab. A NAS host holds the durable data: Postgres, the Garage S3 object store, and NFS shares. A mini-PC runs the always-on services as LXC containers: the Git forge, the Caddy proxy, DNS, SigNoz for telemetry, and the Redpanda event broker. A GPU tower carries the k3s cluster and the CI server.
The cluster has two nodes, both on the tower. The server is a VM that runs the control plane and ordinary workloads. The agent is an unprivileged LXC container, which lets the tower's GPU be shared between it and the tower's other containers. A VM would need the card passed through, and passthrough hands the whole card to that one VM. The catch is that an unprivileged LXC can't mount NFS or raise some kernel socket limits, and both of those caused problems described below.
Harbor, the container registry, runs in the cluster but keeps its image layers in Garage and its metadata in the NAS Postgres. Velero writes backups to Garage, and the OpenTelemetry collector exports to SigNoz on the mini-PC. Keeping that state off the cluster is deliberate, so losing the cluster doesn't lose it. The control-plane VM has moved hosts once, by backup and restore, and every app came back the same as before the move.
Terraform creates every VM and container, and Ansible configures them. Ansible also installs k3s and Argo CD, once. After that, everything inside the cluster comes from one git repository.
From push to running
Build. A push to an app's main branch on the forge triggers Woodpecker,
which builds the image with buildx, tags it from the commit, and pushes it to
Harbor using a robot account that only CI uses. Most pipelines tag images
main-<sha8>, and older ones use the bare hash. Both are immutable. Some
pipelines also push a moving prod-latest tag, which nothing deploys.
Registry. Harbor scans each image as it arrives. It also acts as a pull-through cache for Docker Hub and GitHub's registry, so apps' third-party images come through it too.
Tag to git. Argo CD Image Updater polls Harbor every couple of minutes, using a pull-only robot account, so the controller that rewrites tags can't push or delete an image. Each app it watches has annotations on its Application that say which image to follow, which tags count, and where to write the new tag:
# apps/<app>.yaml (abridged)
metadata:
annotations:
argocd-image-updater.argoproj.io/image-list: app=<registry>/fleet/<app>
argocd-image-updater.argoproj.io/app.update-strategy: newest-build
argocd-image-updater.argoproj.io/app.allow-tags: regexp:^main-[0-9a-f]{8}$
argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/<write-credentials>
argocd-image-updater.argoproj.io/write-back-target: helmvalues:values.yaml
spec:
source:
path: charts/<app>Commit hashes have no version order, so the strategy is newest-build. The tag
regex only matches immutable tags, which keeps prod-latest out. When a newer
build shows up, the controller commits the new tag to charts/<app>/values.yaml
on the GitOps repo's main branch, with a message like build: automatic update of <app>. The write-back path is relative to the chart's directory, which the
first configuration got wrong: the very first automated commit created
charts/<app>/charts/<app>/values.yaml. The controller is pinned to the 0.14
line because the 1.x rewrite ignores these annotations without any error.
Sync. Argo CD polls the repo every three minutes, sees the new tag, renders the chart with Helm, and applies the difference. There's no webhook from the forge to Argo CD yet, so a push takes a few minutes to reach the cluster.
Serve. Caddy holds a wildcard for the cluster's hostnames and forwards to
Traefik on a fixed node port, and Traefik routes by Host header to the app's
Service. Caddy is the only TLS hop. Internal .lab names use certificates from
Caddy's own certificate authority, and public names get Let's Encrypt
certificates through a Cloudflare DNS-01 challenge. The public DNS records point
at Caddy's tailnet address, so a public name resolves anywhere but only connects
from the tailnet or the home LAN. No port is forwarded to the lab. A new app on
a cluster hostname needs no DNS, certificate or proxy change.
CI's only job is to build and push the image. Deciding what runs happens on the
cluster side, by a controller that reads the registry and writes git. That makes
the git history of each app's values.yaml its deploy log. For an app the
controller watches, every deploy is a commit naming the old and new tag, and
rolling back is a git revert. Pinned third-party images, such as the model
servers and Harbor itself, are updated by ordinary reviewed commits.
One repository for the cluster
The bootstrap applies one root, which lists two Argo CD Applications, infra
and apps. Each one recurses a directory of child Applications, with pruning
and self-heal on. Adding a platform component means adding a file to infra/,
and adding an app means adding a file to apps/. The root itself never changes.
Upstream charts come in as native Helm sources pinned to an exact chart version,
never a range or HEAD. Charts for the lab's own apps live in charts/<app>/,
outside the recursed apps/ directory. If they were inside it, the app-of-apps
would apply the chart's raw templates as manifests. The CI gate checks that
every top-level file in apps/ and infra/ is an Application. Encrypted
secrets, Kyverno policies and namespaces sit outside the recursed directories
for the same reason.
Many app charts are thin wrappers around three shared charts: one for static
sites, one for a stateful web service with an optional migration Job, volume and
service account, and one for a batch job that's suspended by default. The shared
charts are ordinary application charts on purpose. Image Updater writes to a
values path like static-spa.image.tag, and in a Helm library chart no template
would read that key. The app would keep running its old image while git showed
the new tag.
Inside a chart, sync waves set the order. One stateful app brings up its database pods first, then runs its migration Job, then starts the app. The migration runs as a Sync hook, because a PreSync hook deadlocks on a first install, and the CI gate pins the hook phase so it can't drift back. A lint also keeps every Ingress in its chart's last wave, for a reason covered below.
A new app needs an Application with its Image Updater annotations, a wrapper
chart, its secrets, its namespace added to the policies, and a CI pipeline in
its own repo. An onboarding command generates the chart, the Application and the
routing entry. It never commits, and it only creates files that don't exist yet
unless it's forced. That protects values.yaml, which it shares with Image
Updater: regenerating that file would replace the live tag with whatever the
registry returned at that moment, so onboarding an app a second time could roll
it back. Its test checks that the generated Application and Chart.yaml match a
hand-written pilot byte for byte, and that a run against a repo that already has
the pilot changes nothing.
Secrets in a repository anyone could read
The GitOps repo is treated as readable by anyone, so it never holds a plaintext
credential. Secrets are committed encrypted with SOPS to an age key. Only the
data and stringData fields are encrypted, so the metadata stays readable and
a diff still shows which secret changed. A plugin sidecar on Argo CD's repo
server decrypts each *.sops.yaml file as it renders the Application.
The master copy of the age private key lives in the infrastructure repo's
Ansible Vault, and the bootstrap installs it in the cluster as a Secret. That
key and the credential Argo CD uses to clone the repo are the only two things
the bootstrap installs directly, since both have to exist before Argo CD can
read the repo. The command that writes an app's secret file renders the values
from the vault in memory and pipes them straight into sops --encrypt, so
plaintext never touches the disk.
Sealed Secrets was ruled out because it adds a second, Kubernetes-only root of trust. The External Secrets Operator, backed by Vault or Infisical, was ruled out because it's another service to run, monitor and back up. The cost of the current setup is that rotating a secret means updating it in two places, and there's no runbook for that yet.
Checks before and after a commit
The CI gate. Every push and pull request to the GitOps repo runs a Woodpecker pipeline with Helm, the Kyverno CLI and yq pinned by version and checksum. Helm stays on 3.x because that's what Argo CD renders with, and a gate on a different major version could pass output that never ships. The gate runs in CI against a commit, so what it checked is exactly what's in git.
The main step renders every chart, every values overlay and every shared-chart fixture, then:
- renders each one twice and compares the bytes. This was added after Harbor's chart turned out to generate new random secrets on every render, which restarted Harbor on unrelated commits;
- runs the Kyverno policies against the output, taking what blocks from each policy's own declared action;
- fails on a
:latestor untagged image, on missing resource requests, and on a credential embedded in a URL, masking the match before it prints to the log; - keeps every Ingress in its chart's last sync wave;
- requires a set of deliberately broken fixtures to fail, to show the gate actually catches what it's meant to.
Image Updater's commits and small bookkeeping commits go straight to main, so
for those the gate runs after Argo CD may already have applied the change.
Admission control in the cluster covers that gap.
Admission. Kyverno runs five cluster policies. Three of them audit cluster-wide and enforce in the app namespaces they list:
- an image must carry a tag, and the tag can't be
latest; - a container must set CPU and memory requests and a memory limit. CPU limits are left out on purpose, to avoid throttling;
- images must come from a short list of approved registries. This one switched to enforcing once Harbor's pull-through caches gave every app image an approved source.
A fourth audits pod probes without enforcing anything. The fifth enforces in the GPU workflow namespace: a pod there that requests a GPU must carry the Kueue queue label and the annotation that the approved GPU step templates add.
Events without dual writes
Services publish domain events, like a device going offline, through a transactional outbox. A service writes its business change and a row in an outbox table in the same Postgres transaction, and it never talks to the broker. Both rows commit or neither does, so an event can't go out for a change that rolled back, or go missing because a second write failed. Apps hold no broker credentials, and their CI doesn't need to know the broker exists.
Change data capture handles the rest. The shared Postgres runs with logical replication on. Each tenant, meaning one project in one environment, gets its own replication role, a publication covering its outbox table, and its own replication slot, so one tenant's problems stay with that tenant. A Debezium connector per tenant, running in Kafka Connect, reads that slot and publishes each new outbox row to Redpanda.
Debezium's EventRouter routes each row to a topic named
<env>.<project>.<aggregate>, keyed by the aggregate's id, which keeps the
events for any one aggregate in order. A device lifecycle event from the device
fleet lands on prod.embedded.Device. The router strips Debezium's change-data
envelope, and the connectors now emit the payload as a JSON object, so new
records decode once. The conventional outbox table looks like this:
CREATE TABLE outbox (
id BIGSERIAL PRIMARY KEY, -- becomes the record's id header
aggregate_type TEXT NOT NULL, -- picks the topic
aggregate_id TEXT NOT NULL, -- becomes the message key
event_type TEXT NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);Tenants can rename the table and pick their own route-by, key and payload columns, and some use UUID ids, which the event envelope below expects. One app already kept an append-only command log, so that log is its outbox, without a second write per command.
On a shared Postgres, a quiet tenant is a hazard. Its replication slot holds
the write-ahead log at its last read position while busier tenants keep
writing, and the log grows until the disk fills. Each tenant's publication
includes a small debezium_heartbeat table that the connector upserts every
five minutes, which keeps the slot moving.
Once a tenant's connector is running, every committed event gets published, possibly more than once, and ordering holds per aggregate. Consumers are expected to dedupe on the event id. Postgres stays the system of record, and the broker is a single Redpanda node.
Adding a producer starts with a row in a contracts file, with its credentials scaffolded into the vault. Then three idempotent deploys run in order: one creates the broker user and its prefix ACLs, one creates the replication role and publication, and one registers the connector once the publication exists. All three depend on the app's outbox table. Until the app's own migration has created it, they report success and create nothing. The contract row has to list the producer's aggregates, and a lint checks the list is well-formed, so the row shows a producer's topics without anyone reading its code. A topic is named after its aggregate, so renaming an aggregate moves its events to a new topic, and the old one stops receiving them.
Each event's payload is meant to follow a shared envelope: a schema version; an
event id, a UUID that should equal the outbox row's id; when the producer
recorded the event; the aggregate's type and id; the event's type and data; and
optional user, correlation and causation ids. A JSON Schema backs it, with
fixtures for the dev producers, and it rejects unknown top-level fields, so
adding one means changing the contract. The version lives in the event type's
name, as in …recorded.v1. An additive change keeps the name, and a breaking
change becomes .v2, which can share a topic with .v1 while consumers move
over. The envelope came out of finding that three producers had hand-copied one
shape and three others emitted something different. No producer conforms to it
yet, and moving them onto it is the next step.
In production, the device fleet publishes lifecycle events. The check was
rebooting a bench board, which put Device.offline and Device.online on the
log. Several personal apps publish in their dev environments. The devices
themselves only speak MQTT, and their broker has two subscribers: Telegraf
forwards metrics to SigNoz, and a fleet-consumer service writes device state to
Postgres. Because that write goes through the outbox, each lifecycle change also
becomes an event.
Seeing the system
The OpenTelemetry collector runs in the cluster from SigNoz's Kubernetes chart and sends cluster and kubelet metrics, Kubernetes events, container logs and host metrics to SigNoz, which runs outside the cluster. A scrape configuration adds the model servers' metrics. Alert rules live in the repo as JSON, including alerts on control-plane memory and on Kueue's availability that came out of one of the incidents below. Argo CD's own notifications aren't set up yet.
Canaries and backups
Argo Rollouts and Velero are both installed, and neither has been exercised much yet.
One demo app runs as a canary. Its rollout steps through 25, 50 and 75 percent with a minute's pause at each, then goes to 100. An analysis Job queries SigNoz for 5xx lines in the app's logs and aborts the rollout above a threshold. The real apps still roll out as plain Deployments. The analysis also fails open right now: if SigNoz is unreachable, the query counts zero errors and the canary continues. That has to be fixed before a real app depends on it.
A nightly Velero schedule backs up to Garage on the NAS and keeps seven days. The NFS storage class has no snapshot support, so volumes are backed up at the file level with Kopia. The schedule is itself in the GitOps repo. A restore has been tested once: a small stamp file on a demo volume, the namespace deleted, and the stamp restored intact into a new volume. No real app has been restored yet. High availability isn't a goal, and the k3s datastore is a single SQLite server.
What broke
Each of these has a write-up in the lab's repos, written the same day or within a few days.
Image pulls failed for a day and a half
Every image pull through Harbor started returning 502, while Harbor itself was healthy. The ticket assumed the failing node had too many iptables rules, but the healthy VM node turned out to have more. Once the right thing was measured, the diagnosis took minutes.
The problem was on the LXC node. iptables-nft sends its whole ruleset as one
netlink message and grows the socket buffer to fit using SO_SNDBUFFORCE, which
needs CAP_NET_ADMIN in the initial user namespace. An unprivileged container
can never have that. Once the ruleset outgrew the default buffer, kube-proxy
stopped writing rules, and the node kept routing with its last good set.
Harbor's internal hop pointed at a pod IP that no longer existed.
The fix raises the default socket buffer size on the LXC host, in code. The first attempt raised the maximum instead and made no difference. It was left in the history on purpose.
Argo CD never marked an app with an Ingress Healthy
For a long stretch, every app with an Ingress showed as Progressing in Argo CD and never reached Healthy. Traefik copies load-balancer status onto Ingresses from its published Service, and a NodePort Service has none. Setting the ingress endpoint's hostname explicitly gave the Ingresses a status, and Argo CD started reporting those apps Healthy.
Before the cause was found, one app never started a pod at all, because its Deployment sat in a later sync wave behind an Ingress that could never become Healthy. That's where the lint keeping Ingresses in the last wave came from. It stayed after the root cause was fixed.
Moved apps returned 404 on old bookmarks
Moving an app from Dokploy to the cluster spans two repos and two moments. The GitOps change gives the app an Ingress when it merges, and Caddy starts sending the app's old hostname to Traefik the next time its playbook runs. Traefik only answers for hostnames an Ingress claims. The first moved app's chart only claimed its new cluster hostname, so once Caddy re-rendered, bookmarks on the old hostname reached a Traefik with no route for them and got a 404. The onboarding command now puts both hostnames in every Ingress.
A curl returning 200 didn't help here, because it can't tell which backend
answered. The check that works is to request a path unique to the new deployment
and look for it in the pod's log.
A database pod hung on the node that can't mount NFS
A Postgres pod sat stuck for about eight hours. Removing a taint had let it
schedule onto the LXC node, where mounting NFS needs a capability an
unprivileged container can't have. Workloads that need an NFS volume now select
nodes by a homelab/nfs-capable capability label.
Pods could resolve .lab names only some of the time
While Harbor was first being set up, it couldn't reliably reach the database or
the object store by their internal .lab names, even though the same lookup
from the node answered correctly every time. The failures looked tied to
particular pods and nodes, and weren't. The node's resolver config listed both
Pi-hole and a public resolver, and CoreDNS forwards to either. The public one
answers "no such domain" for .lab, and CoreDNS cached that answer for 30
seconds. A small lab:53 server block in a coredns-custom ConfigMap now sends
.lab lookups only to Pi-hole, without touching the Corefile that k3s manages.
A find-and-replace sent Argo CD to clone a container registry
When the Applications were repointed at Harbor, a bulk find-and-replace across
every Application file also rewrote four of their repoURL fields, so Argo CD
tried to read its git source from a container registry. The error looked like a
passing git fault, but it survived a restart of the repo server. The giveaway
was the one Application that stayed in sync: its repoint had been done by hand.
Restoring the four repoURL values fixed it.
A pipeline check reported an empty topic that had a message in it
The first real run of a new producer's verification script reported that its topic held no consumable message. The message was there. Two parsing bugs were stacked: the Redpanda CLI pretty-prints a record across many lines, and the record's body was JSON inside JSON. The script's self-test had passed the whole time, because its fixture was hand-written and only encoded once. The fixture is now the real command's output, captured verbatim. The fixed check's first passing run printed two topics: its filter also matched the heartbeat topic, and had only picked the right one because of how the names sorted.
Some event payloads were encoded three times
The contracts file said a consumer would need to decode each message twice, since the payload's JSON column came through the connector as a string. That was an inference. Measuring real records showed some needed three decodes. One producer was serializing its payload to a string before writing it into its own JSON column, and none of that app's tests caught it. A one-line query on the column's type now settles the question, and the connectors were later changed to emit the payload as a JSON object, which removed the pipeline's own extra layer.
The control plane ran out of memory, and pod creation stopped
The server VM had 8 GB and no swap, and it ran the API server, its datastore and Argo CD with all of its Applications. During a drill, Argo CD's application controller grew from 833 MB to 1.2 GB in thirteen minutes and the node started thrashing. The API server slowed down but stayed up.
Kueue, the batch scheduler for the GPU work, ran on the other node with plenty of memory, but it couldn't start against the struggling API server, so its liveness probe kept killing it. Kueue admits pods through webhooks that fail closed, and while it was down no new pod could be created in its namespaces, including the drill's own cleanup step. Restarting the application controller recovered the node within a minute.
The follow-ups all closed within two days. The VM moved to the tower with 16 GB, Argo CD got memory limits, routine hard refreshes stopped, memory alerts went in, a cleanup path that doesn't depend on admission was added, and Kueue's pod webhooks were scoped to pods that ask for a queue. A drill then showed an ordinary pod being created while Kueue was down, and a pod with a queue label still being refused.
What's left
The lab has no external users and no production load. Open items:
- The Dokploy migration is mostly done. A few apps and some cleanup are left, including moving the last pipelines off the forge's built-in registry.
- Image scans don't gate deploys. Harbor scans every image, nothing blocks a deploy on the result, and images aren't signed.
- Canaries and network policy are only proven on demos. k3s enforces network policy, and a demo app shows blocked traffic being refused, but no real app ships a NetworkPolicy yet.
- Restore has been tested once, on a demo volume.
- The event log has no consumers yet, and the producers still have to move onto the shared envelope.
- Argo CD polls for changes and sends no notifications.
- Bot commits are gated after the fact, with admission control as the backstop.