Networking and Ingress

Host Traefik as Ingress Controller

The cluster has no ingress controller of its own. Traefik on the server reads IngressRoute CRDs straight from the Kubernetes API and routes to pod IPs over the WireGuard mesh, which keeps TLS termination and certificate handling in one place on the host.

Configuration lives in modules/features/traefik.nix (kubernetesCRD provider, allowCrossNamespace = false):

Because the provider is cross-provider aware, middlewares defined in Traefik’s file provider are referenced with the name@file syntax (for example sso@file) — no Kubernetes Middleware CRD is required.

Routes currently published from the cluster:

Host Service
argocd.yhkze.net ArgoCD server
vault.yhkze.net OpenBao (own auth, Keycloak OIDC)

ArgoCD runs with configs.params."server.insecure"=true so Traefik terminates TLS once instead of re-encrypting to the pod.

Pod Routing

The pod CIDR 10.244.0.0/16 is declared as an AllowedIP on the k8s-worker peer in modules/features/wireguard.nix, so the server can reach pod IPs directly. Reference WireGuard VPN for the mesh itself.

The nodes’ own side of that tunnel is mesh-only: each node lists 10.100.0.0/24 as the server peer’s allowedIPs. Pod CIDR must not be added there. AllowedIPs doubles as a routing table, so claiming 10.244.0.0/16 on the node would pull pod traffic into the tunnel and take it away from Cilium, which owns pod routing.

Neither may it carry a default route. AllowedIPs doubles as a routing table, so 0.0.0.0/0 sends internet-bound pod traffic into the tunnel, where the server has no matching NAT rule and drops it. Egress has to leave by the Incus bridge instead:

pod → Cilium SNAT to node address → incusbr0 → server masquerade → internet

The server masquerades 10.200.3.0/24 and not the pod CIDR, which is why Cilium’s SNAT to the node address is what makes egress work. Nothing masquerades 10.100.0.0/24, so anything routed over wg0 toward the internet is lost.

Path MTU — The WAN path MTU is below 1500 and drops full-size segments without an ICMP reply, so TLS handshakes succeed while bulk downloads stall. net.ipv4.tcp_mtu_probing in modules/features/sysctl.nix covers only sockets on the server itself. Forwarded traffic gets MSS clamping instead: the incus-mss-clamp nftables table in modules/features/incus.nix lowers the MSS of every TCP SYN entering or leaving incusbr0 to 1360, so both peers send segments that fit.

Verify the limit from the server; the largest passing size plus 28 is the path MTU, and MSS is that minus 40:

ping -M do -s 1464 -c2 1.1.1.1

Which Server Address to Use

Each host holds two addresses, and only one of them means the same thing everywhere:

Range Scope Use for
10.100.0.x WireGuard mesh, unique per host anything crossing a host boundary
10.200.3.x Incus bridge, identical on every host nothing beyond the local machine

The bridge gateway is 10.200.3.1 on the server, the laptop and any future Incus member alike, because a managed Incus network is a cluster-level object — ipv4.address is global, and only bridge.external_interfaces, parent and the BGP next-hop can be set per member. So 10.200.3.1 names “whichever host asked”, which is useless in a config file read on more than one machine.

Always reach the server as 10.100.0.1. The route is symmetric: nodes reach 10.100.0.0/24 over wg0, and the server returns pod traffic over the same tunnel via the k8s-worker peer’s AllowedIPs. The bridge gateway would instead send traffic out one path and have it return over another.

Resolving Service Names Inside the Cluster

Every *.yhkze.net name resolves publicly to the server’s WAN address. Left alone, a pod reaching one of them leaves the cluster, traverses the internet and Cloudflare, and re-enters through the server’s own router to reach a service on a machine it can already see directly. That round trip depends on the router hairpinning traffic back to its own WAN address, and it burns upstream bandwidth for a purely local call.

The fix is split-horizon resolution at the resolver the nodes already use. Talos hands the nodes the Incus bridge gateway as their nameserver, and CoreDNS forwards there, so one override covers every workload:

incus network set incusbr0 raw.dnsmasq='address=/yhkze.net/10.100.0.1'

Host Traefik listens on wildcard sockets for :80, :443 and Forgejo’s :30354, so a single address covers the web services and git over SSH alike. TLS still verifies — the URL host stays vault.yhkze.net, so that remains the SNI Traefik selects on and the name checked against the wildcard certificate. Only the transport address changes, which validation ignores.

Confirm resolution, and that unrelated names still leave the house:

dig +short @10.200.3.1 vault.yhkze.net   # expect 10.100.0.1
dig +short @10.200.3.1 example.com       # expect public addresses

Two constraints explain why the override lives in Incus rather than anywhere more obvious. CoreDNS is not available: its ConfigMap is owned by Talos (config.k8s.io/owning-inventory=talos-bootstrap-manifests-inventory) and reconciled back on upgrades, and Talos exposes no Corefile customization, only cluster.coreDNS.disabled and .image. Per-workload hostAliases are the other alternative, and a worse one: they scale with the number of consumers, and ArgoCD needs the same resolution while not being GitOps-managed itself.

raw.dnsmasq is Incus state rather than a file, so modules/features/incus.nix enforces it: the incus-split-horizon-dns unit compares the live value on every boot and rewrites it only when it has drifted. The bridge addressing above is still restored by hand after a rebuild.

Managed-network keys are applied across every cluster member, so the command fails with peer node <addr> is down while any member is offline. Either bring the member up or remove it before changing network configuration.

Registry Pull-Through Cache

Node image pulls go to Docker Hub and GHCR over the WAN, where some connections lose packets and a long layer download can crawl past every timeout. The server runs a pull-through cache per upstream (modules/features/registry-cache.nix), and the nodes use them as registry mirrors, so each image crosses the WAN once and later pulls stay on the mesh.

Upstream Cache Auth to upstream
docker.io http://10.100.0.1:5000 anonymous
ghcr.io http://10.100.0.1:5001 SOPS registry-cache/ghcr-pull-username, registry-cache/ghcr-pull-token

The caches listen on the mesh address and the firewall opens them on wg0 only. Plain HTTP is enough there: the tunnel already encrypts the transport, and blobs are verified against their content digests. The GHCR cache holds a read:packages token, so anything on the mesh can pull the private rgxr images through it — the same reach the cluster’s own pull secret already grants.

Talos falls back to the upstream registry whenever a mirror is unreachable, so stopping the caches costs speed, not availability.

Apply — the mirrors live in talos-config/patches/registry-mirrors.yaml. Registry settings take effect without a reboot. Apply the patch once per node; re-applying a fragment patch appends to the endpoints lists, so later edits go through the running-config procedure under WireGuard Port Coupling instead:

talosctl -e 10.100.0.10 -n 10.100.0.10 patch mc --mode=no-reboot \
  --patch @talos-config/patches/registry-mirrors.yaml
talosctl -e 10.100.0.11 -n 10.100.0.11 patch mc --mode=no-reboot \
  --patch @talos-config/patches/registry-mirrors.yaml

Verify — the cache answers on the mesh, and a node pull lands in it:

curl -s http://10.100.0.1:5000/v2/_catalog
talosctl -e 10.100.0.11 -n 10.100.0.11 image pull docker.io/library/alpine:3
curl -s http://10.100.0.1:5000/v2/_catalog   # now lists library/alpine

WireGuard Port Coupling

The Talos nodes dial the server’s WireGuard endpoint, and that port is written in two places that git cannot keep in sync:

Side Where Tracked in git?
Server modules/features/wireguard.nix (ListenPort, peer Endpoint, allowedUDPPorts) yes
Talos talos-config/patches/wireguard-{master,worker}.yaml (peer endpoint) no — gitignored, they hold private keys

Changing the server’s listen port therefore requires patching both Talos nodes by hand. A repo-wide search will not find the Talos side.

The endpoint is 10.200.3.1:30352 — the bridge gateway, and the one sanctioned exception to the rule above. A tunnel cannot be its own transport, so the node has to dial the server over an address that does not depend on WireGuard already being up. 10.100.0.1 is unreachable until the handshake it would be establishing has completed.

Do not use patch mc with these files. A YAML-fragment patch merges lists by appending, so re-applying an edited peers: entry adds a second peer rather than replacing the first. WireGuard identifies peers by public key and rejects a configuration carrying the same key twice, which takes the whole interface down — not just the edited peer. JSON6902 patches, which could target a list index directly, are not an option either: Talos rejects them for multi-document configurations.

Edit the node’s own running configuration instead, so nothing depends on a local base file being current. Extract it, change the one field, and confirm the diff is only that field:

talosctl -e 10.100.0.11 -n 10.100.0.11 get mc v1alpha1 -o yaml > live.yaml
awk 'f{sub(/^    /,""); print} /^spec: \|$/{f=1}' live.yaml > apply.yaml
# edit apply.yaml, then:
talosctl -e 10.100.0.11 -n 10.100.0.11 apply-config --dry-run -f apply.yaml

Apply with --mode=try first. The node reverts on its own after the timeout, so a change that severs the tunnel repairs itself instead of needing bridge-side recovery — 10.200.3.x is machine-local, so out-of-band access would mean working from the server rather than a workstation:

talosctl -e 10.100.0.11 -n 10.100.0.11 apply-config --mode=try --timeout 3m \
  -f apply.yaml

Verify inside that window, then re-apply with --mode=no-reboot to persist. Mirror the change back into talos-config/patches/ so the gitignored files do not drift from the nodes.

The extracted configuration contains the node’s WireGuard private key and join token. Keep it in a scratch directory and delete it afterwards.

Confirm both peers handshake, and that each key appears once:

wg show wg0 | grep -A3 '10.100.0.1[01]/32'
wg show wg0 public-key

This bit once: commit 10f3470 (“migrate service ports from 4035x to 3035x”) updated the server to 30352 but left both Talos configs on 40352. The mesh stayed down until the nodes were next restarted, at which point every workload that needed the API server crash-looped. The symptom to recognise is wg show listing the k8s peers with no endpoint and no handshake while the nodes still answer ICMP on the Incus bridge.