The cluster has no ingress controller of its own. Traefik on the
server reads IngressRoute CRDs straight from the Kubernetes
API and routes to pod IPs over the WireGuard mesh, which keeps TLS
termination and certificate handling in one place on the host.
Configuration lives in modules/features/traefik.nix
(kubernetesCRD provider,
allowCrossNamespace = false):
https://10.100.0.10:6443kubernetes/traefik-kubeconfigtalos-config/manifests/traefik-rbac.yaml
(ServiceAccount in the traefik-external namespace)*.yhkze.net wildcard
certificateBecause the provider is cross-provider aware, middlewares defined in
Traefik’s file provider are referenced with the name@file
syntax (for example sso@file) — no Kubernetes
Middleware CRD is required.
Routes currently published from the cluster:
| Host | Service |
|---|---|
argocd.yhkze.net |
ArgoCD server |
vault.yhkze.net |
OpenBao (own auth, Keycloak OIDC) |
ArgoCD runs with configs.params."server.insecure"=true
so Traefik terminates TLS once instead of re-encrypting to the pod.
The pod CIDR 10.244.0.0/16 is declared as an AllowedIP
on the k8s-worker peer in modules/features/wireguard.nix,
so the server can reach pod IPs directly. Reference WireGuard VPN for the mesh
itself.
The nodes’ own side of that tunnel is mesh-only: each node lists
10.100.0.0/24 as the server peer’s allowedIPs.
Pod CIDR must not be added there.
AllowedIPs doubles as a routing table, so claiming
10.244.0.0/16 on the node would pull pod traffic into the tunnel and
take it away from Cilium, which owns pod routing.
Neither may it carry a default route. AllowedIPs doubles
as a routing table, so 0.0.0.0/0 sends internet-bound pod
traffic into the tunnel, where the server has no matching NAT rule and
drops it. Egress has to leave by the Incus bridge instead:
pod → Cilium SNAT to node address → incusbr0 → server masquerade → internet
The server masquerades 10.200.3.0/24 and not the pod
CIDR, which is why Cilium’s SNAT to the node address is what makes
egress work. Nothing masquerades 10.100.0.0/24, so anything
routed over wg0 toward the internet is lost.
Path MTU — The WAN path MTU is below 1500 and drops
full-size segments without an ICMP reply, so TLS handshakes succeed
while bulk downloads stall. net.ipv4.tcp_mtu_probing in
modules/features/sysctl.nix covers only sockets on the
server itself. Forwarded traffic gets MSS clamping instead: the
incus-mss-clamp nftables table in
modules/features/incus.nix lowers the MSS of every TCP SYN
entering or leaving incusbr0 to 1360, so both peers send
segments that fit.
Verify the limit from the server; the largest passing size plus 28 is the path MTU, and MSS is that minus 40:
ping -M do -s 1464 -c2 1.1.1.1Each host holds two addresses, and only one of them means the same thing everywhere:
| Range | Scope | Use for |
|---|---|---|
10.100.0.x |
WireGuard mesh, unique per host | anything crossing a host boundary |
10.200.3.x |
Incus bridge, identical on every host | nothing beyond the local machine |
The bridge gateway is 10.200.3.1 on the server, the
laptop and any future Incus member alike, because a managed Incus
network is a cluster-level object — ipv4.address is global,
and only bridge.external_interfaces, parent
and the BGP next-hop can be set per member. So 10.200.3.1
names “whichever host asked”, which is useless in a config file read on
more than one machine.
Always reach the server as 10.100.0.1. The route is
symmetric: nodes reach 10.100.0.0/24 over wg0,
and the server returns pod traffic over the same tunnel via the
k8s-worker peer’s AllowedIPs. The bridge gateway would
instead send traffic out one path and have it return over another.
Every *.yhkze.net name resolves publicly to the server’s
WAN address. Left alone, a pod reaching one of them leaves the cluster,
traverses the internet and Cloudflare, and re-enters through the
server’s own router to reach a service on a machine it can already see
directly. That round trip depends on the router hairpinning traffic back
to its own WAN address, and it burns upstream bandwidth for a purely
local call.
The fix is split-horizon resolution at the resolver the nodes already use. Talos hands the nodes the Incus bridge gateway as their nameserver, and CoreDNS forwards there, so one override covers every workload:
incus network set incusbr0 raw.dnsmasq='address=/yhkze.net/10.100.0.1'Host Traefik listens on wildcard sockets for :80,
:443 and Forgejo’s :30354, so a single address
covers the web services and git over SSH alike. TLS still verifies — the
URL host stays vault.yhkze.net, so that remains the SNI
Traefik selects on and the name checked against the wildcard
certificate. Only the transport address changes, which validation
ignores.
Confirm resolution, and that unrelated names still leave the house:
dig +short @10.200.3.1 vault.yhkze.net # expect 10.100.0.1
dig +short @10.200.3.1 example.com # expect public addressesTwo constraints explain why the override lives in Incus rather than
anywhere more obvious. CoreDNS is not available: its ConfigMap is owned
by Talos
(config.k8s.io/owning-inventory=talos-bootstrap-manifests-inventory)
and reconciled back on upgrades, and Talos exposes no Corefile
customization, only cluster.coreDNS.disabled and
.image. Per-workload hostAliases are the other
alternative, and a worse one: they scale with the number of consumers,
and ArgoCD needs the same resolution while not being GitOps-managed
itself.
raw.dnsmasq is Incus state rather than a file, so
modules/features/incus.nix enforces it: the
incus-split-horizon-dns unit compares the live value on
every boot and rewrites it only when it has drifted. The bridge
addressing above is still restored by hand after a rebuild.
Managed-network keys are applied across every cluster member, so the
command fails with peer node <addr> is down while any
member is offline. Either bring the member up or remove it before
changing network configuration.
Node image pulls go to Docker Hub and GHCR over the WAN, where some
connections lose packets and a long layer download can crawl past every
timeout. The server runs a pull-through cache per upstream
(modules/features/registry-cache.nix), and the nodes use
them as registry mirrors, so each image crosses the WAN once and later
pulls stay on the mesh.
| Upstream | Cache | Auth to upstream |
|---|---|---|
docker.io |
http://10.100.0.1:5000 |
anonymous |
ghcr.io |
http://10.100.0.1:5001 |
SOPS registry-cache/ghcr-pull-username,
registry-cache/ghcr-pull-token |
The caches listen on the mesh address and the firewall opens them on
wg0 only. Plain HTTP is enough there: the tunnel already
encrypts the transport, and blobs are verified against their content
digests. The GHCR cache holds a read:packages token, so
anything on the mesh can pull the private rgxr images through it — the
same reach the cluster’s own pull secret already grants.
Talos falls back to the upstream registry whenever a mirror is unreachable, so stopping the caches costs speed, not availability.
Apply — the mirrors live in
talos-config/patches/registry-mirrors.yaml. Registry
settings take effect without a reboot. Apply the patch once per node;
re-applying a fragment patch appends to the endpoints
lists, so later edits go through the running-config procedure under
WireGuard Port Coupling instead:
talosctl -e 10.100.0.10 -n 10.100.0.10 patch mc --mode=no-reboot \
--patch @talos-config/patches/registry-mirrors.yaml
talosctl -e 10.100.0.11 -n 10.100.0.11 patch mc --mode=no-reboot \
--patch @talos-config/patches/registry-mirrors.yamlVerify — the cache answers on the mesh, and a node pull lands in it:
curl -s http://10.100.0.1:5000/v2/_catalog
talosctl -e 10.100.0.11 -n 10.100.0.11 image pull docker.io/library/alpine:3
curl -s http://10.100.0.1:5000/v2/_catalog # now lists library/alpineThe Talos nodes dial the server’s WireGuard endpoint, and that port is written in two places that git cannot keep in sync:
| Side | Where | Tracked in git? |
|---|---|---|
| Server | modules/features/wireguard.nix
(ListenPort, peer Endpoint,
allowedUDPPorts) |
yes |
| Talos | talos-config/patches/wireguard-{master,worker}.yaml
(peer endpoint) |
no — gitignored, they hold private keys |
Changing the server’s listen port therefore requires patching both Talos nodes by hand. A repo-wide search will not find the Talos side.
The endpoint is 10.200.3.1:30352 — the bridge gateway,
and the one sanctioned exception to the rule above. A tunnel cannot be
its own transport, so the node has to dial the server over an address
that does not depend on WireGuard already being up.
10.100.0.1 is unreachable until the handshake it would be
establishing has completed.
Do not use patch mc with these files. A
YAML-fragment patch merges lists by appending, so re-applying an edited
peers: entry adds a second peer rather than replacing the
first. WireGuard identifies peers by public key and rejects a
configuration carrying the same key twice, which takes the whole
interface down — not just the edited peer. JSON6902 patches, which could
target a list index directly, are not an option either: Talos rejects
them for multi-document configurations.
Edit the node’s own running configuration instead, so nothing depends on a local base file being current. Extract it, change the one field, and confirm the diff is only that field:
talosctl -e 10.100.0.11 -n 10.100.0.11 get mc v1alpha1 -o yaml > live.yaml
awk 'f{sub(/^ /,""); print} /^spec: \|$/{f=1}' live.yaml > apply.yaml
# edit apply.yaml, then:
talosctl -e 10.100.0.11 -n 10.100.0.11 apply-config --dry-run -f apply.yamlApply with --mode=try first. The node reverts on its own
after the timeout, so a change that severs the tunnel repairs itself
instead of needing bridge-side recovery — 10.200.3.x is
machine-local, so out-of-band access would mean working from the server
rather than a workstation:
talosctl -e 10.100.0.11 -n 10.100.0.11 apply-config --mode=try --timeout 3m \
-f apply.yamlVerify inside that window, then re-apply with
--mode=no-reboot to persist. Mirror the change back into
talos-config/patches/ so the gitignored files do not drift
from the nodes.
The extracted configuration contains the node’s WireGuard private key and join token. Keep it in a scratch directory and delete it afterwards.
Confirm both peers handshake, and that each key appears once:
wg show wg0 | grep -A3 '10.100.0.1[01]/32'
wg show wg0 public-keyThis bit once: commit 10f3470 (“migrate service ports
from 4035x to 3035x”) updated the server to 30352 but left both Talos
configs on 40352. The mesh stayed down until the nodes were next
restarted, at which point every workload that needed the API server
crash-looped. The symptom to recognise is wg show listing
the k8s peers with no endpoint and no handshake while the nodes still
answer ICMP on the Incus bridge.