Centralized secret management for in-cluster workloads. Kubernetes authentication lets a pod’s ServiceAccount token stand in for a credential, so nothing has to ship hardcoded secrets; operators authenticate separately through OIDC.
URL: vault.yhkze.net (OpenBao native auth, Keycloak
OIDC)
OpenBao runs as a NixOS service on the server, not as a workload in
the cluster. It holds the credentials the cluster needs in order to
start, so it must not depend on the cluster being up. Configuration
lives in modules/features/openbao.nix.
flowchart LR
pod["Pod\n(projected SA token)"] -->|https| traefik["Host Traefik\nvault.yhkze.net"]
traefik -->|http| bao["openbao.service\n127.0.0.1:8200"]
bao -->|TokenReview| api["kube-apiserver\n10.100.0.10:6443"]
sops["SOPS\nopenbao/seal-key"] -->|EnvironmentFile| bao
bao --> raft[("Raft\n/var/lib/openbao")]
Single-node Raft storage at /var/lib/openbao, node ID
server, persisted through impermanence. The listener binds
127.0.0.1:8200 with TLS disabled — host Traefik terminates
the ACME wildcard, so OpenBao is never directly routable.
A static seal wraps the Raft encryption keys with a key held outside the barrier, so the service unseals itself on restart with no operator present:
seal.static = {
current_key_id = "v1";
current_key = "env://OPENBAO_SEAL_KEY";
};The key is read from the environment at every start. It comes from
the SOPS secret openbao/seal-key, rendered by
sops.templates."openbao-env" into a KEY=value
file and passed to the unit as EnvironmentFile. The seal
key is therefore required at every boot, not just at initialization — it
is not a recovery share, and losing it costs the Raft store.
Recovery shares (3 of 5) exist for the separate case of regaining operator access. They do not substitute for the seal key.
Changing current_key after initialization requires a
seal migration, not just a configuration edit.
The upstream module sets restartIfChanged = false, so
nixos-rebuild reports
NOT restarting the following changed units: openbao.service
and leaves the old configuration running. Restart it explicitly after
any change here:
sudo systemctl restart openbao
bao status # expect Sealed: falseOpenBao runs off-cluster, so it cannot read the cluster’s signing
keys to validate ServiceAccount tokens offline — the API server returns
401 on anonymous /openid/v1/jwks. Instead it calls the
TokenReview API as a dedicated reviewer ServiceAccount.
That reviewer is declared in
charts/external-secrets/templates/openbao-token-reviewer.yaml:
a ServiceAccount bound to the system:auth-delegator
ClusterRole, plus a kubernetes.io/service-account-token
Secret. The long-lived Secret is required because projected tokens
expire and cannot serve as a stored credential.
One-time bootstrap, run on the server with the root token in the environment so it never reaches a file or shell history:
read -rs VAULT_TOKEN && export VAULT_TOKEN
export VAULT_ADDR=http://127.0.0.1:8200
bao auth enable kubernetes
bao write auth/kubernetes/config \
kubernetes_host=https://10.100.0.10:6443 \
[email protected] \
[email protected]k8s-ca.crt and reviewer.jwt come from the
reviewer Secret:
kubectl -n external-secrets get secret openbao-token-reviewer \
-o jsonpath='{.data.ca\.crt}' | base64 -d > k8s-ca.crt
kubectl -n external-secrets get secret openbao-token-reviewer \
-o jsonpath='{.data.token}' | base64 -d > reviewer.jwtOperators log in through Keycloak rather than with the root token, so access follows the same group membership as every other service and the root token can stay in the password manager.
Keycloak side, client openbao in realm
internal (tofu/keycloak/internal.tf):
groups, which puts plain group
names in the ID tokenhttps://vault.yhkze.net/ui/vault/auth/oidc/oidc/callback
for the web UI, and http://localhost:8250/oidc/callback for
the loopback listener that bao login -method=oidc
spawnsopenbao/oidc-client-secretOpenBao side: the oidc auth method points at the realm’s
issuer with groups_claim=groups, and a single role named
default handles both the UI and CLI flows. Two policies
exist — openbao-admin and openbao-readonly —
and they are not bound to the role directly. Instead each is attached to
an external identity group whose alias is the matching Keycloak group,
openbao_admin or openbao_readonly, so a user’s
capabilities follow group membership with no change inside OpenBao.
The auth method’s configuration, applied with a root or admin token (the secret travels on stdin):
sops decrypt --extract '["openbao"]["oidc-client-secret"]' hosts/server/secrets/secrets.yaml \
| bao write auth/oidc/config \
oidc_discovery_url=https://id.yhkze.net/realms/internal \
oidc_client_id=openbao oidc_client_secret=- default_role=defaultGive each consumer its own mount, a read-only policy scoped to it, and a role bound to the ServiceAccounts that may use it. Read-only by design: a compromised consumer must not be able to rewrite the credentials it reads.
Kargo, which reads through External Secrets, is the worked example:
bao secrets enable -path=kargo kv-v2
bao policy write kargo-eso - <<'EOF'
path "kargo/data/*" { capabilities = ["read"] }
path "kargo/metadata/*" { capabilities = ["read", "list"] }
EOF
bao write auth/kubernetes/role/kargo-eso \
bound_service_account_names=kargo-eso \
bound_service_account_namespaces=kargo,kargo-system-resources,kargo-shared-resources \
policies=kargo-eso \
ttl=1hAll three namespaces reuse the ServiceAccount name
kargo-eso, so one role covers them. Reference GitOps and CI/CD for how the consuming
SecretStore and ExternalSecret objects are
declared.
Seed the values the consumer reads, then confirm the keys landed without printing them:
bao kv put -mount=kargo config KEY='<value>'
bao kv get -format=json -mount=kargo config | jq '.data.data | keys'Use bao kv patch instead of put to rotate
one key while leaving the rest intact.
Pull credentials serve consumers in many namespaces, so they sit
behind one cluster-wide store, openbao-registry, declared
in charts/external-secrets/templates/registry-store.yaml.
It authenticates as the registry-eso ServiceAccount and may
only be referenced from kargo-shared-resources and
namespaces ending in -test or -prod.
Bootstrap the mount, a read-only policy and the role bound to that ServiceAccount:
bao secrets enable -path=registry kv-v2
bao policy write registry-pull - <<'EOF'
path "registry/data/*" { capabilities = ["read"] }
path "registry/metadata/*" { capabilities = ["read", "list"] }
EOF
bao write auth/kubernetes/role/registry-pull \
bound_service_account_names=registry-eso \
bound_service_account_namespaces=external-secrets \
policies=registry-pull \
ttl=1hStore the GHCR token without echoing it. It must be a classic PAT
with only read:packages, since fine-grained tokens do not
cover GHCR:
read -rs T && bao kv put -mount=registry ghcr-pull username=yuuhikaze token="$T"; unset TAuth methods, policies, roles and KV data live in the Raft store, not in git. Only the service definition is declarative. Recreating the server therefore means restoring a snapshot or repeating the bootstrap above.
Raft snapshots capture the full encrypted state — safe to store offsite:
bao operator raft snapshot save "openbao-$(date +%Y%m%d).snap"Restore with
bao operator raft snapshot restore <file>. The
snapshot is encrypted with the seal key, so a restore needs that key as
well as the file.