OpenBao Secret Management

Overview

Centralized secret management for in-cluster workloads. Kubernetes authentication lets a pod’s ServiceAccount token stand in for a credential, so nothing has to ship hardcoded secrets; operators authenticate separately through OIDC.

URL: vault.yhkze.net (OpenBao native auth, Keycloak OIDC)

Architecture

OpenBao runs as a NixOS service on the server, not as a workload in the cluster. It holds the credentials the cluster needs in order to start, so it must not depend on the cluster being up. Configuration lives in modules/features/openbao.nix.

flowchart LR
    pod["Pod\n(projected SA token)"] -->|https| traefik["Host Traefik\nvault.yhkze.net"]
    traefik -->|http| bao["openbao.service\n127.0.0.1:8200"]
    bao -->|TokenReview| api["kube-apiserver\n10.100.0.10:6443"]
    sops["SOPS\nopenbao/seal-key"] -->|EnvironmentFile| bao
    bao --> raft[("Raft\n/var/lib/openbao")]

Single-node Raft storage at /var/lib/openbao, node ID server, persisted through impermanence. The listener binds 127.0.0.1:8200 with TLS disabled — host Traefik terminates the ACME wildcard, so OpenBao is never directly routable.

Auto-Unseal

A static seal wraps the Raft encryption keys with a key held outside the barrier, so the service unseals itself on restart with no operator present:

seal.static = {
  current_key_id = "v1";
  current_key = "env://OPENBAO_SEAL_KEY";
};

The key is read from the environment at every start. It comes from the SOPS secret openbao/seal-key, rendered by sops.templates."openbao-env" into a KEY=value file and passed to the unit as EnvironmentFile. The seal key is therefore required at every boot, not just at initialization — it is not a recovery share, and losing it costs the Raft store.

Recovery shares (3 of 5) exist for the separate case of regaining operator access. They do not substitute for the seal key.

Changing current_key after initialization requires a seal migration, not just a configuration edit.

The upstream module sets restartIfChanged = false, so nixos-rebuild reports NOT restarting the following changed units: openbao.service and leaves the old configuration running. Restart it explicitly after any change here:

sudo systemctl restart openbao
bao status          # expect Sealed: false

Kubernetes Authentication

OpenBao runs off-cluster, so it cannot read the cluster’s signing keys to validate ServiceAccount tokens offline — the API server returns 401 on anonymous /openid/v1/jwks. Instead it calls the TokenReview API as a dedicated reviewer ServiceAccount.

That reviewer is declared in charts/external-secrets/templates/openbao-token-reviewer.yaml: a ServiceAccount bound to the system:auth-delegator ClusterRole, plus a kubernetes.io/service-account-token Secret. The long-lived Secret is required because projected tokens expire and cannot serve as a stored credential.

One-time bootstrap, run on the server with the root token in the environment so it never reaches a file or shell history:

read -rs VAULT_TOKEN && export VAULT_TOKEN
export VAULT_ADDR=http://127.0.0.1:8200

bao auth enable kubernetes
bao write auth/kubernetes/config \
  kubernetes_host=https://10.100.0.10:6443 \
  [email protected] \
  [email protected]

k8s-ca.crt and reviewer.jwt come from the reviewer Secret:

kubectl -n external-secrets get secret openbao-token-reviewer \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > k8s-ca.crt
kubectl -n external-secrets get secret openbao-token-reviewer \
  -o jsonpath='{.data.token}' | base64 -d > reviewer.jwt

Human Authentication via OIDC

Operators log in through Keycloak rather than with the root token, so access follows the same group membership as every other service and the root token can stay in the password manager.

Keycloak side, client openbao in realm internal (tofu/keycloak/internal.tf):

OpenBao side: the oidc auth method points at the realm’s issuer with groups_claim=groups, and a single role named default handles both the UI and CLI flows. Two policies exist — openbao-admin and openbao-readonly — and they are not bound to the role directly. Instead each is attached to an external identity group whose alias is the matching Keycloak group, openbao_admin or openbao_readonly, so a user’s capabilities follow group membership with no change inside OpenBao.

The auth method’s configuration, applied with a root or admin token (the secret travels on stdin):

sops decrypt --extract '["openbao"]["oidc-client-secret"]' hosts/server/secrets/secrets.yaml \
  | bao write auth/oidc/config \
      oidc_discovery_url=https://id.yhkze.net/realms/internal \
      oidc_client_id=openbao oidc_client_secret=- default_role=default

Granting Access to a Workload

Give each consumer its own mount, a read-only policy scoped to it, and a role bound to the ServiceAccounts that may use it. Read-only by design: a compromised consumer must not be able to rewrite the credentials it reads.

Kargo, which reads through External Secrets, is the worked example:

bao secrets enable -path=kargo kv-v2

bao policy write kargo-eso - <<'EOF'
path "kargo/data/*"     { capabilities = ["read"] }
path "kargo/metadata/*" { capabilities = ["read", "list"] }
EOF

bao write auth/kubernetes/role/kargo-eso \
  bound_service_account_names=kargo-eso \
  bound_service_account_namespaces=kargo,kargo-system-resources,kargo-shared-resources \
  policies=kargo-eso \
  ttl=1h

All three namespaces reuse the ServiceAccount name kargo-eso, so one role covers them. Reference GitOps and CI/CD for how the consuming SecretStore and ExternalSecret objects are declared.

Seed the values the consumer reads, then confirm the keys landed without printing them:

bao kv put -mount=kargo config KEY='<value>'
bao kv get -format=json -mount=kargo config | jq '.data.data | keys'

Use bao kv patch instead of put to rotate one key while leaving the rest intact.

Registry Pull Credentials

Pull credentials serve consumers in many namespaces, so they sit behind one cluster-wide store, openbao-registry, declared in charts/external-secrets/templates/registry-store.yaml. It authenticates as the registry-eso ServiceAccount and may only be referenced from kargo-shared-resources and namespaces ending in -test or -prod.

Bootstrap the mount, a read-only policy and the role bound to that ServiceAccount:

bao secrets enable -path=registry kv-v2

bao policy write registry-pull - <<'EOF'
path "registry/data/*"     { capabilities = ["read"] }
path "registry/metadata/*" { capabilities = ["read", "list"] }
EOF

bao write auth/kubernetes/role/registry-pull \
  bound_service_account_names=registry-eso \
  bound_service_account_namespaces=external-secrets \
  policies=registry-pull \
  ttl=1h

Store the GHCR token without echoing it. It must be a classic PAT with only read:packages, since fine-grained tokens do not cover GHCR:

read -rs T && bao kv put -mount=registry ghcr-pull username=yuuhikaze token="$T"; unset T

Runtime State Is Not Declarative

Auth methods, policies, roles and KV data live in the Raft store, not in git. Only the service definition is declarative. Recreating the server therefore means restoring a snapshot or repeating the bootstrap above.

Backup

Raft snapshots capture the full encrypted state — safe to store offsite:

bao operator raft snapshot save "openbao-$(date +%Y%m%d).snap"

Restore with bao operator raft snapshot restore <file>. The snapshot is encrypted with the seal key, so a restore needs that key as well as the file.