Kargo decides which image version each environment runs. ArgoCD still applies manifests and CI still builds images, but neither chooses a version: CI publishes to GHCR and stops, and ArgoCD syncs whatever git says. Kargo sits between them, observing the registry and writing the chosen version into the charts repo.
URL: kargo.yhkze.net (Keycloak OIDC)
The split matters because it keeps CI out of the cluster and out of the charts repo. A build job needs no git token and no Kubernetes credential — reference GitOps and CI/CD.
Four objects carry the whole model:
The sequence below follows one build from push to deployment. CI’s steps run in order within one job, so the image is in GHCR before Kargo is notified:
sequenceDiagram
participant CI as Forgejo Actions
participant G as GHCR
participant R as Kargo receiver
participant W as Warehouse
participant S as Stage
participant C as gitops/charts
participant A as ArgoCD
CI->>G: push layers, PUT manifest
G-->>CI: 201 Created
CI->>R: POST {"repository":{"url":"ghcr.io/…/<app>"}}
R->>W: refresh Warehouses subscribed to that image
W->>G: list tags, resolve digest
W->>S: new Freight
S->>C: commit digest to values-<env>.yaml, push
S->>A: argocd-wait until healthy
A->>C: fetch, render, sync <app>-<env>
A lost or early webhook costs only latency: every Warehouse also polls on its own five-minute interval.
One Kargo Project per application, declared in
apps/promotion/<app>.yaml. A Project is also a
namespace, so its Warehouses, Stages and credentials are scoped to
it.
The dummy project is the worked example and the template
for real ones:
| Object | Selection strategy | Source |
|---|---|---|
warehouse/test |
Digest |
ghcr.io/yuuhikaze/dummy, tag main |
warehouse/prod |
SemVer |
ghcr.io/yuuhikaze/dummy,
allowTagsRegexes |
Two Warehouses rather than one, because the strategies differ in
kind. The test Warehouse tracks a mutable tag, so it must
follow the digest to notice that main now points somewhere
else. The prod Warehouse tracks immutable release tags and
picks the highest version matching the regex. A release tag is therefore
the only thing that can produce prod Freight — an accidental push to
main cannot reach production.
ProjectConfig carries the promotion policies, each
matching a Stage by stageSelector.
Each Stage runs the same sequence, differing only in which values file it edits:
git-clone — the charts repo over HTTPS, at
branch: masteryaml-update — write the new image digest into
values/values-<env>.yamlgit-commit — attribute to Freight, so history records
what moved and whygit-pushargocd-wait — block until
${{ ctx.project }}-${{ ctx.stage }} is healthyThe final step is what makes a promotion mean “deployed” rather than “committed”: without it a Promotion succeeds the moment git accepts the push, regardless of whether the workload ever became healthy.
Each Stage has its own ServiceAccount, so promotion permissions are scoped per environment rather than shared project-wide.
Kargo discovers credentials by label, not by name —
kargo.akuity.io/cred-type on a Secret in the project or
shared-resources namespace. The label is load bearing; renaming a Secret
is safe, dropping its label is not.
All of them come from OpenBao through External Secrets, so none is hand-created. Reference OpenBao Secret Management.
| Secret | Namespace | cred-type |
Purpose |
|---|---|---|---|
kargo-api-secret |
kargo |
— | admin password hash, token signing key |
kargo-ci-webhook |
kargo-system-resources |
generic |
obscures receiver URL |
charts-repo |
kargo-shared-resources |
git |
push promotion commits |
ghcr-pull |
kargo-shared-resources |
image |
discover private GHCR images |
The git credential is HTTPS with a username and token, not the SSH
URL ArgoCD uses — a token pair only authenticates over HTTPS. Its
repoURL must match the git-clone URL exactly,
since Kargo matches credentials to repositories by URL.
ghcr-pull matches ^ghcr\.io/yuuhikaze/.* as
a regex, so one Secret covers every image under the namespace. It reads
the read-only token from the cluster-wide openbao-registry
store, which also supplies kubelet pull secrets in app namespaces.
Without it a Warehouse reports
DENIED: requested access to the resource is denied from the
GHCR token endpoint.
GHCR cannot call into this cluster and the source repository lives on Forgejo, so nothing would otherwise tell Kargo an image landed — Warehouses would only notice at their own five-minute interval.
ClusterConfig declares a generic receiver
named ci, with its shared secret in
kargo-ci-webhook. A ClusterConfig receiver acts on
Warehouses across all Projects, and this one selects them by the image
named in the request body:
indexSelector:
matchIndices:
- key: subscribedURLs
operator: Equals
value: ${{ normalizeImage(request.body.repository.url) }}A push therefore refreshes only the Warehouses subscribed to that image. That makes one receiver URL safe to share across every service repository, which is why it lives as a single organization secret in Forgejo.
Kargo derives a hard-to-guess path from that secret, so the receiver URL is itself a credential — treat it like one and do not paste it into issues or chat:
kubectl get clusterconfig cluster -o jsonpath='{.status.webhookReceivers}'Rotating CI_WEBHOOK_SECRET in OpenBao changes the path.
Use bao kv patch so the other keys survive, then update the
KARGO_WEBHOOK_URL organization secret on
k8s-services in Forgejo.
Keycloak is the OIDC provider
(realm internal, client kargo declared in
tofu/keycloak/internal.tf). Kargo is a
public client using authorization code with PKCE: its
config has no field for a client secret, so there is nothing to store.
Keycloak supports public PKCE clients directly, which is why no Dex
bridge is needed.
Access is scoped per Project, through two Keycloak groups per Project:
| Group | Kargo | ArgoCD |
|---|---|---|
k8s_<project>_maintainer |
promote and view | get, sync, action/*, logs on
*/<project>-* |
k8s_<project>_viewer |
view | get, logs on */<project>-* |
In Kargo each group is bound through an annotated ServiceAccount in
the Project namespace (rbac.kargo.akuity.io/claims), and
also listed under api.oidc.users in
charts/kargo/values.yaml so its members can list Projects
at all. api.oidc.admins is left unmapped on purpose: a
system-wide admin bypasses Project RBAC, and the break-glass account in
kargo-api-secret covers emergencies. ArgoCD binds the same
groups in charts/argocd/values.yaml.
The groups are declared in tofu/keycloak/internal.tf;
membership is managed in the Keycloak admin console.
Three separate surfaces, each served by whichever authority can issue for the name involved:
| Surface | Issuer |
|---|---|
kargo.yhkze.net API and webhooks |
host Traefik, ACME wildcard |
Kubernetes admission webhook (*.svc) |
cert-manager, self-signed Issuer |
Public CAs cannot issue for .svc names, which is the
whole reason cert-manager is deployed — reference
charts/cert-manager/values.yaml.
The API and external webhooks server listen on plain HTTP because
Traefik terminates TLS, but both set
tls.terminatedUpstream: true. That flag is required, not
cosmetic: without it Kargo builds URLs from its own listener and
advertises http://, which for the webhook receiver would
mean CI posting a shared secret in cleartext.
k8s-services
organization and copy .forgejo/workflows/ci.yml from
dummy unchanged — it derives the image path from the
repository name and reads every shared credential from the
organizationgitops/charts/<app>/
with values/values-test.yaml and
values-prod.yaml, including the ghcr-pull
ExternalSecret and imagePullSecrets from the dummy chart —
the ApplicationSet picks it up with no edit to
gitops/appsapps/promotion/<app>.yaml with the Project,
ProjectConfig, Warehouses, Stages and tier ServiceAccountsk8s_<app>_maintainer and
k8s_<app>_viewer to the groups in
tofu/keycloak/internal.tf and apply, then add them to
api.oidc.users in charts/kargo/values.yaml and
to the ArgoCD policy.csv