Kargo Promotion Control Plane

Overview

Kargo decides which image version each environment runs. ArgoCD still applies manifests and CI still builds images, but neither chooses a version: CI publishes to GHCR and stops, and ArgoCD syncs whatever git says. Kargo sits between them, observing the registry and writing the chosen version into the charts repo.

URL: kargo.yhkze.net (Keycloak OIDC)

The split matters because it keeps CI out of the cluster and out of the charts repo. A build job needs no git token and no Kubernetes credential — reference GitOps and CI/CD.

Concepts

Four objects carry the whole model:

The sequence below follows one build from push to deployment. CI’s steps run in order within one job, so the image is in GHCR before Kargo is notified:

sequenceDiagram
    participant CI as Forgejo Actions
    participant G as GHCR
    participant R as Kargo receiver
    participant W as Warehouse
    participant S as Stage
    participant C as gitops/charts
    participant A as ArgoCD

    CI->>G: push layers, PUT manifest
    G-->>CI: 201 Created
    CI->>R: POST {"repository":{"url":"ghcr.io/…/<app>"}}
    R->>W: refresh Warehouses subscribed to that image
    W->>G: list tags, resolve digest
    W->>S: new Freight
    S->>C: commit digest to values-<env>.yaml, push
    S->>A: argocd-wait until healthy
    A->>C: fetch, render, sync <app>-<env>

A lost or early webhook costs only latency: every Warehouse also polls on its own five-minute interval.

Project Layout

One Kargo Project per application, declared in apps/promotion/<app>.yaml. A Project is also a namespace, so its Warehouses, Stages and credentials are scoped to it.

The dummy project is the worked example and the template for real ones:

Object Selection strategy Source
warehouse/test Digest ghcr.io/yuuhikaze/dummy, tag main
warehouse/prod SemVer ghcr.io/yuuhikaze/dummy, allowTagsRegexes

Two Warehouses rather than one, because the strategies differ in kind. The test Warehouse tracks a mutable tag, so it must follow the digest to notice that main now points somewhere else. The prod Warehouse tracks immutable release tags and picks the highest version matching the regex. A release tag is therefore the only thing that can produce prod Freight — an accidental push to main cannot reach production.

ProjectConfig carries the promotion policies, each matching a Stage by stageSelector.

Promotion Steps

Each Stage runs the same sequence, differing only in which values file it edits:

  1. git-clone — the charts repo over HTTPS, at branch: master
  2. yaml-update — write the new image digest into values/values-<env>.yaml
  3. git-commit — attribute to Freight, so history records what moved and why
  4. git-push
  5. argocd-wait — block until ${{ ctx.project }}-${{ ctx.stage }} is healthy

The final step is what makes a promotion mean “deployed” rather than “committed”: without it a Promotion succeeds the moment git accepts the push, regardless of whether the workload ever became healthy.

Each Stage has its own ServiceAccount, so promotion permissions are scoped per environment rather than shared project-wide.

Credentials

Kargo discovers credentials by label, not by name — kargo.akuity.io/cred-type on a Secret in the project or shared-resources namespace. The label is load bearing; renaming a Secret is safe, dropping its label is not.

All of them come from OpenBao through External Secrets, so none is hand-created. Reference OpenBao Secret Management.

Secret Namespace cred-type Purpose
kargo-api-secret kargo — admin password hash, token signing key
kargo-ci-webhook kargo-system-resources generic obscures receiver URL
charts-repo kargo-shared-resources git push promotion commits
ghcr-pull kargo-shared-resources image discover private GHCR images

The git credential is HTTPS with a username and token, not the SSH URL ArgoCD uses — a token pair only authenticates over HTTPS. Its repoURL must match the git-clone URL exactly, since Kargo matches credentials to repositories by URL.

ghcr-pull matches ^ghcr\.io/yuuhikaze/.* as a regex, so one Secret covers every image under the namespace. It reads the read-only token from the cluster-wide openbao-registry store, which also supplies kubelet pull secrets in app namespaces. Without it a Warehouse reports DENIED: requested access to the resource is denied from the GHCR token endpoint.

Webhook Receiver

GHCR cannot call into this cluster and the source repository lives on Forgejo, so nothing would otherwise tell Kargo an image landed — Warehouses would only notice at their own five-minute interval.

ClusterConfig declares a generic receiver named ci, with its shared secret in kargo-ci-webhook. A ClusterConfig receiver acts on Warehouses across all Projects, and this one selects them by the image named in the request body:

indexSelector:
  matchIndices:
    - key: subscribedURLs
      operator: Equals
      value: ${{ normalizeImage(request.body.repository.url) }}

A push therefore refreshes only the Warehouses subscribed to that image. That makes one receiver URL safe to share across every service repository, which is why it lives as a single organization secret in Forgejo.

Kargo derives a hard-to-guess path from that secret, so the receiver URL is itself a credential — treat it like one and do not paste it into issues or chat:

kubectl get clusterconfig cluster -o jsonpath='{.status.webhookReceivers}'

Rotating CI_WEBHOOK_SECRET in OpenBao changes the path. Use bao kv patch so the other keys survive, then update the KARGO_WEBHOOK_URL organization secret on k8s-services in Forgejo.

Authentication

Keycloak is the OIDC provider (realm internal, client kargo declared in tofu/keycloak/internal.tf). Kargo is a public client using authorization code with PKCE: its config has no field for a client secret, so there is nothing to store. Keycloak supports public PKCE clients directly, which is why no Dex bridge is needed.

Access is scoped per Project, through two Keycloak groups per Project:

Group Kargo ArgoCD
k8s_<project>_maintainer promote and view get, sync, action/*, logs on */<project>-*
k8s_<project>_viewer view get, logs on */<project>-*

In Kargo each group is bound through an annotated ServiceAccount in the Project namespace (rbac.kargo.akuity.io/claims), and also listed under api.oidc.users in charts/kargo/values.yaml so its members can list Projects at all. api.oidc.admins is left unmapped on purpose: a system-wide admin bypasses Project RBAC, and the break-glass account in kargo-api-secret covers emergencies. ArgoCD binds the same groups in charts/argocd/values.yaml.

The groups are declared in tofu/keycloak/internal.tf; membership is managed in the Keycloak admin console.

TLS

Three separate surfaces, each served by whichever authority can issue for the name involved:

Surface Issuer
kargo.yhkze.net API and webhooks host Traefik, ACME wildcard
Kubernetes admission webhook (*.svc) cert-manager, self-signed Issuer

Public CAs cannot issue for .svc names, which is the whole reason cert-manager is deployed — reference charts/cert-manager/values.yaml.

The API and external webhooks server listen on plain HTTP because Traefik terminates TLS, but both set tls.terminatedUpstream: true. That flag is required, not cosmetic: without it Kargo builds URLs from its own listener and advertises http://, which for the webhook receiver would mean CI posting a shared secret in cleartext.

Adding an Application

  1. Create the source repository in the k8s-services organization and copy .forgejo/workflows/ci.yml from dummy unchanged — it derives the image path from the repository name and reads every shared credential from the organization
  2. Add a chart directory under gitops/charts/<app>/ with values/values-test.yaml and values-prod.yaml, including the ghcr-pull ExternalSecret and imagePullSecrets from the dummy chart — the ApplicationSet picks it up with no edit to gitops/apps
  3. Add apps/promotion/<app>.yaml with the Project, ProjectConfig, Warehouses, Stages and tier ServiceAccounts
  4. Add k8s_<app>_maintainer and k8s_<app>_viewer to the groups in tofu/keycloak/internal.tf and apply, then add them to api.oidc.users in charts/kargo/values.yaml and to the ArgoCD policy.csv
  5. Set repository secrets only for what this service’s own CI needs