GitOps and CI/CD

Repositories

Two Forgejo organizations split the repositories by role: gitops holds what describes the cluster, and k8s-services holds one source repository per deployed service.

Repo Purpose
gitops/apps ArgoCD Application and ApplicationSet definitions
gitops/charts One Helm chart directory per deployable, values per environment
k8s-services/<app> Application source plus .forgejo/workflows/ci.yml

ArgoCD reaches both gitops repos over Forgejo’s SSH server:

ssh://[email protected]:30354/gitops/apps.git
ssh://[email protected]:30354/gitops/charts.git

ssh-git.yhkze.net is the unproxied DNS record — the Cloudflare-proxied name cannot carry SSH.

Repository credentials — ArgoCD authenticates as a dedicated restricted Forgejo machine account, not as the human admin account. The account holds an SSH deploy key and read-only team access to both repos, so a compromise of the cluster yields no write path into git. Forgejo resolves the account from the key itself, which is why the URL user is always the literal forgejo.

Delivery Model

gitops/apps is applied by a single root Application that recurses the whole repo and excludes only its own bootstrap file, so ArgoCD self-manages every other Application and ApplicationSet after the one manual apply.

flowchart LR
    rootfile["root-app.yaml<br/>(applied once by hand)"] --> root[root-applications]
    root --> std["Standalone Applications<br/>cert-manager · external-secrets · kargo<br/>ceph-csi-rbd · local-path-provisioner<br/>cloudnative-pg · monitoring · forgejo-runner"]
    root --> appset["ApplicationSet<br/>development-services"]
    appset --> svc["&lt;service&gt;-test<br/>&lt;service&gt;-prod"]
    std --> charts[(gitops/charts)]
    svc --> charts

Platform components are standalone Applications because they are singletons with no test/prod split, and each is explicitly excluded from the ApplicationSet’s git generator so it is not picked up twice. All of them sync automatically with prune and selfHeal.

The development-services ApplicationSet is a matrix of a git directory generator over gitops/charts and a list generator of test and prod. A new chart directory therefore yields <service>-test and <service>-prod Applications, each deployed into its own <service>-<env> namespace with values/values-<env>.yaml layered over the chart’s base values. Adding a service needs no edit to gitops/apps.

Chart layout expected by the generator:

gitops/charts/
  <service>/
    Chart.yaml
    values.yaml                  # base values: image.repository, service.port, …
    values/
      values-test.yaml           # domain, replicas, image tag
      values-prod.yaml
    templates/
      _helpers.tpl               # must live here, not at chart root
      deployment.yaml
      service.yaml
      ingress.yaml               # IngressRoute CRD for the host Traefik

Helm only loads helper templates from templates/. A _helpers.tpl at the chart root is silently ignored, and every template referencing it fails to render.

Pipeline

Forgejo Actions runs the pipeline on runners inside the cluster. CI only builds and publishes an image; it never edits the charts repo and never talks to the Kubernetes API. Advancing an image through environments is Kargo’s job.

push → build-push (GHCR) → notify Kargo → Warehouse → Freight → Promotion → ArgoCD sync

Trunk-based, so the tag shape decides the environment rather than the branch name:

Env Trigger Image tag Domain
test push to main mutable main <app>-test.yhkze.net
prod push of a vX.Y.Z tag immutable vX.Y.Z <app>.yhkze.net

Each environment has its own Warehouse subscribed to the matching tags, so a release tag is the only thing that can produce prod Freight.

GHCR cannot call into Forgejo, so nothing would otherwise tell Kargo an image landed — the Warehouses would only notice when their own five-minute interval elapsed. The final CI step POSTs to Kargo’s webhook receiver to collapse that delay. The receiver URL contains a hard-to-guess path derived from CI_WEBHOOK_SECRET, so treat the whole URL as a credential:

kubectl get clusterconfig cluster -o jsonpath='{.status.webhookReceivers}'

Shared CI Configuration

Configuration every service needs is set once on the k8s-services organization and inherited by each repository in it:

Name Kind Purpose
GHCR_PUSH_TOKEN secret classic GitHub PAT, write:packages
GHCR_PUSH_USERNAME variable GitHub account owning that PAT
GHCR_NAMESPACE variable image prefix, ghcr.io/<namespace>/<repo>
KARGO_WEBHOOK_URL secret Kargo receiver; the URL is itself a credential

The username is a variable rather than a secret because it is not sensitive, and Forgejo masks secret values in logs — as a secret it would blank the image path out of every build log.

Because the image path is built from GHCR_NAMESPACE and the repository name, ci.yml contains nothing service-specific and copies between repositories unchanged. Repository-level secrets hold only what that one service’s CI needs.

Registry credentials are split by direction. The push token lives only in Forgejo, since only CI pushes. The pull token lives in OpenBao at registry/ghcr-pull, since its consumers — Kargo discovery and kubelet image pulls — run inside the cluster. No git token is needed either: Kargo pushes promotion commits with its own credential. Reference OpenBao Secret Management.

Runners

The runner lives in the ci namespace, deployed by ArgoCD from the forgejo-runner chart in gitops/charts, with a Docker-in-Docker sidecar for image builds and an HPA scaling 1-4 replicas on CPU. Concurrency is runner.capacity jobs per replica, so the ceiling is capacity times maxReplicas.

Runner 12.10.1 uses config-based connections; the register command is gone. The config hands the Docker TLS material to job containers so docker/build-push-action works: the runner shares a /certs emptyDir with DinD, job containers get -v /certs/client:/certs/client through container.options and DOCKER_HOST=tcp://localhost:2376 through runner.envs, and container.network: host lets them reach the daemon on localhost. The pattern follows code.forgejo.org/forgejo/runner/examples/kubernetes/dind-docker.yaml.

The runner’s credential is held out of band and is absent from the chart, so the chart is always safe to re-sync. Create a runner under Site Admin → Actions → Runners → Create New Runner, then store the token:

kubectl -n ci create secret generic runner-secret --from-literal=token='<TOKEN>'

The UUID that registration returns is the runner’s identity rather than a credential, so it lives in values.yaml as runner.uuid. Set it there and let ArgoCD sync; a mismatch makes the daemon register as a second, separate runner instead of reclaiming the existing one.