Observability

Overview

Metrics, logs and traces from every host and from the Kubernetes cluster land on the server, where they are stored, evaluated for alerts and shown in Perses at https://hari.yhkze.net. Dashboards, datasources, access rules and alert rules are all code in this repository; nothing is configured by hand in a UI.

Every component is a foundation-governed open-source project without a paid edition: Prometheus, Thanos, Alertmanager, Jaeger, OpenTelemetry and Perses (CNCF), and OpenSearch (Linux Foundation). Senders and stores talk OTLP and PromQL, so any piece can be replaced without touching the others.

flowchart LR
  subgraph hosts["NixOS hosts"]
    NA["node_exporter and service exporters"] --> AG["OTel agent: journald, local scrapes, disk queue"]
  end
  subgraph cluster["k8s-cluster"]
    KA["OTel agent DaemonSet: pod logs, kubelet, OTLP, OBI"]
    KS["OTel scraper and target allocator: ServiceMonitors, events"]
  end
  AG -->|"OTLP, TLS, token"| GW
  KA -->|"OTLP, TLS, token"| GW
  KS -->|"OTLP, TLS, token"| GW
  subgraph server["server"]
    GW["OTel gateway, otlp.yhkze.net:4318 on wg0"] -->|"remote write"| P[("Prometheus, 3 days")]
    P --> SC["Thanos sidecar"] --> B[("RGW bucket thanos, quota")]
    B --> ST["Thanos store"] --> Q["Thanos query"]
    P --> Q
    CO["Thanos compactor"] --> B
    P -->|"rules"| AM["Alertmanager"] -->|"mail"| M["[email protected]"]
    GW -->|"logs-project-date"| OS[("OpenSearch")]
    GW -->|"routed by project"| J["Jaeger per project"] --> OS
    PE["Perses"] --> LP["prom-label-proxy per service"] --> Q
    PE --> Q
    PE --> OS
    PE --> J
  end

Versions in use (from the flake’s nixpkgs unless noted):

Component Version Source
Prometheus 3.12.0 nixpkgs
Thanos 0.40.1 nixpkgs
Alertmanager 0.31.1 nixpkgs
OpenSearch 3.5.0 nixpkgs
OpenTelemetry Collector (contrib) 0.151.0 nixpkgs
Perses 0.55.1 values/packages/perses.nix (release tarball)
Jaeger 2.20.0 values/packages/jaeger.nix (release tarball); held there because 2.21 dropped the query API the Perses Jaeger plugin calls
prom-label-proxy 0.16.2 values/packages/prom-label-proxy.nix

Data Flow

Collection

Every NixOS host that lists observability-agent.nix (server and laptop) runs node_exporter and an OpenTelemetry Collector. The collector reads the journal, scrapes the host’s exporters and forwards everything to the gateway through a queue on disk, so a host that loses the gateway (the laptop on the move, every host while the server reboots) delivers what it collected once it reconnects.

A module that runs something worth scraping registers it next to its own configuration, as postgresql.nix does:

services.opentelemetry-collector.settings.receivers."prometheus/local".config.scrape_configs = [
  {
    job_name = "postgres";
    static_configs = [ { targets = [ "127.0.0.1:${toString port}" ]; } ];
  }
];

Local applications send their own traces to the agent on 127.0.0.1:14317 (gRPC) and 127.0.0.1:14318 (HTTP); Traefik and Keycloak do.

The cluster side is the observability chart in the charts repository (ssh://[email protected]:30354/gitops/charts.git):

Component Collects
OTel agent (DaemonSet) Pod logs, kubelet and cAdvisor metrics of its node, OTLP from pods and OBI on host ports 4317/4318
OTel scraper (StatefulSet, target allocator) Every ServiceMonitor and PodMonitor target, cluster events
OBI Traces and request metrics of every pod without code changes (eBPF)
kube-state-metrics, node-exporter Object state and node metrics, through their ServiceMonitors

A chart that ships a ServiceMonitor is scraped without any change here.

Gateway and Projects

The gateway (observability-gateway.nix) is the only telemetry endpoint another machine can reach. It listens on 10.100.0.1:4317/4318, presents the *.yhkze.net certificate (senders map otlp.yhkze.net to 10.100.0.1 through a hosts entry) and accepts a bearer token per sender.

It assigns every record a project from its Kubernetes namespace. values/observability/projects.json lists the service projects and their namespaces; anything else belongs to project internal. Then it:

Storage and Retention

Signal Store Retention
Metrics, recent Prometheus on obs-prometheus 3 days or 32 GB
Metrics, history Thanos bucket thanos on Ceph RGW raw 90 days, 5-minute 1 year, 1-hour 2 years
Logs OpenSearch on obs-opensearch 90 days (ISM)
Traces OpenSearch, through Jaeger 14 days (ISM)

Thanos query merges the sidecar (recent blocks) and the store gateway (bucket) into one PromQL endpoint; every dashboard reads it. Retention values live in values/observability.nix.

Graceful Degradation

Disk Isolation

Each store sits on its own logical volume in VG pool, so a store that runs away fills only its own filesystem, never /persist or the OS. A missing volume does not stop the boot: the services that need it carry RequiresMountsFor and refuse to start instead of writing to the tmpfs root.

Volume Mount point Size
pool/obs-prometheus /var/lib/prometheus2 40 G
pool/obs-thanos /var/lib/thanos-compact 50 G
pool/obs-opensearch /var/lib/opensearch 200 G

Safeguards

Prometheus — deletes its oldest blocks at retention.size, 80% of its volume, as the Prometheus storage documentation advises. Local compaction is off (two-hour blocks), which the sidecar requires.

Thanos compactor — the only component that deletes from the bucket, and it applies retention only after a successful compaction. A compactor that halts on bad blocks keeps running and stops deleting, which is how a bucket fills without anything crashing. Three measures prevent that:

Its scratch directory is emptied on every start, which the Thanos documentation describes as safe.

OpenSearch — at 95% disk use (flood stage) indices turn read-only and turn writable again once use drops below 90%; queries keep working throughout. ISM deletes indices by age regardless of size.

Collectors — every queue is bounded. A full queue refuses new data instead of growing; nothing crashes because the gateway is away.

Every safeguard has an alert that fires before it has to act: volumes predicted to fill within three days, the bucket at 80% of its quota, Prometheus at 90% of retention.size, read-only OpenSearch indices, collector queues at 80%.

Access

Tiers

Perses signs users in through Keycloak (realm internal, client perses). Groups map to roles:

Keycloak group Perses access
observability_admin Everything, every project
observability_internal_maintainer Edit dashboards of project internal
observability_internal_viewer View project internal
k8s_<service>_maintainer Edit dashboards of project <service>
k8s_<service>_viewer View project <service>

Project internal covers the hosts, the server’s services, the cluster as a whole and the observability stack. Signed-in users without a mapped group see nothing. Group changes apply at the next token refresh, at most 8 hours later.

Isolation

A service project can only reach its own data, enforced outside Perses:

Dashboards cannot embed datasources of their own (datasource.disable_local), the datasource proxy only reaches loopback, and editors cannot create datasources or secrets. Every OpenSearch password is generated on the server by observability-credentials.service into /var/lib/observability-credentials; none is in git or SOPS.

Alerting

Prometheus evaluates the rules in values/observability/rules/; Alertmanager mails [email protected] through the local Postfix, critical alerts every 4 hours while firing and warnings every 12. The Alertmanager UI is at https://alerts.yhkze.net, reachable from WireGuard peers after Keycloak sign-in.

Each rule file has a test file; run them after any change:

nix shell nixpkgs#prometheus.cli -c sh -c 'cd values/observability/rules && promtool test rules *_test.yaml'

Dashboards as Code

values/observability/dashboards/ is a Go program on the Perses Go SDK. It reads projects.json and writes one JSON file per dashboard to generated/, which perses.nix provisions; edits made in the UI are overwritten. Every dashboard opens with a plain-language guide panel.

Project Dashboards
internal overview, host, cluster, namespace, workload, Traefik, observability stack, logs, traces
each service namespace, workload, logs, traces

The workload breakdown relies on the recording rules in rules/k8s.yaml, which map every pod to the Deployment, StatefulSet, DaemonSet or Job that owns it.

Regenerate after changing the Go sources or projects.json, lint, and commit the output:

cd values/observability/dashboards
nix shell nixpkgs#go -c go run . -out generated

percli lint checks the result against the plugin schemas bundled with Perses:

perses=$(nix build --no-link --print-out-paths --impure --expr \
  '(import <nixpkgs> {}).callPackage ./values/packages/perses.nix {}')
plugins=$(mktemp -d)
for a in $perses/share/perses/plugins-archive/*.tar.gz; do
  mkdir -p "$plugins/$(basename "$a" .tar.gz)" && tar -xzf "$a" -C "$plugins/$(basename "$a" .tar.gz)"
done
$perses/bin/percli lint -d values/observability/dashboards/generated --plugin.path "$plugins"

Operations

Adding a Service

  1. Add the service and its namespaces to values/observability/projects.json
  2. Add k8s_<service>_viewer and k8s_<service>_maintainer to keycloak_group.internal in tofu/keycloak/internal.tf if they are new, then apply: nix develop path:.#keycloak --command nu scripts/keycloak-tofu.nu apply
  3. Regenerate the dashboards
  4. rebuild 10.100.0.1

The project, its datasources, its label proxy, its Jaeger and its OpenSearch users are all derived from projects.json.

A service that uses the host MariaDB adds mariadb-host to its namespaces: the label proxy then admits the database’s metrics (and only those of the host) and the generator adds a Database dashboard to the project.

Checking a Change Before Deploying

The gateway and agent configuration are one collector config; after a rebuild, journalctl -u opentelemetry-collector shows any component that failed to start. Jaeger and the collectors refuse unknown keys at start-up, so a typo surfaces in the rebuild output.

Deployment

The first deployment needs these steps, in order.

Volumes — check free space in VG pool, then create and format the volumes:

vgs pool
for lv in obs-prometheus:40G obs-thanos:50G obs-opensearch:200G; do
  lvcreate -L "${lv#*:}" -n "${lv%%:*}" pool && mkfs.ext4 "/dev/pool/${lv%%:*}"
done

Thanos bucket — create the RGW user with its quota and the bucket. Store the generated key pair in SOPS as observability/thanos-s3-access-key and observability/thanos-s3-secret-key:

radosgw-admin user create --uid=thanos --display-name="Thanos"
radosgw-admin quota set --quota-scope=user --uid=thanos --max-size=200G
radosgw-admin quota enable --quota-scope=user --uid=thanos

The bucket is created with any S3 client using those keys, as for actions-cache in Storage.

Secrets — add to hosts/server/secrets/secrets.yaml:

observability:
  # openssl rand -hex 32 (one per sender)
  ingest-token:
  ingest-token-laptop:
  ingest-token-k8s:
  thanos-s3-access-key:
  thanos-s3-secret-key:
perses:
  # openssl rand -hex 32; also the client secret Keycloak holds
  oidc-client-secret:
  # exactly 32 characters: openssl rand -base64 24
  encryption-key:

And to hosts/laptop/secrets/secrets.yaml, the same value as ingest-token-laptop:

observability:
  ingest-token:

OpenBao — the cluster reads its token from KV mount observability, key k8s, property ingest-token (the value of ingest-token-k8s), through a Kubernetes auth role bound to ServiceAccount observability-eso in namespace observability, as for Kargo in OpenBao.

Rollout — rebuild 10.100.0.1, then rebuild on the laptop; apply Keycloak (keycloak-tofu.nu apply, commit the state); add hari.yhkze.net as DNS only (reference Cloudflare DNS); push the apps repository so ArgoCD deploys the observability chart.