Metrics, logs and traces from every host and from the Kubernetes cluster land on the server, where they are stored, evaluated for alerts and shown in Perses at https://hari.yhkze.net. Dashboards, datasources, access rules and alert rules are all code in this repository; nothing is configured by hand in a UI.
Every component is a foundation-governed open-source project without a paid edition: Prometheus, Thanos, Alertmanager, Jaeger, OpenTelemetry and Perses (CNCF), and OpenSearch (Linux Foundation). Senders and stores talk OTLP and PromQL, so any piece can be replaced without touching the others.
flowchart LR
subgraph hosts["NixOS hosts"]
NA["node_exporter and service exporters"] --> AG["OTel agent: journald, local scrapes, disk queue"]
end
subgraph cluster["k8s-cluster"]
KA["OTel agent DaemonSet: pod logs, kubelet, OTLP, OBI"]
KS["OTel scraper and target allocator: ServiceMonitors, events"]
end
AG -->|"OTLP, TLS, token"| GW
KA -->|"OTLP, TLS, token"| GW
KS -->|"OTLP, TLS, token"| GW
subgraph server["server"]
GW["OTel gateway, otlp.yhkze.net:4318 on wg0"] -->|"remote write"| P[("Prometheus, 3 days")]
P --> SC["Thanos sidecar"] --> B[("RGW bucket thanos, quota")]
B --> ST["Thanos store"] --> Q["Thanos query"]
P --> Q
CO["Thanos compactor"] --> B
P -->|"rules"| AM["Alertmanager"] -->|"mail"| M["[email protected]"]
GW -->|"logs-project-date"| OS[("OpenSearch")]
GW -->|"routed by project"| J["Jaeger per project"] --> OS
PE["Perses"] --> LP["prom-label-proxy per service"] --> Q
PE --> Q
PE --> OS
PE --> J
end
Versions in use (from the flake’s nixpkgs unless noted):
| Component | Version | Source |
|---|---|---|
| Prometheus | 3.12.0 | nixpkgs |
| Thanos | 0.40.1 | nixpkgs |
| Alertmanager | 0.31.1 | nixpkgs |
| OpenSearch | 3.5.0 | nixpkgs |
| OpenTelemetry Collector (contrib) | 0.151.0 | nixpkgs |
| Perses | 0.55.1 | values/packages/perses.nix (release tarball) |
| Jaeger | 2.20.0 | values/packages/jaeger.nix (release tarball); held
there because 2.21 dropped the query API the Perses Jaeger plugin
calls |
| prom-label-proxy | 0.16.2 | values/packages/prom-label-proxy.nix |
Every NixOS host that lists observability-agent.nix
(server and laptop) runs node_exporter and an OpenTelemetry Collector.
The collector reads the journal, scrapes the host’s exporters and
forwards everything to the gateway through a queue on disk, so a host
that loses the gateway (the laptop on the move, every host while the
server reboots) delivers what it collected once it reconnects.
A module that runs something worth scraping registers it next to its
own configuration, as postgresql.nix does:
services.opentelemetry-collector.settings.receivers."prometheus/local".config.scrape_configs = [
{
job_name = "postgres";
static_configs = [ { targets = [ "127.0.0.1:${toString port}" ]; } ];
}
];Local applications send their own traces to the agent on
127.0.0.1:14317 (gRPC) and 127.0.0.1:14318
(HTTP); Traefik and Keycloak do.
The cluster side is the observability chart in the
charts repository
(ssh://[email protected]:30354/gitops/charts.git):
| Component | Collects |
|---|---|
| OTel agent (DaemonSet) | Pod logs, kubelet and cAdvisor metrics of its node, OTLP from pods and OBI on host ports 4317/4318 |
| OTel scraper (StatefulSet, target allocator) | Every ServiceMonitor and PodMonitor target, cluster events |
| OBI | Traces and request metrics of every pod without code changes (eBPF) |
| kube-state-metrics, node-exporter | Object state and node metrics, through their ServiceMonitors |
A chart that ships a ServiceMonitor is scraped without any change here.
The gateway (observability-gateway.nix) is the only
telemetry endpoint another machine can reach. It listens on
10.100.0.1:4317/4318, presents the *.yhkze.net certificate
(senders map otlp.yhkze.net to 10.100.0.1 through a hosts
entry) and accepts a bearer token per sender.
It assigns every record a project from its Kubernetes namespace.
values/observability/projects.json lists the service
projects and their namespaces; anything else belongs to project
internal. Then it:
host_name,
namespace, obs_project)logs-<project>-<yyyy.MM.dd><project>-jaeger-*traces_span_metrics_*) and log lines into
log_records_total and log_records_error_total,
so logs and traces can raise alerts through Prometheus rules| Signal | Store | Retention |
|---|---|---|
| Metrics, recent | Prometheus on obs-prometheus |
3 days or 32 GB |
| Metrics, history | Thanos bucket thanos on Ceph RGW |
raw 90 days, 5-minute 1 year, 1-hour 2 years |
| Logs | OpenSearch on obs-opensearch |
90 days (ISM) |
| Traces | OpenSearch, through Jaeger | 14 days (ISM) |
Thanos query merges the sidecar (recent blocks) and the store gateway
(bucket) into one PromQL endpoint; every dashboard reads it. Retention
values live in values/observability.nix.
Each store sits on its own logical volume in VG pool, so
a store that runs away fills only its own filesystem, never
/persist or the OS. A missing volume does not stop the
boot: the services that need it carry RequiresMountsFor and
refuse to start instead of writing to the tmpfs root.
| Volume | Mount point | Size |
|---|---|---|
pool/obs-prometheus |
/var/lib/prometheus2 |
40 G |
pool/obs-thanos |
/var/lib/thanos-compact |
50 G |
pool/obs-opensearch |
/var/lib/opensearch |
200 G |
Prometheus — deletes its oldest blocks at
retention.size, 80% of its volume, as the Prometheus
storage documentation advises. Local compaction is off (two-hour
blocks), which the sidecar requires.
Thanos compactor — the only component that deletes from the bucket, and it applies retention only after a successful compaction. A compactor that halts on bad blocks keeps running and stops deleting, which is how a bucket fills without anything crashing. Three measures prevent that:
cluster="yhkze",
replica="server"), so no two sources upload overlapping
blocks, the usual cause of a haltThanosCompactHalted and
ThanosCompactStalled alerts, both criticalthanos (200 G), so a stuck
compactor cannot push Ceph to its full ratio, which would block RBD for
the cluster and IncusIts scratch directory is emptied on every start, which the Thanos documentation describes as safe.
OpenSearch — at 95% disk use (flood stage) indices turn read-only and turn writable again once use drops below 90%; queries keep working throughout. ISM deletes indices by age regardless of size.
Collectors — every queue is bounded. A full queue refuses new data instead of growing; nothing crashes because the gateway is away.
Every safeguard has an alert that fires before it has to act: volumes
predicted to fill within three days, the bucket at 80% of its quota,
Prometheus at 90% of retention.size, read-only OpenSearch
indices, collector queues at 80%.
Perses signs users in through Keycloak (realm internal,
client perses). Groups map to roles:
| Keycloak group | Perses access |
|---|---|
observability_admin |
Everything, every project |
observability_internal_maintainer |
Edit dashboards of project internal |
observability_internal_viewer |
View project internal |
k8s_<service>_maintainer |
Edit dashboards of project <service> |
k8s_<service>_viewer |
View project <service> |
Project internal covers the hosts, the server’s
services, the cluster as a whole and the observability stack. Signed-in
users without a mapped group see nothing. Group changes apply at the
next token refresh, at most 8 hours later.
A service project can only reach its own data, enforced outside Perses:
namespace=~"<service>-test|<service>-prod"
to every queryreader-<service>) may read
logs-<service>-* and nothing else<service>-jaeger-*Dashboards cannot embed datasources of their own
(datasource.disable_local), the datasource proxy only
reaches loopback, and editors cannot create datasources or secrets.
Every OpenSearch password is generated on the server by
observability-credentials.service into
/var/lib/observability-credentials; none is in git or
SOPS.
Prometheus evaluates the rules in
values/observability/rules/; Alertmanager mails
[email protected] through the local Postfix, critical
alerts every 4 hours while firing and warnings every 12. The
Alertmanager UI is at https://alerts.yhkze.net, reachable from WireGuard peers
after Keycloak sign-in.
Each rule file has a test file; run them after any change:
nix shell nixpkgs#prometheus.cli -c sh -c 'cd values/observability/rules && promtool test rules *_test.yaml'values/observability/dashboards/ is a Go program on the
Perses Go SDK. It reads projects.json and writes one JSON
file per dashboard to generated/, which
perses.nix provisions; edits made in the UI are
overwritten. Every dashboard opens with a plain-language guide
panel.
| Project | Dashboards |
|---|---|
internal |
overview, host, cluster, namespace, workload, Traefik, observability stack, logs, traces |
| each service | namespace, workload, logs, traces |
The workload breakdown relies on the recording rules in
rules/k8s.yaml, which map every pod to the Deployment,
StatefulSet, DaemonSet or Job that owns it.
Regenerate after changing the Go sources or
projects.json, lint, and commit the output:
cd values/observability/dashboards
nix shell nixpkgs#go -c go run . -out generatedpercli lint checks the result against the plugin schemas
bundled with Perses:
perses=$(nix build --no-link --print-out-paths --impure --expr \
'(import <nixpkgs> {}).callPackage ./values/packages/perses.nix {}')
plugins=$(mktemp -d)
for a in $perses/share/perses/plugins-archive/*.tar.gz; do
mkdir -p "$plugins/$(basename "$a" .tar.gz)" && tar -xzf "$a" -C "$plugins/$(basename "$a" .tar.gz)"
done
$perses/bin/percli lint -d values/observability/dashboards/generated --plugin.path "$plugins"values/observability/projects.jsonk8s_<service>_viewer and
k8s_<service>_maintainer to
keycloak_group.internal in
tofu/keycloak/internal.tf if they are new, then apply:
nix develop path:.#keycloak --command nu scripts/keycloak-tofu.nu applyrebuild 10.100.0.1The project, its datasources, its label proxy, its Jaeger and its
OpenSearch users are all derived from projects.json.
A service that uses the host MariaDB adds mariadb-host
to its namespaces: the label proxy then admits the database’s metrics
(and only those of the host) and the generator adds a Database dashboard
to the project.
The gateway and agent configuration are one collector config; after a
rebuild, journalctl -u opentelemetry-collector shows any
component that failed to start. Jaeger and the collectors refuse unknown
keys at start-up, so a typo surfaces in the rebuild output.
The first deployment needs these steps, in order.
Volumes — check free space in VG pool,
then create and format the volumes:
vgs pool
for lv in obs-prometheus:40G obs-thanos:50G obs-opensearch:200G; do
lvcreate -L "${lv#*:}" -n "${lv%%:*}" pool && mkfs.ext4 "/dev/pool/${lv%%:*}"
doneThanos bucket — create the RGW user with its quota
and the bucket. Store the generated key pair in SOPS as
observability/thanos-s3-access-key and
observability/thanos-s3-secret-key:
radosgw-admin user create --uid=thanos --display-name="Thanos"
radosgw-admin quota set --quota-scope=user --uid=thanos --max-size=200G
radosgw-admin quota enable --quota-scope=user --uid=thanosThe bucket is created with any S3 client using those keys, as for
actions-cache in Storage.
Secrets — add to
hosts/server/secrets/secrets.yaml:
observability:
# openssl rand -hex 32 (one per sender)
ingest-token:
ingest-token-laptop:
ingest-token-k8s:
thanos-s3-access-key:
thanos-s3-secret-key:
perses:
# openssl rand -hex 32; also the client secret Keycloak holds
oidc-client-secret:
# exactly 32 characters: openssl rand -base64 24
encryption-key:And to hosts/laptop/secrets/secrets.yaml, the same value
as ingest-token-laptop:
observability:
ingest-token:OpenBao — the cluster reads its token from KV mount
observability, key k8s, property
ingest-token (the value of ingest-token-k8s),
through a Kubernetes auth role bound to ServiceAccount
observability-eso in namespace observability,
as for Kargo in OpenBao.
Rollout — rebuild 10.100.0.1, then
rebuild on the laptop; apply Keycloak
(keycloak-tofu.nu apply, commit the state); add
hari.yhkze.net as DNS only (reference Cloudflare DNS); push the apps repository so
ArgoCD deploys the observability chart.