Log Rotation

Every host caps log growth in two places: systemd-journald for unit output, and logrotate for services that write their own file. Both are configured in modules/features/logging.nix.

An unbounded log file is a slow disk-full outage — it fails at the moment the filesystem fills, not when the log was written, so the service that breaks is rarely the one at fault.

Journal

The journal is capped by size and age rather than left to the default, which scales with the filesystem and so grows as the disk grows:

SystemMaxUse=1G
SystemMaxFileSize=128M
MaxRetentionSec=1month

Check current usage:

journalctl --disk-usage

Service Logs

logging.nix enables logrotate and sets shared defaults (compression, dated suffixes, 7 rotations). Services that write to their own file declare rotation in their own module, next to the option that creates the file.

Traefik is the example worth copying, because it reopens its log files on USR1. The file can therefore be renamed and a fresh one created, rather than truncated underneath a process that is still writing to an offset:

services.logrotate.settings."/var/lib/traefik/access.log" = {
  su = "traefik traefik";
  size = "128M";
  rotate = 5;
  postrotate = "systemctl kill -s USR1 traefik.service";
  delaycompress = true;
};

delaycompress is required alongside a signal-based reopen. Compressing in the same pass races the signal: the service keeps appending to the renamed file until it handles USR1, and gzip fails the run with file size changed while zipping.

A service that cannot reopen its log needs copytruncate instead, which accepts a small window where lines written between the copy and the truncate are lost.

Verification

Logrotate runs daily via logrotate.timer. Check what it would do without touching anything:

sudo logrotate --debug /etc/logrotate.conf

Find whatever is growing without a bound:

sudo du -ah /var/log /var/lib --exclude=/var/lib/incus 2>/dev/null \
  | sort -rh | head -20

References