Disk Health Monitoring

Overview

smartd runs on every host and polls the SMART data of every attached disk, emailing an alert when a drive reports a failing attribute, an unreadable sector, or a temperature outside the configured band. Self-tests run on a schedule so problems surface before they are hit during normal use.

Alerts are relayed through msmtp to [email protected], the mailbox reserved for machine-generated mail.

flowchart LR
    D[("Disks")] -->|"SMART poll"| S["smartd"]
    S -->|"exec notify script"| N["smartd-notify.sh"]
    N -->|"stdin"| M["msmtp"]
    M -->|"port 587, STARTTLS"| X["mail.yhkze.net"]
    X --> B["[email protected]"]

Reference: modules/features/smartd.nix

Configuration

The module enables device autodetection, so any disk present at service start is picked up without being named explicitly.

Monitoring directives applied to every detected device:

Directive Effect
-a Monitor all attributes, health status and the error log
-n standby,q Skip a poll rather than spinning a sleeping HDD back up
-W 4,55,70 Report a 4 °C swing, warn at 55 °C, critical at 70 °C
-s (S/../.././02\|L/../../6/03) Short self-test daily at 02:00, long self-test Saturday at 03:00

The temperature band is set for NVMe, which idles considerably hotter than a SATA drive — 47 °C at idle is normal and a lower warning threshold produces constant false alarms.

Attribute history is written to /var/lib/smartmontools via --savestates and persisted. Without it, smartd would forget prior values on every boot and re-report settled conditions, since the root filesystem is a tmpfs.

Mail relay — smartd is pointed at the msmtp binary directly instead of services.mail.sendmailSetuidWrapper. The server already runs Postfix, which owns /run/wrappers/bin/sendmail; invoking msmtp by store path keeps one configuration working on every host without branching on hostname.

Secret

SMTP AUTH transmits the password so the server can hash and compare it. The existing mail/users/srv secret holds only the hash Dovecot verifies against, which is a verifier rather than a credential and cannot be replayed to authenticate — a separate plaintext secret is required.

Each host’s hosts/{host}/secrets/secrets.yaml therefore carries:

smtp:
  users:
    srv:

The account used is controlled by smtpAccount in the module, which derives both the address and the secret path. Changing it requires the matching smtp/users/<account> key in every host vault.

Verification

Confirm the service is running and enumerate what it decided to monitor:

systemctl is-active smartd
journalctl -u smartd | grep -E "Monitoring|Adding to|Temperature is"

Exercise the relay independently of smartd, which separates a credential or network problem from a smartd problem:

printf 'Subject: test\nFrom: [email protected]\nTo: [email protected]\n\ntest\n' \
    | sudo msmtp -i [email protected]
tail /var/log/msmtp.log

A successful send logs smtpstatus=250. An smtpstatus=535 is an authentication failure and points at the secret rather than at smartd.

Checking wear

smartd mails only when a threshold is breached. Read the counters directly to track remaining drive life between alerts.

List every device that can be queried:

sudo smartctl --scan

For an NVMe drive, these fields summarise health and consumed life:

sudo smartctl -a /dev/nvme0 | grep -iE \
    "Model Number|Percentage Used|Available Spare|Data Units Written|Media and Data|Unsafe Shut|Temperature:"
Field Meaning
Percentage Used Share of rated endurance consumed. Passes 100% at the rated TBW; drives usually keep working past it
Available Spare Reserve blocks left, against Available Spare Threshold. Dropping toward that threshold is serious
Data Units Written Total written, in units of 512000 bytes. smartctl prints a converted TB value alongside
Media and Data Integrity Errors Uncorrectable errors. Anything other than 0 warrants attention
Unsafe Shutdowns Power removed without a clean flush

SATA drives have no equivalent standard. Wear lives in numbered, vendor-defined attributes whose names differ between manufacturers, so a wide match is more reliable than naming specific ones:

sudo smartctl -A /dev/sda | grep -iE \
    "Wear|Lifetime|Reallocated|Host_Writes|Total_LBAs|NAND_GB|Available_Reserv|Grown_Bad|Power_On_Hours"

Names encountered across vendors for the same concepts:

Concept Attribute names
Life consumed Media_Wearout_Indicator, Wear_Leveling_Count, Percent_Lifetime_Remain, SSD_Life_Left
Host data written Host_Writes_GiB, Total_LBAs_Written
Actual flash written NAND_GB_Written_TLC, NAND_GB_Written_SLC
Spare capacity Available_Reservd_Space
Failing cells Reallocated_Sector_Ct, Grown_Bad_Blocks

Read the VALUE column, not the RAW_VALUE, for wear indicators: it is a normalised health score starting at 100 and counting down toward the THRESH column — the opposite direction from NVMe Percentage Used, which counts up from 0.

Divide flash writes by host writes to get the write amplification factor. A factor above roughly 3× means endurance is being consumed considerably faster than host write figures alone suggest.

Trusting these values — counters are reported by drive firmware and are not always accurate, particularly on OEM models, which sometimes return implausible figures for power-on hours or written volume. Cross-check any figure before acting on it rather than relying on a single counter.

Coverage

USB-attached storage is not monitored. SMART commands are not passed through by most USB bridges, so smartctl --scan does not list these devices and smartd cannot poll them. This affects the Ceph OSD device on the server, which is USB-attached.