smartd runs on every host and polls the SMART data of
every attached disk, emailing an alert when a drive reports a failing
attribute, an unreadable sector, or a temperature outside the configured
band. Self-tests run on a schedule so problems surface before they are
hit during normal use.
Alerts are relayed through msmtp to
[email protected], the mailbox reserved for machine-generated
mail.
flowchart LR
D[("Disks")] -->|"SMART poll"| S["smartd"]
S -->|"exec notify script"| N["smartd-notify.sh"]
N -->|"stdin"| M["msmtp"]
M -->|"port 587, STARTTLS"| X["mail.yhkze.net"]
X --> B["[email protected]"]
Reference: modules/features/smartd.nix
The module enables device autodetection, so any disk present at service start is picked up without being named explicitly.
Monitoring directives applied to every detected device:
| Directive | Effect |
|---|---|
-a |
Monitor all attributes, health status and the error log |
-n standby,q |
Skip a poll rather than spinning a sleeping HDD back up |
-W 4,55,70 |
Report a 4 °C swing, warn at 55 °C, critical at 70 °C |
-s (S/../.././02\|L/../../6/03) |
Short self-test daily at 02:00, long self-test Saturday at 03:00 |
The temperature band is set for NVMe, which idles considerably hotter than a SATA drive — 47 °C at idle is normal and a lower warning threshold produces constant false alarms.
Attribute history is written to /var/lib/smartmontools
via --savestates and persisted. Without it, smartd would
forget prior values on every boot and re-report settled conditions,
since the root filesystem is a tmpfs.
Mail relay — smartd is pointed at the
msmtp binary directly instead of
services.mail.sendmailSetuidWrapper. The server already
runs Postfix, which owns /run/wrappers/bin/sendmail;
invoking msmtp by store path keeps one configuration working on every
host without branching on hostname.
SMTP AUTH transmits the password so the server can hash and compare
it. The existing mail/users/srv secret holds only the hash
Dovecot verifies against, which is a verifier rather than a credential
and cannot be replayed to authenticate — a separate plaintext secret is
required.
Each host’s hosts/{host}/secrets/secrets.yaml therefore
carries:
smtp:
users:
srv:The account used is controlled by smtpAccount in the
module, which derives both the address and the secret path. Changing it
requires the matching smtp/users/<account> key in
every host vault.
Confirm the service is running and enumerate what it decided to monitor:
systemctl is-active smartd
journalctl -u smartd | grep -E "Monitoring|Adding to|Temperature is"
Exercise the relay independently of smartd, which separates a credential or network problem from a smartd problem:
printf 'Subject: test\nFrom: [email protected]\nTo: [email protected]\n\ntest\n' \
| sudo msmtp -i [email protected]
tail /var/log/msmtp.log
A successful send logs smtpstatus=250. An
smtpstatus=535 is an authentication failure and points at
the secret rather than at smartd.
smartd mails only when a threshold is breached. Read the counters directly to track remaining drive life between alerts.
List every device that can be queried:
sudo smartctl --scan
For an NVMe drive, these fields summarise health and consumed life:
sudo smartctl -a /dev/nvme0 | grep -iE \
"Model Number|Percentage Used|Available Spare|Data Units Written|Media and Data|Unsafe Shut|Temperature:"
| Field | Meaning |
|---|---|
Percentage Used |
Share of rated endurance consumed. Passes 100% at the rated TBW; drives usually keep working past it |
Available Spare |
Reserve blocks left, against Available Spare Threshold.
Dropping toward that threshold is serious |
Data Units Written |
Total written, in units of 512000 bytes. smartctl prints a converted TB value alongside |
Media and Data Integrity Errors |
Uncorrectable errors. Anything other than 0 warrants attention |
Unsafe Shutdowns |
Power removed without a clean flush |
SATA drives have no equivalent standard. Wear lives in numbered, vendor-defined attributes whose names differ between manufacturers, so a wide match is more reliable than naming specific ones:
sudo smartctl -A /dev/sda | grep -iE \
"Wear|Lifetime|Reallocated|Host_Writes|Total_LBAs|NAND_GB|Available_Reserv|Grown_Bad|Power_On_Hours"
Names encountered across vendors for the same concepts:
| Concept | Attribute names |
|---|---|
| Life consumed | Media_Wearout_Indicator,
Wear_Leveling_Count, Percent_Lifetime_Remain,
SSD_Life_Left |
| Host data written | Host_Writes_GiB, Total_LBAs_Written |
| Actual flash written | NAND_GB_Written_TLC,
NAND_GB_Written_SLC |
| Spare capacity | Available_Reservd_Space |
| Failing cells | Reallocated_Sector_Ct,
Grown_Bad_Blocks |
Read the VALUE column, not the RAW_VALUE,
for wear indicators: it is a normalised health score starting at 100 and
counting down toward the THRESH column — the
opposite direction from NVMe Percentage Used, which counts
up from 0.
Divide flash writes by host writes to get the write amplification factor. A factor above roughly 3× means endurance is being consumed considerably faster than host write figures alone suggest.
Trusting these values — counters are reported by drive firmware and are not always accurate, particularly on OEM models, which sometimes return implausible figures for power-on hours or written volume. Cross-check any figure before acting on it rather than relying on a single counter.
USB-attached storage is not monitored. SMART commands are not passed
through by most USB bridges, so smartctl --scan does not
list these devices and smartd cannot poll them. This affects the Ceph
OSD device on the server, which is USB-attached.