Backups that are actually consistent
The default home lab backup is a nightly job that copies container volumes somewhere else. It runs, it produces files, the files have plausible sizes, and everyone moves on.
The problem is that "the backup ran" and "the backup can be restored" are different claims, and copying a volume while the service is writing to it is a good way to satisfy the first without the second.
Why copying a live volume is not enough
A database that is mid-write when you copy its files gives you a snapshot of an inconsistent state. Some pages updated, some not, a write-ahead log that does not match. It restores into something that may open, may open and be subtly wrong, or may not open at all.
You will not find out which at backup time. Everything looks fine at backup time. You find out during a restore, which by definition is a day when you already have a problem.
So the useful question is not "am I copying the data." It is "what does this specific service need in order for the copy to be restorable," and the answer is different per service.
Four services, four answers
The backup runs nightly against four containers, and each one is handled differently.
Git server, SQLite backed. SQLite does not tolerate being copied mid-transaction. The backup tool stops the container for the few seconds it takes to archive the data directory, then starts it again. Brief downtime, guaranteed clean copy. Acceptable because nothing is using it at one in the morning.
SQL Server. Stopping this one is unnecessary, because the database engine already knows how
to produce a consistent copy of itself. A pre-backup hook runs BACKUP DATABASE to write a
.bak file, and the archive picks up that file rather than the live data files. No downtime at
all, and the resulting artifact is the format the restore procedure actually expects.
This is the pattern to reach for whenever it exists: let the application make the consistent copy, then back up its output. You are not smarter about that database's internals than the database is.
Media server. Backed up live, with exclusions. The library database and preferences are kept. The metadata, artwork, and transcode caches are excluded, because they are large, constantly changing, and fully regenerable from the media itself. Excluding them turns a multi-gigabyte archive into a small one and removes most of the churn that would otherwise cause an inconsistent copy.
DNS server. Also live, also with exclusions. The configuration, block lists, and custom records are kept. The query statistics database is excluded, because it is volatile, constantly written, and I do not care about restoring last Tuesday's query graph.
The shape across all four: identify what is genuinely irreplaceable, exclude what regenerates, and give the things that cannot be copied safely a mechanism that makes them safe.
The scheduling detail I nearly got wrong
The server powers itself off overnight to save electricity. The backup runs at one in the morning.
Those two facts were set at different times, for different reasons, and the shutdown was originally earlier. Which would have meant the backup either never ran, or was killed partway through, producing a truncated archive that still looked like a file.
Moving the shutdown to give the backup room to finish is a one-line change and completely invisible if you never think to check. It is the kind of interaction that only exists because two independent pieces of automation share a machine, and neither one knows about the other.
Worth auditing generally: anything scheduled on a host that also has scheduled state changes. Reboots, shutdowns, updates, other jobs. The individual pieces can each be correct and the combination still broken.
Deliberate omissions
Two things are not backed up, on purpose, and both were decisions rather than oversights.
The media library is not included. It already lives on the NAS that the backups are written to, so backing it up there would be copying data onto the volume it is already on. That is not a backup, it is a duplicate.
Encryption is off. The archives sit on a NAS in my house, and the realistic failure mode for a home lab is not someone stealing the NAS, it is me needing a restore in two years having forgotten a passphrase. An encrypted backup you cannot open is worse than an unencrypted one you can. That calculus changes completely the moment a copy goes off-site, and the option is staged and ready for when it does.
Both of those are worth writing down rather than just doing, because six months from now the absence of encryption looks like a mistake unless the reasoning is recorded next to it.
Test the restore
Everything above is theory until you have pulled an archive back and rebuilt a service from it.
I have documented the restore steps per service.That is the honest state of it. A backup you have never restored is a hypothesis.
The good news is that a lab is exactly where you get to test that cheaply. Nothing here is production, and the cost of finding out that a restore procedure is wrong is an afternoon rather than an incident.