The backup that wrote to the wrong disk
A few weeks after I wrote about making backups consistent, I found out my backups had been succeeding in the worst possible way. The job ran, the log said the archive was stored, and the archive was on the wrong disk.
The setup
The backup is an offen/docker-volume-backup container that runs nightly. It archives a few services and writes a tarball to /archive inside the container. On the host, /archive is a bind mount of a directory where the NAS share is mounted over CIFS/SMB.
My home server powers off every night and I switch it on by hand in the morning. That detail matters, because it means the backup container is recreated every morning and races the network mount every morning.
What went wrong
A bind mount is resolved once, when the container starts. If the NAS is not mounted at that moment, the host path is just an ordinary empty directory on local disk. Docker is happy to bind it anyway, and if the directory does not exist, Docker will helpfully create it.
So the container wrote its archive to local disk, then logged Stored copy of backup ... in /archive. From the container's point of view that was true.
It happened twice. The first time, the NAS was unreachable and seven nightly runs in a row went to local disk, each with a clean log. That was a week with no copy of anything off the box. The second time was more interesting. The server booted with the NAS cable on a dead switch port, and the share did not mount until hours later. But the container had already started. Mounts that appear on the host later do not propagate into a running container's view of that bind, so the host looked healthy while the container kept writing locally. It reported success at the next scheduled run.
How I noticed
Not from the green log. The log looked identical to a good night, including a line that said storage=Local. I assumed for a while that this line meant "written to local disk." It does not. It is the tool's name for the local filesystem storage backend, as opposed to something like S3, and it appears on every correct run too. It says nothing about where the bytes went.
The line that does discriminate is the prune summary: None of N existing backups were pruned. Retention is 14 days, so the share should hold about 14 archives. The container reported two. A container looking at a different directory sees a different count. Two against fourteen was the whole case.
The fix
The bind source used to be the mountpoint itself. Now it is a subdirectory that exists only on the NAS share, and the compose file sets create_host_path: false:
- type: bind
source: /mnt/nas-backups/archive
target: /archive
bind:
create_host_path: false
When the share is not mounted, that subdirectory does not exist on local disk, and Docker refuses to start the container with bind source path does not exist. No container, no archive, no false confidence.
I tested both ways the container can start, because they are different code paths. docker compose up -d refused. So did docker start, which is the one that matters, because the morning power-on restarts the container through the Docker daemon and never touches Compose.
One trap I wrote into the comments in the compose file: do not create that subdirectory while the NAS is unmounted. That puts it on local disk, the guard passes, and every backup goes nowhere again. It would defeat the control silently and permanently.
Verifying at the destination
Same lesson as the green build that deployed nothing: a success message is the tool describing what it did, not what happened. Now I check the container's own view:
findmnt /mnt/nas-backups
docker exec backup grep ' /archive ' /proc/mounts
ls -l /mnt/nas-backups/archive | tail -3
The first should say cifs. The second should say cifs from inside the container, and it is the one that catches the second failure, where the host was fine and the container was not. The third shows file ownership. The share forces its own owner on every file, while a local write lands as root, so an archive owned by root is on the wrong disk.
What is still open
The failure is now loud but not announced. A container that refuses to start leaves a stopped container in docker ps -a and an absence of new archives, and nothing is watching for either. The tool's built-in notifications cannot help, because a container that never starts never sends anything.
The next step is a check from outside the container, such as a timer that complains when the newest archive is more than 48 hours old. I chose 48 rather than 24 because I switch the server on by hand and some mornings I do not. That check would catch every cause of a missing backup, not just this one. It is not built yet, so for now an absent backup is discoverable, not reported.