A log that fed itself to 94 GB
On the morning I fixed my Valheim server's failing backups, my fix created a worse problem. Within about three hours a single log file had grown to 94 GB, and the server had stopped being able to tell me whether anyone was playing. This is the story of that loop. The server itself is described in the overview post.
The backups were failing
The server ran in a container on Azure, with its state on Azure Files shares. A tool called Odin took a tarball of the saves directory every 15 minutes. Its job state showed 67 failures in 490 runs, a 13.7 percent failure rate, and the failures clustered when the server was busiest.
The cause was a boot log. I tee'd everything the container printed at startup into a file, _boot.log, because that was the only reliable way to see why a start had gone wrong. The file lived inside the saves directory, so tar was trying to archive a file that was still being appended to. When tar hit that, it abandoned the whole archive.
The obvious fix was to move the log somewhere that was not tarred. I picked the server's own logs directory on a different share, deployed it, and watched backups go green.
The symptom
I did not catch the next problem with a monitor. About three hours later someone said the server felt strange.
When I looked, three things were wrong at once:
- CPU sat at a sustained 1,100 to 1,250 millicores. A normal two-player session peaks around 913, so the container was burning more than a full core on something.
- Huginn, the small status service that auto-stop uses to read the player count, had stopped answering
/status. A player was connected and my cost-saving layer could not see them. - The log file was 94 GB, against a 100 GB quota on a share that also held the game install and the mod config.
The mechanism
The logs directory I had picked was not neutral. Odin tails everything in it. It reads each file there and echoes the contents to its own stdout.
And stdout was exactly what I was tee'ing into the file.
So the loop went like this. Odin reads a line from _boot.log and prints it. The tee captures that output and appends it to _boot.log. Odin reads the new line, and prints it again. Each pass also re-wrapped the previous line in its own ANSI color prefix, so every line was longer than the one before it, and the file grew exponentially rather than linearly. I measured the growth at roughly 550 MB a minute.
Nothing in the loop was failing. Every process was doing precisely what it was configured to do. The CPU was spent copying a log into itself, and Huginn was starved enough that it stopped answering.
If you manage Windows servers, think of an event log subscription that forwards events to a collector, where the collector is also logging into the channel it subscribes to. Each event produces another event. The difference is that nothing here had a rate limit.
The fix
I did not move it again. I gave the log its own share, mounted at a path outside both the saves directory and Odin's directory. Nothing tars it, nothing tails it, and nothing prunes it.
I checked the result after the deploy. Steady state was 2,052 bytes a minute, about 1.6 MB across a 14 hour day, against roughly 550 MB a minute during the loop. CPU averaged 137 millicores. All 19 mods loaded, Huginn answered, and backups succeeded.
Then I deleted the 94 GB file and the stale copy still sitting in the saves share, and set the share quota back to 100 GiB on purpose. With the loop gone, a small quota is a circuit breaker rather than a constraint. If something like this happens again, the share fills and the write fails, instead of the problem running unnoticed.
I also changed my log-download helper to check the file size first and refuse anything over 64 MB. It would otherwise have started pulling 94 GB over the network.
What I took from it
Where you write a log is an architecture decision. I had treated it as a convenience, a place to put a file, and I picked the location by asking only what would stop the backup from failing. I never asked what else already reads that directory.
The rule I wrote down is simple: never log into a directory that something else reads and re-emits. That applies to anything that tails, ships, indexes, or archives a path, and the tool that does it may be one you did not set up yourself.
The second lesson is about verification. After the first deploy the backups did succeed, so the thing I was fixing had stopped failing. By that test the change was verified. It had also been burning a core for three hours. Checking that the original symptom is gone is not enough, and I now look at CPU, disk growth and the status endpoint after any change that touches where files go.
I also left a comment in the infrastructure file listing both failed homes for the log and why each failed, so nobody simplifies it back, including me.