The OOM Kill That Was Really a Mount Layout

One of the media apps in my home lab, Radarr, kept restarting. When I finally counted, it had been OOM-killed 25 times in about two and a half weeks. It ran on Kubernetes with a 512 MiB memory limit, which is plenty for an app whose heap sits well under that. The obvious fix was to raise the limit and move on.

Raising the limit did stop the kills. It also would have hidden the actual problem, which had nothing to do with how much memory the app uses and everything to do with how I’d mounted its directories.

The kills didn’t match the app

The pod’s memory graphs were unhelpful. Working set looked moderate, nowhere near the limit, and then the container would be gone. That mismatch is the first clue: if the usual metrics say the app is fine and the kernel still kills it, the kernel is counting something the dashboard isn’t showing.

So I read the kernel’s own OOM reports. journalctl -k on the node prints a memcg breakdown for every kill. (Gotcha: it only covers the current boot, so add -b -1 if the node rebooted since, or the older kills simply aren’t there.)

The breakdown for each kill looked roughly like this:

memory: usage 524288kB, limit 524288kB
...
file_dirty       220 MB
file_writeback   108 MB

Of the 512 MiB limit, about 330 MB was dirty page cache and pages being written back. The app’s own heap (anon) was only a modest slice of the rest. The app wasn’t using too much memory; it was writing a lot of data, and the writes were counted against it.

Page cache counts, and dirty pages don’t leave fast

The rule most people learn is “page cache doesn’t count, the kernel reclaims it.” That’s true for clean cache, which the kernel can drop the moment it needs the memory.

Dirty pages are different. They hold data that hasn’t reached disk yet, so freeing them requires I/O. Under cgroup v2, a container’s memory.max includes page cache, so when a process copies a large file, the destination’s dirty pages pile up inside its cgroup. If the disk can’t keep up (mine was a busy array, about 99% full) the cgroup hits its limit before writeback catches up, and the kernel kills the process even though its own memory is a fraction of the limit.

Each kill landed mid-write, and that had side effects: a corrupted key file, and a database is locked error in its SQLite database. Those looked like separate bugs until I lined up the timestamps.

So why was a media manager copying gigabytes of files at all?

It was supposed to be a move

Radarr’s job, after a download finishes, is to import the file into the library. On the host, the downloads directory and the library directory were on the same filesystem. Importing should have been a rename(2) (instant, no data moved) or a hardlink.

But in the pod they were two separate mounts:

volumeMounts:
  - name: downloads
    mountPath: /downloads
  - name: movies
    mountPath: /movies
volumes:
  - name: downloads
    hostPath: { path: /srv/media/downloads }
  - name: movies
    hostPath: { path: /srv/media/movies }

Same disk underneath, but two mount points. The kernel checks for a cross-device move per mount, not per filesystem. rename and link across a mount boundary fail with EXDEV (“Invalid cross-device link”), even when both sides live on the same block device.

Most tools, Radarr included, handle EXDEV by quietly falling back to copy-then-delete. Nothing errors, nothing logs a warning you’d notice. You just get the side effects:

imports take minutes instead of milliseconds
every import writes the whole file again
free space briefly halves for each file in flight
and every copy fills the container’s page cache with dirty pages, which is where the OOM kills came from

You can check this in ten seconds from inside the container:

$ ln /downloads/some-file /movies/some-file
ln: failed to create hard link '/movies/some-file' => '/downloads/some-file': Invalid cross-device link

Invalid cross-device link means the layout is wrong, whatever the host disks look like.

The fix: one mount, not two

The fix is to mount a single common parent so that source and destination sit under the same mount:

volumeMounts:
  - name: data
    mountPath: /data
volumes:
  - name: data
    hostPath: { path: /srv/media }    # contains downloads/ and movies/

My downloads directory wasn’t originally under the media directory, so step one was an mv on the host to put it there. That was instant, because it was a same-filesystem rename. I picked the narrowest parent that held both rather than mounting the whole array, which also held backups and other data the app has no business seeing.

The container side is the part people skip. The app still had /downloads and /movies configured internally, so I had to:

change the library root folders to /data/movies (with Radarr’s bulk editor set to not move files, so it only updated the paths)
update the download client path mapping so finished downloads resolve under /data/downloads
update the same paths in every other tool that talks to it (request manager, a file checker)
only then remove the old mounts, in a separate change, after confirming hardlinks worked

After that, ln inside the container succeeded, imports became instant, and the copies, the dirty pages and the OOM kills all went away. I also raised the memory limit to give in-flight writes some headroom, but sized it on purpose rather than from the observed peak, because a peak measured right at the limit is just the limit.

The TV-show manager next to it had the same split layout, so it got the same single mount.

Takeaways

When a container is OOM-killed with a small heap, read the kernel’s memcg report. Compare anon with file_dirty and file_writeback. If writeback dominates, the problem is I/O, not memory.
Page cache counts toward container limits, and dirty pages can’t be freed until they hit disk. On slow or full storage, a big copy can OOM-kill an app that barely uses memory.
Two bind mounts of the same filesystem are still two mounts. Rename and hardlink between them fail with EXDEV, and most tools fall back to copying without telling you.
For anything that moves or hardlinks files between directories, mount the shared parent once. That covers download managers and media importers, build caches, and backup rotation. Test with ln inside the container.
Raising the limit treats the symptom. It would have stopped the restarts here and left every import doing a full copy on a nearly full disk.

The community guides for the “arr” apps recommend exactly this single-mount layout. It’s easy to assume it doesn’t apply when the host paths are already on the same disk. It does: what matters isn’t the host layout, it’s the container’s view of it.