Linux Disk Space Paradox

Our server crashed the other day because the disk was full.

I quickly discovered a rogue bandwidth log file not getting cleaned up was to blame, so I deleted it. Strangely, running df after deleting the file showed that 100% of the disk was still in use even though I had gotten rid of the biggest space hog. Perhaps there was something else I was missing?

I ran the following command as root to list top 25 largest files/directories:

du -a / | sort -n -r | head -n 25

But nothing was even close to the size of that bandwidth file.

Thankfully, I came across this great Stack Exchange post which cleared up the mystery. If you delete a file while it’s being written to, the file gets unlinked; but until the process that’s writing to the file gets stopped, it doesn’t actually get removed off the disk. Which means that if you run df, the disk appears full; but if you try to find the file using du, it’s nowhere to be found. Thus to df/du paradox.

I used the following command to look for unlinked files that were still open:

$ lsof +L1

Once I found the process, I killed it, and the file was actually removed from the disk and I got my space back.

Mystery solved.