Server Stuff

I was confronted with an issue recently where users were not able to log into a site that I maintain. Turns out, PHP was throwing errors about not being able to create a session because the file system was out of storage capacity. My VPS has 24GB of disk space for me to work with in whatever manner I desire, but something was crunching through it. I learned a few things while trying to fix it.

Some Useful Commands

df -h - Show how much disk space is being used in the file system for each mount point.
sudo du -sh /* - Show disk usage for a particular path

Using these, I was able to start drilling into the problem areas. First, I cleaned out some cruft in /usr. /usr/share/doc is essentially useless to me in a server context, so that went. /usr/src is likewise useless after I have compiled whatever resources I've needed. /usr/lib/modules needed some help - I got rid of some old utilities that I wasn't using anymore. Snap also went the way of the dodo; I can't even recall what I used that for in the past.

The largest problem area was the method I was using to create site backups. The process works, but I can now see how wildly inefficient it was in terms of disk usage. In essence, I packaged all of the site's files, configuration, and data (user-generated files plus a SQL dump of the MySQL database which is hosted locally on the VPS) into a zip archive. I was retaining 3 weeks worth of these zip archives and shipping them up to GitHub. I pared this down to 2 weeks worth which didn't make a large impact on file size, but even so. I also noticed that due to the way that git retains file deltas, the .git/objects folder was 9GB+. Insanity. I was able to install git LFS and start tracking the zip archives with that - I had to nuke and reclone the repo on the file system though because simply untracking from git and retracking with git LFS was not enough to remove the objects folder. After making this change, it's back down to 385MB.

There may be more things I can delete from the file system, but right now it's sitting at 13GB available, so hopefully it does not spin out of control again from something that I inadvertantly do. Obviously a normal production system would have disk space monitoring and alerts, but for a hobby site I don't feel the need to build that level of observability.