Tonight's Gonna Be a Good Night
I had quite the fiasco last night with the server. Apparently, Debian Linux released a new version over the last few months. The way Debian works, they denote their releases by stable, unstable, and testing. I was running stable. When there is a new release, the new release becomes stable, and the previous version becomes an “old” release.
The problem was: my update configuration (default from the server host I use, I must add) said, “Pull update packages for the stable release.”
When I ran updates, it proceeded to pull down patches for the new release and merge them into my now-“old” install. This would be akin to Windows Update pulling Windows Vista updates for your XP machine.
Everything proceeded to break. Because the dependencies of the new packages did not match the foundation of my old packages, I could neither install nor uninstall just about anything. This took down my web server, mail server, application server, and disabled a few other key installs.
I originally thought I had some kind of circular dependency, so it took me quite a bit of time to figure out I needed to specify the exact release for updates and re-run the updater. Once I did that, it restored most of the system, minus a little tweak or two to get back configuration it over-wrote in the process.
The cool thing here is that, although it destroyed quite a bit of the functionality of the server for an hour or so, the vast majority of my configuration and all of my data were completely safe. I didn’t fear for them at all, except in a worst-case kind of way that is pretty much the standard server admin mentality.
I was mostly freaked out because everything was down. That’s probably vanity; nobody is really depending on my server being up. I definitely was afraid I would have to backup everything (error-prone) and reinstall the server, which would have been pretty painful now that I have made so many modifications and have such a vast array of goodies running.
I never fail to learn something when anything like that happens.
The more I do stuff like this, and after seeing what I saw in Turkey, the more I begin to realize I was meant to do this system stuff: making things work, designing, optimizing, deploying and explaining to people who don’t have to know the inner workings. I’m working very hard right now mentally to reconcile that with my current trajectory. Paperwork calls…