Post Rock? Post Punk Prog? Turntablism? Art Rock? Noise? A friend of mine thinks that The Superblock Corruption would make a fine name for a band and so would I if it were not for the fact that those three little words ruined my Friday night.
All fired up to install the set of 400GB SATA drives, I upgraded my installation of Ubuntu earlier that evening, and around 9:20 PM rebooted and logged in to make sure that everything was running fine. Confident that it would only take the time to track down a torx driver I shut down, popped the case, and got to work setting up the drives. Everything went smooth until I booted up.
The new drives were detected and with a flash a message flew by saying that the BIOS had been updated then it prompted me with the on-time question of which drive I wanted to boot from. Choosing the master drive I struck the enter key and waited. “No OS Found,” was the message that greeted me. Ctrl-Alt-Delete thinking maybe I chose wrong but this time the monitor went dark while the hard drives churned. Reducing the variables is my mantra when troubleshooting so I powered down and removed everything from the chain that could cause a problem except the boot drive. Nothing.
Gnashing my teeth in frustration I turned off the PC, unplugged and reset all the cables, counted to ten, and rebooted. This time greeted by the graphical boot process monitor my spirits lifted and I felt all was under control, that is until it paused on the Checking Root File system. “One-One-Thousand. Two-One-Thousand. Three-One-Thousand…” At fifteen, “FAILED”, and the system belches out to the command line as root stating matter of factly that Superblock Corruption was detected and I need to run fsck. I reboot into safe mode with the same error plaguing me and attempt to run fsck to repair the superblock.
By 9:50 I learn that UFS is comprised of several parts and the one vexing me is described as “containing a magic number identifying this as a UFS file system, and some other vital numbers describing this file system’s geometry and statistics and behavioral tuning parameters.” Great! I also learn that the command “newfs -N /dev/rdsk/c0t0d0s6” doesn’t work, “The command newfs is unrecognized”, and that “fsck -F ufs -o b=32 /dev/rdsk/c0t0d0s6” yields the same error that superblock corruption exists and I need to run fsck. Fsck indeed!
Around this time my niece fires off the first of many IMs in a near panic as her Sims2 installation melted down. Between reboots I walk her through uninstalling the game:
Her - “UJ! Sims is broken. cant play it!”
Me - “Won’t launch?”
Her - “no. :`-(”
Me - “Try uninstalling, use Add/Remove”
Her - “UJ, there is no addremove”
Me - “Start–>Control Panel–> Add/Remove Programs”
Her - “I did it from the EA menu”
Me - “Try a reboot and install again.”
Me - “Hold on. fsck is fscking up on me.”
Her - “k”
Her - “wait. fsck???”
And so it goes on as we try and get the game to install and run. Fortunately, she managed to get the core Sims2 game to install but unfortunately none of the expansions will even though we work on it until past midnight.
By this time it is nearly 10:00 PM, and while juggling my niece I resign myself to the plain fact that I cannot boot to this drive and that I need to begin recovering data and getting my hands on my backups. Luckily, I can boot without X running and run mysqldump and tarball the pertinent directories for a near full recovery. Superblock still plagues me during this process but I am able to recover nearly everything I can remember and I tuck it into a safe location so that I can boot to the live CD and dump it to my flash drive.
11:35 PM and the server is 85% operational the only exception is that this blog was rendering as if I had no defined CSS and that permalinks yielded 404 errors. This is the trouble I have with Linux. It runs stable enough that you only need to do things once and when something goes wrong or you need to re-build it is like you are a n00b all over again. Around 12:15 AM I realize that my configuration for WP requires mod-rewrite to be enabled: “sudo ln -s /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/rewrite.load“. Hey, it’s my website complete with images, CSS, and permalinks!
The night wears on as I put together my cron-jobs, troubleshoot a buggy Drupal installation, bonus for being able to upgrade to the latest build without trashing the database; score Drupal (3) - Me (1), and being rebuilding my music collection on these massive drives. Sometime on the cusp of 2:00 AM I finish with eyes burning and shoulders stiff I turn off the monitor and shuffle off to get ready for bed thinking that I only have a handful of hours before I need to talk Peri out for his morning walk.
Lessons learned:
- Always backup before upgrading either the OS or the hardware.
- Make regular complete backups and always have them available on removable media.
- Try and be helpful to your niece even though she should be in bed.
- Superblock corruption makes for a better band name than an error late on a Friday night.
Wow, I’ve never heard of superblock corruption…you sure you’re not making it up?
Cross my fingers I never get to encounter it firsthand…
Seriously, it is total balls. Thankfully, though it does not affect the ability to browse the volume from a Live CD. Just call me Quincy for my mad forensic skills.
Quincy, that was a sweet show back in the day. I have had some recent issues that have made me think about my backup soulution. I still need to look into it.
Tell you what, one of the best methods is simply copying data to another drive. I use a second/third hard drive and just set up a schedule for backups both differential and snapshot and it has saved me both on the hassle of burning CDs that seem to die after a couple of years as well as on the nightmare of losing everything.
Yeah that is the solution I am using, I just Rsync to a firewire drive, works like a charm, but I am running out of space, I need to get some bigger drives, I have about 400GB on 2 250 GB drives, and I am running out of space. Just need some money to buy more drives.
Well, depending on your cash-to-burn ratio building a storage array and mounting a linux disto could be the ticket. Go RAID 5 with 400 GB SATAs and your get 1.6 TB, then again it is a $1200 solution! (Just trying to live my fantasies through the wallets of others)