I awoke to find my BT phone line dead, so no internet nor comms (I had no mobile phone then). My email was still broken since migrating to Boredband until I could work out how to reconfigure it. My car has been pouring smoke through the ventilation system like some kind of James Bond anti-theft device. [UPDATE: Soon afterwards, my house boiler broke down (on a Sunday!), so I was without heating for a few chilly days. ]-:
And now this...
Broken email, phoneline, boiler, smoking car... What next?
On Wednesday night I switched my computer on to hear that horrible ticking
of Death... The computer reported a Primary Disk Failure; all attempts to
revive it have failed. I tried mounting the partitions directly from another
Linux system but neither
find a valid superblock, even if I use alternative values. There may be a
way of salvaging it, but I can't afford to spend hundreds having it taken
apart in a lab.
What's even more annoying is that while gazing by chance at
/var/log/syslog a few days beforehand, I noticed that
SMARTd was reporting 2 Uncorrectable Errors on the drive.
I'd started reading the very fine man page and was looking into the problem,
but had no idea it would suddenly become critical.
So I lost my entire operating system and all data since my last backup, which *thank-the-Lord* I'd done on 2007-10-14. This was my first backup in ages, and had I not done so, I'd probably have given up the ghost by now and gone to join the rebels in the mountains. As you can probably gather, I'm not like normal people and my filing system is an epic work of human ingenuity that future generations will marvel at (I'll give a talk on it one day). In future I need to ensure Proper Backups which is now my Top Priority.
GO AND BACKUP YOUR DATA, RIGHT NOW. Do it!!!
All this means that I've been plunged back in time to the dark depths of the horrid Mercury Retrograde period that was late October, with all its associated equipment failures and comms problems. I'm trying to piece together all that has happened in the past three weeks. I wrote a few journal entries but didn't get round to finishing them and uploading them, so they are lost; I've rewritten about Boredband and my Akai S3200XL sampler but another long article about synths has been lost (unless anyone pays me to rewrite it!). There are no doubt other things I've forgotten - I just hope I didn't make any plans to work for people and then not turn up! Luckily the past month was largely spent lost in a Motivation Hole playing Doom and reading Slashdot, instead of getting on with business plans and my new company website; if I had been more productive, it would have all been lost anyway. Also fortuitous is that my earlier email breakage had meant that most of my recent emails have mostly been conducted on my GoogleMail account, which is obviously still alive on their servers. And most of the work that I had done was in configuring mutt, which was mostly just copied from the fine manuals anyway. So I can probably live with the data loss, and am not crying. (*Big strong man*)
BT finally fixed my phone so I at least was able to get back online and download new software. My smoking car is being fixed and Ian has kindly lent me his in the meantime :-)
Please bear with me while I catch up (oh, and
email works now thankfully :-)
All emails I sent and received between 2007-10-14 and 2007-11-07 inclusive
have been lost (but not messages to/from my Googlemail account).
So if you saved any emails from during that time, please re-send them.
I, erm, also neglected to backup my email inbox of 4000+ messages...
Over the next few days I resurrected another machine I had lying around and built a new operating system on it using the latest version of Debian (Etch), something I'd been planning to do for a while, once I'd fixed my broken email. It's a good chance to start afresh, and the new software is NICE and *shiny* with lots of cool things to enhance my productivity like Named Sessions in Kate :-)
I spent much of the rest of November reading computer manuals and about CSS ready to power my new website once my machinery was all back together.
A while back, there was talk at HertsLUG (which then inspired some Linux gurus Down Under) of starting a game called Command Line Bingo where people would choose a lesser known *nix command and prepare a short presentation about it for the group. The more obscure the better; it was suggested that you score a point for every person present who'd not heard of the command, but I'm not sure what points make...
The idea was forgotten for some months, but at this month's meeting, I gave
the first of hopefully many talks. Since I'd been so impressed with the
smartd, that was my choice; along the way I also brought in
hddtemp hoping for extra points. As a follow-up to the talk,
you can read more below about my experiences of using them to try and
resurrect and protect my drives.
There was a nice article featured on Slashdot about Rare Soviet Retro-Future Space Art. My favourites were:
Heady stuff and very different from our current view of space. I've always been drawn to Russian science fiction: films such as Stalker and Solaris, based on the fiction of Stanislav Lem and The Strugatsky Brothers (check their amazing Chronicles of the XXII Century). I remember once attending a talk at university given by some Russian ex-cosmonauts - very inspiring tales from these guys who had been into space. They described how noisy it is inside a spacecraft, and how strange it feels to be away from Earth. My generation were brought up with the notion that we would soon live and work in space. Reality has fallen short of the mark, but space tourism might be widespread by the time we're olde. Whether or not that's a good thing or not is a another matter...
[UPDATE: Here's some of Cassini's best images of Saturn which show that Nature is still rather good at graphic design.]
Part of the reason it took me so long to recover from my hard disk failure was trying to find a decent backup solution. The most important thing about backups is that they are easy, preferably automatic. If the process involves any human effort, it often won't get done. I knew I needed some kind of external drive that could be disconnected (to save noise all day long) and kept safe when not in use. Since I now have hotswappable USB working, the choice was between buying a separate hard drive and enclosure, or a dedicated standalone external drive. A friend had lent me one enclosure that had issues with Linux, so I was wary of buying a lemon. Then I started reading about Seagate FreeAgent external drives being lame on Linux and Western Digital restricting use of network drives. Further research into Western Digital's FAQ found that:
"The Western Digital NetCenter and WD My Book World Edition hard drive uses a proprietary file system and cannot be reformatted as FAT32, NTFS, or a Macintosh File System."
"...file time and date attributes change. This is caused because these hard drives contain and run their own Operating Systems (Linux based) and essentially take control of the file from the time it is copied or moved to the hard drive."
Hmmm, I hope they release source code with that... So that's WD out of the running for messing about with standards. I just want a drive that does what I ask like any other, not one that makes me toast in the morning. I hate the way stupid companies keep trying to leverage more useless 'features' on top of what you need, in the process locking you out of what you're trying to do.
In the end, to escape these potential headaches, I decided to go for a separate USB enclosure containing a laptop drive, which would be much quieter and cooler, and could be powered directly via USB (although Rob says this is bad ;-)
The new generation of Samsung SpinPoint M5 5400RPM SATA 2.5" drives are practically inaudible and offer new levels of areal density. For optimum reliability, cost, acoustics and thermal performance I chose (what I believe to be) a single-platter 160GB model HM160HI rather than the 250GB model as I don't yet have that much data. In addition I ordered an Akasa Integral P2 2.5" enclosure. I'm impressed:
My olde PCs only have USB1.1 (copying 27GB took 7 hours!), but for future
use, in addition to USB 2.0 this enclosure also has eSATA which is much faster.
During that long wait while files were copied I didn't dare do anything else
so spent the time productively Reading The Fine Manuals for
rsnapshot, an awesome
backup program which I am now using to make daily incremental backups.
Inspired by a fine article called
Snapshot-Style Backups with Linux and Rsync, it uses
rsync to only copy the differences since the previous backup,
minimising data traffic. So daily updates only take minutes instead of hours
(over USB1.1). This
(Free) software is easily
configurable as to what/how you want to backup, and highly recommended.
[UPDATE: These backups soon proved very useful :-]
Now that I had a safety copy of my data, I then reinstalled the dead hard
drive to try and salvage its data using
(a similar but different program from
which doesn't use log files). I was hoping I could at least resurrect my
email inbox which I seemed to have, erm, overlooked
in my last backup...
But it wasn't looking good: on bootup, the drive was not detected by the
BIOS. Disabling that, Linux reported a drive size of 117552TB (yes,
117,552,780,861 Megabytes!!!). If only it would work, I could sell
it to Google for $$$ to host their entire worldwide platform on one drive ;-)
could read/write to/from the drive. I left
ddrescue running for
some time and broke my
fast, but returned to a report of 30GB of unreadable disk, 0bytes good.
So I took the drive out and tried Nicolas' trick of banging it very hard on the side against the edge of a metal desk, in an effort to loosen the clicking drive heads. That still gave no joy, so I removed the drive again and submitted it for the last resort: The Bench Test. Only instead of bench height, I dropped it from head height (since this was probably a disk head problem) onto a hard floor. The drive still reported it's tantalisingly large size of 117552TB, and was still inaccessible.
"He's dead, Jim."
The scary part came upon removing the dead drive and rebooting. I'd already
kittens earlier when my new replacement
drive failed due to me
jumpering it as IDE slave instead of master, so the OS was trying to mount
/home on the CDRW drive! But now I noticed some scary messages
littered throughout the boot sequence about "No space left on device".
df told me that my root partition was full (!!!) (the OS is on
an old 4GB drive). And rather spookily, someone had left an empty file called
dead.letter in my / directory, like some kind of pirate's
I quickly deleted some extra backups I'd stored in
clear some breathing room, and then looked in
which was now a 387MB file (on a system only days old!). Scrolling through it
less, I saw that most of the logs were the errors from
trying to access the dead drive (thousands per second). So do be careful
with that log, Eugene! I now have to figure out how to chop all the millions
of unnecessary lines out of the log file to save disk space;
Ian suggests some regexp
Danger over now... we hope!
After extensive reading, I managed to fix the issues I had with email back when I migrated to Broadband. I've now got my data safely on a new working operating system, minus the 3-week black hole since my last backup, and also minus my entire inbox of 4000+ emails that "I was just getting round to answering", but somehow forgot to include in my last backups before the crash. Most of them were just mailing list posts dating back years that are (mostly) publicly archived (although I had kept them for a reason, meaning to read them or reply), but the last few hundred were semi-important emails from friends and contacts, I can't remember what/who.
PLEASE NOTE: All emails I sent and received
between 2007-10-14 and 2007-11-07
inclusive have been lost (but not messages to/from my Googlemail account).
So if you saved any emails from during that time, please re-send them.
So there you have it: a simple way to achieve Inbox Zero - just don't do backups, and you'll get there sooner or later!
© copyleft Malcolm Smith 2007-11-01 - last updated 2008-01-01