Dapocalypse Now

The Data Apocalypse, aka the Dapocalypse, is a favorite of science fiction writers.

It goes something like this.

Some evildoer plants a data bomb deep into a compiler language. One day they pull the trigger, and Google's data stores are irreparably corrupted. It will take months of labor to restore from deliberately isolated time-delayed offline archives (Google does have these, right?).

By this time Google holds a large part of the world's knowledge, and it accounts for 40% of the metamind's IQ. The lobotomized metamind can't keep a lean mean non-redundant economy running; the economy crashes hard. Google dies in the carnage, so there's nobody to do the restore. An interconnected world crashes, war breaks out, billions die, we return to the 19th century.

Makes for an uplifting story (no Singularity extinction!), but maybe it's not so far-fetched.

I used to control my corporate data. It lived on my drive in thousands of dead-bit documents. Write once, re-read never, though nowadays full-text search brings some of 'em back.

Today, instead of a single document, I may use a loosely-coupled dynamic and shareable collection of smaller documents, linkable blog posts, and long-lived database list views and wikis. It's early going, but I think this distributed approach will win the day.

There's a catch though.

I can't back up the data distributed across the corporate LAN. Can even the very best IT department be relied on to restore a relative handful of critical rows and joins across a petabyte data store?

Restoring a document is trivial. Restoring a distributed data engram across multiple Sharepoint lists is very, very hard. Perhaps prohibitively hard. Entanglement gives meaning, but now data is becoming as fleeting and transient as thought and memory.

I'm good at backup. In perhaps 15 disk crashes I've lost maybe a few hours work. Now though, my personal Dapocalypse feels inevitable. It feels like I'm waiting for a stroke.

I really don't know what to do about this.

Update 11/25/08: Changed my mis-spelled dacopalypse to Dapocalypse when, what do you know, it happened.

Update 1/3/09: Slashdot uses the term "datapocalypse" , but not, alas, Dapocalypse.