Beating the odds - a software story

For the second time in my career I had the great privilege of imagining a software product and seeing it through to release. It started out as a key toolset for another product, then ended up, at the end of its initial product cycle, being repurposed for a significantly larger role.

This isn't the kind of software story you usually read about. We weren't a startup group of young developers, we were a small group of grizzled veterans in a large publicly traded company. We didn't have to worry about VCs, but we had our own version of funding uncertainty. If 37Signals is Superman, we were Bizarro. In a mirror world of software development, we faced our own set of grim obstacles.

Frankly, I'd rather face the obstacles of my startup days but, still, we did succeed. We succeeded because we had a great team (seriously great) and, at critical times, we had close collaboration with a great customer.

Here are a few of the things I learned in the process, in no particular order. In a small way, they were a recursive version of a far more ambitious project I read about many years ago - The Data General Eagle ...

• The core of our team was local, but we had key contributors that were remote. Our collaboration technologies were phones (and teleconferences - 1970s tech) [6], email, a Sharepoint 97 wiki [2], LiveMeeting screen sharing [4], and Rally [3]. When I ran meetings with remote contributors I had everyone dial in. We developed some good techniques for managing remote discussions, including sharing MindManager maps to record and organize discussions. The main lesson here is that you can get good results out of some very limited tech tools -- if you thing hard enough about how work around their issues.
• Early on I spent a lot of thought solving problems we never got to. Some of those imaginings led to patent applications [1], but they didn't have much impact on the product. In a few cases though, those solutions were critical. It was time well spent, but it's worth remembering that the real challenges are likely to be surprises.
• What we ended up with had a lot in common with my earliest designs. I don't know if that means they were good designs, but they did persist.
• The curse and joy of software is that there are so many different ways to solve a set of problems. The trick was figuring out what compromises to accept, even when two good alternatives combine in a troublesome compromise. This wasn't hard within our group, but it was challenging when we had to fit into other models. A mediocre compromise, however, is better than a breakdown in critical partnerships. We threaded the needle.
• Our best decisions weren't coming up with solutions to tricky problems, they were deciding what to keep and what to drop. We had to choose between throwing the seats overboard, or the luggage, or dumping fuel. We couldn't afford to get those choices wrong and we mostly got them right.
• Everyone contributed everywhere. I did everything but write Java code. Our engineers did designs. We were all analysts.
• I like Agile. We couldn't do it fully for several reasons, including the world in which we lived. I think though, we stayed true to the sprit of Agile. Rally helped, though I fear the developers have been too responsive to their customers. There are quite a few rough corners left over. Nothing's perfect, but Rally is pretty good.
• I liked the Agile philosophy of just enough architecture. The key for us was deciding where we needed solid foundations and where we could put up a low cost shed that we'd happily tear down when it decayed.
• There's a trick to choosing between a range of reasonable options. It doesn't matter so much which one is chosen, only that we don't spend a lot of time choosing.
• In the absence of proper resources, a well crafted email with an edited thread attached to a Rally story can be a reasonable stand in for a requirements document.
• Inbox Zero was very important form me. When I cleared my inbox and scheduled my Rally tasks [5] into my calendar I was usually in good shape. When either fell behind I was in trouble.
• No emotion. I worked hard to stay balanced. Things were tough when we were repurposed [7].
• No death marches. We are too old for death marches, it's not an option when you're over 40. Quality goes off a cliff. We rescoped or invented easier solutions rather than pulling all nighters.
• I reported out probabilities of success to management rather than predictions like "we'll do it" or "we won't do it". Somehow that worked better.
• I kept a complexity budget in my head for everyone on the team. We spent our complexity capacity carefully, targeting high value work.
• We kept moving. When we bogged down we stopped and moved on an easier path. There were quite a few obstacles we couldn't control, rather than try to knock them down we went around.

We had fun. It's satisfying to beat the odds.

--

[1] Software patents are a curse upon civilization.
[2] Once you figure out all the traps (don't paste rich text into the rich text editor!) and the trick to embed images, you find out the search is pretty good and that SP's wiki isn't all bad. SP as a document management system is pretty bad, though less intolerable with Office 2007 than 2003. This project made me a Wiki convert. (See also: Gordon's Tech: Vermeer/FrontPage lives in Sharepoint Wiki)
[3] Also, alas, StarTeam. I try not to think about StarTeam.
[4] Only the screen sharing.
[5] I got to write the stories and the tasks then do them. A bit inbred.
[6] I tried very hard to get Google Video Chat working. It failed for us. Partly due to bugs, partly due to a remarkably poor UI, but mostly because the corporate net connectivity was overloaded. This is a more common problem than most realize.
[7] Would have been helpful if I'd done the Conversations - From emotional confrontation to dialog class a year ago, but I'm not sure it would have made that much difference. Managing genuine conflicts and power struggles by phone is less than ideal. We had no travel budget.

Aging boomers and the coming Golden Age of Cyberfraud

Just one recent example: Aggressive Social Engineering Against Consumers

As we boomers age, there will be a rich supply of weakened herd members for online predation. The Golden Age of Fraud is coming.

This is why you will live in an iOS world.

Tornados and global warming - how do we judge predictions?

We can't forecast a tornado, and we can't predict how a tornado will behave. We can, however, characterize tornadogenic climates and geographies. As CO2 accumulates and the earth warms virtually all terrestrial climates will change. Because climates will change they will all become more or less tornadogenic. This seems self-evident; I don't think there's any controversy here.

There is lots of controversy, however, when we try to understand the causes of the great American Tornados of 2011. There is controversy too, when we try to predict what will happen over the decades to come. Will, for example, geographic regions experience an increase in tornados as the earth warms, only to see a decrease when it warms still more? Will "Tornado zones" migrate north, so that Arkansas will have fewer, but Minnesota more?

Insurance companies would dearly love to know. So would homeowners contemplating installation of a basement emergency shelter. Given the purported limitations of historic data, how can insurance companies and homeowners make decisions?

Consider the case of a fair coin. Flip the coin ten times and you get this: TTTTTTTTTT - ten Tails. What's the chance of a Head on the next toss?

It's a trick question. I said it was a fair coin. The chance of Heads is 1/2, just as it was for the previous 10 tosses. Reverend Bayes does not apply.

Now consider that the coin has been altered; it's no longer a fair coin. Flip the coin ten times and you get this: TTTTTTTTH. What's our best estimate of the chance of a Head on the next toss?

It's 1/10.We don't know anything about the coin, so our best estimate of future performance is past performance.

So we can measure tornados like biased coin tosses and, in 30 years or so, we'll get some reasonable answers.

We can do better than that though. I wrote recently ...

... The process of iterating on internally consistent models that make testable predictions, and revising those models when predictions fail, has transformed human history. It is the only guide we have to developing better medicines, understanding the universe, or predicting the consequences of CO2 accumulation...

Consider our biased coin. We might speculate that a variable gravitational field is causing bias. We may predict that if gravity is varying, then local clocks should diverge from distant clocks. Clocks seem unrelated to coin toss, but if we do find clock drift, then our varying gravity explanation for both coin bias and clock drift is strengthened. We can use that new understanding to make more accurate predictions of future coin toss outcomes.

In a connected system, like a climate, a model can be validated by shorter series of multiple measures. So a model that predicted tornadogenic weather might take decades to validate, but a model that predicts summer storms, winter snow and average temperatures might be validated in a shorter time.

At least that's what insurance companies must be banking on. There's a vast amount of money at stake, a good model would be worth a lot. Particularly if it were private ...

See also

• Demography, Design, Atom Bombs and Tornado Deaths - NYT (Revkin)
• Tornadoes and Warming, from the Archives - NYT (Revkin)
• Attributing single events to background conditions — Crooked Timber
• The Facts (and Fiction) of Tornadoes - NYT FAQ
• Gordon's Notes: Bayes theorem - in a nutshell
• Gordon's Notes: Separated at birth: alternative medicine and climate change denial

Understanding democracy

In a democracy, the primary talent of the ruler is winning elections. This explains why, although winning politicians are more clever than the average citizen, they are not significantly better at governing. This also explains why juries are wiser than one would expect.

Failures of American government are failures of American citizens. We get what we deserve.

We should try drafting our political candidates by lottery, then elect from this pool after a traditional campaign. We can start with a State House and see what happens.

See also:

• Campaign Finance Reform: Voluntary Mandated Sharing

Update: Ok, could we at least insist on licensing exams for Senators and Representatives? We license barbers for Pete's sake ...

Reconciliation May 2011: The posts I won't get to

Reconciliation for May 2011 ...

• Elderly Drivers Fail a Test (NYT) and Some Older Workers, Pressured to Quit, Fight Back (NYT): Why raising the retirement age won't work. Our brains don't last.
• The Blind Use The Visual Cortex To Process Sound (SciAm): I've long wondered about how we evolved the ability to read -- to convert images into sounds. It seems an unnatural use of the brain, even for an animal that must have been good at sign language. Given these latest findings, the ability to read may be just another manifestation of astonishing brain plasticity.
• The Early Days of Eugenics: 1911 SciAm article on Eugenics. "The proper attitude to be taken toward the perpetuation of poor types is that which has been attributed to Huxley. "We are sorry for you," he is reported to have said; "we will do our best for you (and in so doing we elevate ourselves, since mercy blesses him that gives and him that takes), but we deny you the right to parentage. You may live, but you must not propagate."
• Medicaid a Big Deal Too | Talking Points Memo: The real budget problem is not social security or medicare or medicaid. The real problem is that our bodies outlive our brains. Demographers knew this day was coming back in the 1970s. That's when we should have invested in slowing the onset of dementia.
• Third Depression Watch - NYT: The good news is that Krugman is no longer talking about the Great Recession. The bad news is that he's talking about the 2nd Long Depression. (See also: Leonhardt, 2008)
• Frothy Real Estate in Manhattan: $5.5 million apartment
• Reassessing The Cost Of The Post-9/11 Era, Post Bin Laden (Froomkin): Was bin Laden the most successful investor in history? With our help he turned mere millions into a successful trillion dollar attack on America.
• Jann Wenner is my hero - Cringely and The People vs. Goldman Sachs | Rolling Stone: Corporate control of American media - more than you think
• Software Reviewing as it Should Be Done - James Fallows and Zeoli on Tinderbox: A quixotic effort to push the boundaries of Xanadu, but not a word about data lock.
• Inequality Rising Across the Developed World - NYT Rampell: Not only America.
• Quote of the Week, Last Week Edition - Jeffrey Goldberg: Obama's famous quote on birthers, spoken while he was secretly leading the bin Laden hit.

The Data General Eagle: Tom West and an old paper of mine

Tom West died recently. He was made briefly famous by a Tracy Kidder book, The Soul of a New Machine.

Reading his story, I remembered that I co-wrote* an essay on the Eagle project, drawing from Kidder's book and interviews with Data General veterans. It was written using MORE 3.1 @1996. I liked the paper and I eventually put a PDF online in 2001. It's still available from my old archived FrontPage web site: The Data General Eagle Super-minicomputer : A project management paper.

The lessons of those days still apply today. Not much has changed ...

Happiness as editing - Sunnyside edition

“No story ends happily. The happy ending is only about knowing where to end on a smile, at the very moment where fortune is still on the ascent. The open road. The wedding." Sunnyside by Glen David Gold, as quoted in Sunnyside II – Count no man lucky until he is dead — Crooked Timber.

Happiness is all about the editing. Things happen around us, we make up our own stories. If we're smart, we pick the happy moments to declare a chapter done, signed, sealed for ever. It can't be undone, the unchanging past is more eternal than the universe.

Some chapters are long, some short. The last chapter ends badly, but it's only one among many.

Google Quick, Sick and Dead - 5th edition. Reader is ailing, but there's been a turnaround

Google Reader is not well. In particular I'm seeing broken bits in the "Following" infrastructure. People Search is hanging for me, I have active "follower" feeds that are missing controls like unfollow, I find "anonymous" in the list of persons I "Follow" and so on.

This is a big deal for me. I rely on Reader.

Which reminds me that it's been seven months since the the 4th edition of Google: The Quick, the Sick and the Dead. Time for my review of the Google Services I use personally (so Android is not on there). Items that have moved up are blue, items that have moved down are red, in parens is the prior state. I had Reader as "Sick last time, so it's unchanged.

The Quick (Q)

• Google Scholar (Q)
• Gmail (Q)
• Chrome browser (Q)
• Picasa Web Albums (Q)
• Calendar (Q)
• Maps and Earth (Q)
• News (Q)
• Google Docs (Q)
• Google Voice (Q)
• Google Search (S)
• Google (Gmail) Tasks (S)
  
• YouTube (S)
  
• Google Apps (S)
• Google Profile (S)
• Google Contacts (S)

The Sick (S)

• Google Reader (S)
• Google’s Data Liberation Front (S)
• Google Translate (S)
• Custom search engines (S)
• Books (S)
• Google Mobile Sync (S)
• Google Video Chat (S)
• Google Checkout (S)
• Orkut (S)
• iGoogle (S)
• Chrome OS (D)
• Blogger (D)
  

The Walking Dead (D)

• Buzz (D)
• Google Groups (D)
• Google Sites (D)
• Knol (D)
• Firefox/IE toolbars (D)
• Google Talk (D)
• Google Parental Controls (D)

The Officially Dead - since last edition

• Google Video
• Google Base (D)
  

Since the last update there are two new recognized and official deaths - Google Video and Google Base. (See prior editions for other terminated products, I don't carry those forward). I missed that Google Base had died, that didn't get a lot of attention! It moved to a merchant service that I don't track. Google Video had an interesting demise. Google at first intended to delete content, but then reneged and now promises to migrate videos to YouTube.

To my surprise, however, Google has done better over the past seven months than I'd thought. Eight products have improved significantly; two moved out of the Dead zone! That tells me there's hope for Reader. It's been ailing for a while, but it's certainly not Walking Dead. There's a good chance for a reboot, probably as part of scrapping Buzz and the "Follower" model.

The most important improvement has been in the most important product -- Search has been much better since Google moved against the content scrapers.

This is a real reversal from seven months ago when I wrote ...

Seven products have moved from Quick to Sick - including Search. That's a big one. Google suggest is fun, but Google is losing the splog wars. Too many of the results I get back are splog noise. I love Reader, but the Notes/Comments silliness has to mark it as Sick. I also love the Data Liberation Front, but they're not getting traction any more. I suspect they've lost funding. Translate hasn't made progress on the non-Euro languages, so it's increasingly irrelevant.

A good turnaround for Google. Keep it up!

See also:

• Google Quick, Sick and Dead #1 (1/09)
• Google Quick, Sick and Dead #2 (11/09)
• Google Quick, Sick and Dead #3 (6/10)
• Google Quick, Sick and Dead #4 (10/10)
• Gordon's Notes: Reliability and the Cloud - Redundancy required (not all is sunny - reliability has been down across Google)

Obligatory Rapture Day post

I'm sorry. It's union rules. I have to say something.

First, because I'm a certified killjoy, I'd like to see someone advising families on how to counsel the unraptured. I'm not joking. Really. Imagine the shock for people who took this seriously. I thought that number was about zero, but after reading the NYT today there must be at least a dozen of them.

Secondly, the unrapture will be blamed on Barack Obama. God's no-show will be proof that Obama has driven him away. This is my easiest prediction ever.

iPod touch wheel - 9 years and an eternity ago

My daughter brought my first iPod to me - the 2002 Touch wheel iPod. It wasn't responding to her touch.

It sits in a cradle, the battery died many years ago. It works as well as ever, the 30GB drive serving music to a local stereo. I sync it every six months or so, when I remember. Toggling the lock switch took care of the problem.

That's not what I'm writing about though.

I'm writing about the shock of holding it. It feels enormous, strange and crude. It feels like a relic from ancient history, it feels older than a VCR, as old a phonograph.

It's just nine years old.

Heck of a decade for Apple.

Conversations - From emotional confrontation to dialog

I'm back from a two day corporate class on VitalSmarts Crucial Conversations (see also: Amazon reviews of the book). I'm going to summarize here what was new to me, and what I'm going to do differently in my personal and corporate life. This is how I process new material, please feel free to skip this post if this material isn't your tea cup.

I'm not going to review or recap the original book by Paterson et al. I skimmed the book and I wasn't impressed. I was, however, pleased with the VitalSmarts "Participant Toolkit", with their educational materials, and with our instructor.

I'm also not going to recapitulate the course. This is a summary of my personal interpretation and transformation of the course material including my own experience and readings. I particularly recommend the complementary book Bidell's Three Steps to Yes. In some respects this post may contradict the course work, in others it extends the material.

Before I begin, I can't resist some cultural context. It's impossible to read this book, with its model of "silence" and "violence" as two styles of aggressive conversation, without thinking of "female" and "male". Among other things, this material can be read as a guide to communication in a multi-gender corporate hierarchy. There are limits to this interpretation of course. Like many geeks of either gender my "style" score was silence/violence balanced, with a bias to "silence". (Important note: "violence" in this context is verbal - sarcasm (attack), verbal control, and verbal labeling. It's an interesting choice of label.)

The concept of a "crucial" conversation is novel and meaningful. There are three ingredients, but one is particularly critical. The first two ingredients are high stakes and conflict. The third and most critical ingredient is (negative) emotion. The primary focus and value of the "Crucial Conversation" (CC) methodology is managing the emotional component of important conversations. The goal is to transform a high-emotion interaction to a low emotion "dialog".

A CC, therefore, is not a good thing (I think the course materials are confused about this). The "good thing" is a productive and positive dialogue. A "CC" is, at best, a means to getting to dialogue. At worst, it's the result of a botched interaction, and a means to get things back on track.

The first goal of the training is to be able to recognize when a Conversation goes Crucial (think Plutonium going Critical). The most important response, at that point, is to give up on the topic of conversation and focus on managing the emotional component. That's a big idea for me. Especially on a phone conference I've planned, I'll miss or disregard the emotional component and try to power through my original agenda.

Speaking of a phones, this is where the course is partly obsolete. It was written for the 1990s world of person-to-person communication. In the post-Great Recession world our corporate interactions are by telephone conference. No, not telepresence or videoconference -- 1970s style teleconference. Person-to-person emotion management is hard enough, but after years of being beaten and pummeled I can just about manage it. Managing emotions on a multi-person half-the-team-is-muted conference call takes things to a whole 'nother level.

So how can we modify the approach of the course to a setting where emotions can rise fast on a teleconference with people we may see every few years? The approach I'm going to test goes like this:

1. Identify signs of emotional intensity. Conspicuous silence or tone of voice are the best remote cues. I can monitor my own responses, such as rapid heart rate, sweating or increasingly slow speech -- but those are late arrivals. I think an early sign of emotional content, for me, is narrowing of my eyes. I'm training myself to detect that and even to forcibly relax my eye muscles.
2. Manage the immediate emotion. This may mean using techniques that CC considers dysfunctional -- such as avoiding and withdrawing. The goal is to get out of the call without an escalation.
3. Schedule a managed one-on-one "CC" -- by phone usually (alas). (Telepresence is better, and with advance planning may be available). Scheduling the one-on-one "CC" gives time to work on the "path" script of Fact definition, scripted Story, and Ask questions. The goal of the scheduled "criticality" is to get through emotion and back to "dialogue".

The training and course materials don't discriminate between a "planned" and "emergent" CC. That's a big distinction. It's the difference between running up hill and running through a mine field. Given where I am now, my personal goals are to recognize an "emergent" CC, calm it as much as possible (abandon agenda, get out of Dodge alive), and then plan a managed CC.

My outline of a managed CC borrows from CC and Bidell. It starts with a planning phase that's largely Bidell ...

1. What is my goal for me and others? What is the true goal of the other person -- even though they may not know it themselves? (In Bidell's world, it's usually personal success in one form or another. That seems to work.)
2. What is it I want to avoid?
3. How can I achieve #1 and avoid #2?

Knowing the other person's true goal, and how that can be achieved, is the key to both Bidell's "Persuasion" and to creating CC's "shared purpose". That "shared purpose" may be to achieve success for the other person, even if their original conversation goal is not met. In Bidell's terms, find a way for my "Prospect" to "Win" -- while making the sale.

CC next focuses on the "do/don't" statement as a way to express my conversational goals. "I do want to get paid, I don't expect to get paid this week." It's not a bad place to start, but I can see how it might need modification.

The next phase in this structured high-emotion conversation is Fact/Story -- avoiding the dreaded "why" (the other banned word is "but"). Fact is supposed to be an enumeration of verifiable statements that will considered "true" by all participants. The Story comes next -- it's where the emotion and opinions come in. The Story is the statement of personal impressions, carefully refactored to avoid "violence" (sarcasm, control, labeling, etc), to avoid identifying a villain or a victim, and to avoid expressing helplessness.

For me, both the Facts and "the Story" are best written out beforehand and practiced aloud.

The Facts and Story are to be presented in an "tentative" and "testing" fashion (What have I left out? Does this sound right? What are your thoughts/feelings?).

The Story is followed by the "Ask". The goal here is to encourage the "Prospect" (Bidell's term) to follow a similar "Path" by asking framing questions and using classic conversational strategies such as  mirroring (I hear you say you're good, but as I imagine your face I think it's .... ok, so this doesn't work so well over the phone) and "paraphrasing".

At that point, if all (miraculously) has gone relatively well, the "Crucial Conversation" is done, and the action conversation (decisions, dialog, discussion) begins.

Or so the theory has it.

I'll be testing that out.

Get your international transpant with MedToGo

It occurred to me that a custom Google news section would help me track the worldwide retail organ business.

The results were more impressive than I'd expected.

Here's one ...

Desperate Americans Buy Kidneys From Peru Poor in Fatal Trade - Bloomberg

... Medical tourism company MedToGo LLC, based in Tempe, Arizona, says it will offer kidney transplants in Mexico and Costa Rica for about $50,000, a fifth of the cost in the U.S...

MedToGo has an agreeable web site. Owned and operated by US physicians, who are facilitating trafficking in the organs of the poor. I wonder; are there any state licensing board issues?

The organ trade is one of those curious stories that get little press attention.

Update 5/19/11: MedToGo's CEO wrote to object to the way they were portrayed in the Bloomberg article. They say they provide access to transplants performed in Mexico to Americans and Canadians, but only with American and Canadian donors. I am curious how that can be done, since I am sure they are bypassing North American transplant boards. They also say they do not pay donors, but they do not say the donors are unpaid. Based on MedToGo's response I've modified my post title and content as above.

See also:

• Organlegging Neuromancer style – China’s liver trade 5/10
• Enlightenment 2.0 and a Hong Kong student post on the Chinese organ trade (5/07)
• Niven scores: The organ trade grows (4/06)

Yakima fails a crucial test: treating customers well

I've bought four Yakima Steelhead bike mounts over the past decade; all through REI.

All four have broken plastic locking tabs. It's not a rare problem ...

Yakima Steelhead Bike Mount Reviews:

... the red plastic locking tabs are complete junk. I have six of these racks and three are broken, three I don't dare remove from the crossbars as they are about to fail. Yakima support will gladly send me replacements for a mere $54 per rack .... [Yakima's] choice of such a cheap plastic and unwillingness to replace at a reasonable cost has me shopping for Thule...

and

Amazon.com: Anonymous' review of Yakima Steelhead Fork Mount Rooftop Bicycl...

... I had Steelheads many years ago and when I needed a rack for my new car I decided to get another set.... Over time, exposure to the sun has caused the red plastic levers to crumble to pieces, and Yakima Customer Service told me that I would have to buy an entire replacement head which is about half the cost of a new mount. ... Luckily, it can be fixed with a stainless worm screw hose clamp threaded through the slot and around the head to keep the halves together. It doesn't look as nice but it's probably stronger than the original design. It would get 5 stars if Yakima provided service parts for the heads...

The plastic degrades in sunlight.

I'm going to take all four to REI and see if they can get more out of Yakima than their customers can.

Yakima has failed two major tests.

One is that they have known for years that they have a flawed design -- but they haven't fixed it. A second is that they have not offered a low cost replacement program, or a trade-in program to a fixed design.

I expect that from low cost brands, but Yakima is a premium brand. I expect more from them.

Buy Thule instead.

Update 5/26/11: Yakima failed, but REI reminded me why I carry an REI VISA card, and why I've been a member since we made fire from rocks. I paid $100 each for the two racks REI has purchase records for. Turns out I bought the "new" ones in 2003, which is as far back as their electronic purchase database goes. If I can find receipts for the old ones they'll pay for those as well. Considering the age of the racks it would be greedy to claim for all four.

REI rules.

Reliability and the Cloud - Redundancy required

Hardly anyone noticed, but yet another Google cloud service failed this week. There was understandably more attention to Amazon's recent service failure (2008 too). These aren't a surprise, I've had my share of complaints with Google's cloud services.

Despite all of the problems with Cloud services, of which the most serious is Cloud provider bankruptcy, Amazon and Google are relatively reliable. In my corporate workplace, the average worker loses 2-5 days of work each year due to machine upgrades, backup failures and hardware failures. Cloud services aren't quite that bad, but corporate IT is a low standard. Cloud services aren't good enough.

The answer to Cloud reliability, is redundancy. The designers of the late 20th century American space shuttle knew this well ...

... The shuttle uses five identical redundant IBM 32-bit general purpose computers (GPCs), model AP-101, constituting a type of embedded system. Four computers run specialized software called the Primary Avionics Software System (PASS). A fifth backup computer runs separate software called the Backup Flight System (BFS). Collectively they are called the Data Processing System (DPS)....

The design goal of the shuttle's DPS is fail-operational/fail-safe reliability. After a single failure, the shuttle can still continue the mission. After two failures, it can still land safely.

The four general-purpose computers operate essentially in lockstep, checking each other. If one computer fails, the three functioning computers "vote" it out of the system...

The Backup Flight System (BFS) is separately developed software running on the fifth computer, used only if the entire four-computer primary system fails. The BFS was created because although the four primary computers are hardware redundant, they all run the same software, so a generic software problem could crash all of them ...

It's not hard to do the math. A series of 5 procedures each with 90% reliability has a 40% chance of failure (1-0.9^5). A different system with 5 systems of similar reliability run in parallel has a 0.001% (.1^5) chance of failure.

In Cloud terms similar redundancy can come from multiple service providers, with the ability to switchover. File requests, for example, could be alternately routed to both Amazon S3 and to a corporate owned server. Reliability comes from two very different systems with uncorrelated failure probabilities [1][2].

This switchover requirements requires Cloud services to be dumb utilities - or to support some kind of local cache. To safely use Google Docs, for example, there has to be some way to fail over to a local device, perhaps by synchronizing files to a local store. Similarly, to use a Cloud blogging service one would want control of the domain name, and blog software that published to two services simultaneously. In the event of failure, the domain name could be redirected to the redundant server.

None of this is new. Back in the days when Cloud services were called "Application Service Providers" (ASP) I went through the same reasoning process with our web-based Electronic Health Record. I'm sure there were very similar discussions in the 1970's era of 'dumb terminals'. These things take time.

We'll know they Cloud is maturing when failover strategies become ubiquitous. Of course by then we'll call the Cloud something else ...

[1] Of course then the switch fails. There are always failure points, the trick is to apply redundancy to those that are least reliable, or where redundancy is most cost-effective. The Shuttle, infamously, couldn't survive a failure on launch of its solid fuel system.
[2] From a security perspective, two systems like this are two sources of security failure. Multiple systems increase reliability, but decrease security.

American torture dishonored us, but we can honor those who said No.

We have been doubly shamed over the past ten years. Once by our embrace of torture, a second time by our inability to prosecute.

We can't undo our past, but there are some things we can do. Even if, as Tracy Lightcap tells us, "Bush's C-in-C order of 13 November 2001 and the subsequent OLC memos (effectively DOJ opinions) make it virtually impossible to prosecute those involved in acts of torture..." we can still establish a truth commission to document what was done.

Even before a truth commission, however, we can begin with a small step ...

Honoring Those Who Said No to Torture - NYTimes.com
Jameel Jaffer is a deputy legal director at the American Civil Liberties Union. Larry Siems is the director of the Freedom to Write program at the PEN American Center

ON January 2004, Spec. Joseph M. Darby, a 24-year-old Army reservist in Iraq, discovered a set of photographs showing other members of his company torturing prisoners at the Abu Ghraib prison. The discovery anguished him, and he struggled over how to respond. “I had the choice between what I knew was morally right, and my loyalty to other soldiers,” he recalled later. “I couldn’t have it both ways.”

So he copied the photographs onto a CD, sealed it in an envelope, and delivered the envelope and an anonymous letter to the Army’s Criminal Investigation Command. Three months later — seven years ago today — the photographs were published. Specialist Darby soon found himself the target of death threats, but he had no regrets. Testifying at a pretrial hearing for a fellow soldier, he said that the abuse “violated everything I personally believed in and all I’d been taught about the rules of war.”

He was not alone. Throughout the military, and throughout the government, brave men and women reported abuse, challenged interrogation directives that permitted abuse, and refused to participate in an interrogation and detention program that they believed to be unwise, unlawful and immoral. The Bush administration’s most senior officials expressly approved the torture of prisoners, but there was dissent in every agency, and at every level.

There are many things the Obama administration could do to repair some of the damage done by the last administration, but among the simplest and most urgent is this: It could recognize and honor the public servants who rejected torture...

It's the very least we should do. We can start at the local level, then the state level, then the federal.

Does anyone know of an "honor role" of those who said No?