The New York Times' bad password advice - and what you should do instead

In the context of a site that claims to check passwords against a published hacker repository [1] Scientific American repeats the NYT's conventional wisdom about passwords and security ...

Observations: How to Know If Hackers Have Stolen Your Password:

... Is your email address listed in any of these databases? The New York Times reports on a easy-to-use web tool that a security professional has created that will check your email address against 13 different databases containing 800,000 email address/password combinations. Called, appropriately, "Should I Change My Password?", the site runs a simple search for your email in the known files. I checked my various emails, and fortunately, the tool didn't turn up anything amiss. But the site also gives some very solid advice: Change critical passwords regularly, and don't reuse the same password across multiple sites... [3]

This is bad advice. The fact that it's repeated ad nauseum doesn't make it any better. Schneier, the doyen of net security, debunked the conventional wisdom about 5-7 years ago [3]. Essentially, these six goals are not mutually compatible ...

1. Use a password that's resistant to password-guessing attacks
2. Change passwords frequently
3. Don't reuse passwords
4. Get stuff done (requires password actually working)
5. Give your partner access to critical accounts, including those s/he will need when you kick off.
6. Have a life

Given that most of us want to to get stuff done, and even have a life, what should a regular person do? Schneier hasn't summarized this recently, probably because he's become bored and discouraged, but I think he'd go with this list:

1. Use as few online services and accounts as possible. The more identities you have, the more you need to secure. If you give up on AOL, don't just add Google. Delete the AOL account. If you can't delete an account (all too common a problem) [4] then remove all of your personal information and email credentials, change the password to 128 random characters, and log out. It's as good as dead then.
2. Don't use important credentials (ex: Banks, Google, etc) on untrusted machines. Keystroke logger malware will defeat the world's greatest password. This includes work machines, anything running XP, and public machines. If you're running XP at home you need to switch to one of these platforms: iOS (most secure - iPad, etc), Win 7 with antiviral, or OS X 10.6+. [5]
3. On your trusted machine (iOS, Win 7 w/ antiviral, OS X 10.6 plus) do use strong passwords [6] on the accounts you care about. Since you should only have a few accounts you care about, you may reuse your secure passwords. If you reuse, consider adding a prefix or suffix that permutes the password, such as "Google", "Fidelity", etc. Don't store your passwords digitally, write them down on paper in your wallet and in a safe place in your home.
4. For the zillion accounts you don't care about, such as kid's baseball signup account, heavily reuse a robust password but assume it's public. Every year or two feel free to change it. Assume these accounts will be hacked -- but, really, who the heck would bother? There's no need to lock a shed that holds refuse! The trick here is that if you decide you do care about an account, you will need to give it a reasonably unique password.
5. Try to avoid the damned "secret questions". They are a huge security risk. I don't have a good answer to these plagues. They are the technological equivalent of Michelle Bachman -- a sign that humanity is a passing fad.
6. Use Chrome for your web browser. It's by far the most secure browser platform, and it includes its own firewalled PDF reader software.
7. Don't install Adobe reader or Flash. They're notoriously risky. This is more practical on OS X, and is a big advantage of OS X over Windows 7.
8. Don't install software that's not from a trusted source. This excludes, incidentally, most of the Android App Store.
9. On OS X, don't login as an Admin user, login as a regular user. I believe this is also possible on Windows 7.

Phew. That's awful, isn't it? Things are bad. There is hope however ...

1. Signed code is here with iOS (iPhone, iPod Touch, iPad and is coming to OS X [7]. This will dramatically decrease malware, including keystroke loggers.
2. Multi-channel multi-factor authentication is here and one day be useable by people with a life.
3. Identify management solutions are oozing out of the mire and will be built into future OS versions (iCloud, Android, ChromeOS)
4. Biometric authentication will work ... one day ... maybe ...
5. IP6, the next generation internet, enables new authentication and security technologies.

The above list of security guidelines is pretty bad, but they are doable by regular humans. Meanwhile, what about geeks who, after all, don't have a life to lose?

Here's what this geek does ...

1. I do enter my Google credentials on my relatively untrusted work machine -- but I use Google's two-channel two-factor authentication while avoiding their vulnerable SMS channel. Because I do that I assume my Google password has been compromised -- so I don't reuse it. This is pure geek stuff; Google has worked hard on their two factor but it's still a pain in the ass to use. They need to work on their iOS apps in particular.
2. I use 1Password on my iPhone and desktop. I need it as much to keep track of my usernames and the #$$!%!#$% secret questions as my passwords [8]. I don't love it, but it's the best solution I can find.
3. I print out my and the family credentials periodically so Emily has an easily accessible set in case of emergency. The password stores are not user friendly.
4. I don't trust the Cloud -- I don't store secret information on any Cloud service.
5. I have settled on using Google for my OpenID/OAuth service provider because of their two factor authentication.
6. Otherwise I follow most of the advice above. Today, after some equivocation, I removed Flash Player from my primary machine.

- fn -

[1] If it's legitimate, then the site runs a cryptographic hash function locally and compares the output to hashed versions of the password repository. I gave it an old disposable password, and to my surprise it didn't match anything stolen. I am pretty sure this site is legitimate, but it's a terrible practice to encourage civilians to enter their passwords for testing. At the very least, the site should be run by either the US government (think on that!) or by a corporation with a lot to lose.
[2] Before I went to the "two-channel" flavor of two factor. See below.
[3] For a full set of conventional wisdom, see Schneier on Security: Password Advice (2009): Note, if you don't read it carefully you think this is his advice. It's really the conventional wisdom.
[4] These days, before I sign up for anything, I check their account deletion policies. If they don't give me a clear path to account removal they don't get my business. See Gordon's Notes: Gordon's Laws for software and service use.
[5] Sorry, there's no nice way to put this. XP is finished.
[6] Schneier on Security: Choosing Secure Passwords (against an offline password-guessing attack) (2007): ".... a typical password consists of a root plus an appendage. A root isn't necessarily a dictionary word, but it's something pronounceable. An appendage is either a suffix ... or a prefix ... You should mix upper and lowercase in the middle of your root. You should add numbers and symbols in the middle of your root, not as common substitutions. Or drop your appendage in the middle of your root. Or use two roots with an appendage in the middle.... the seven-character phonetic pattern dictionary -- together with an uncommon appendage, is not going to be guessed. Neither is a password made up of the first letters of a sentence, especially if you throw numbers and symbols in the mix.... Personally, I just use Code Poetry's utility to run OS X Password Assistant and have it make me a memorable password.
[7] With robust Digital Rights Management and many other expected and unexpected side-effects. Unmitigated goodness is rare.
[8] I wrote a custom FileMaker credential management database back in the early 90s. I would prefer to use it on my iPhone, but FM is pretty much dead. Bento doesn't offer encrypted iOS databases.

See also:

• Gordon's Tech: Implementing Google's two factor authentication (4/2011)
• Gordon's Tech: RIP Password - Google's two factor authentication (2/2011)
• Gordon's Tech: Gmail: be sure you have a working secondary account (2/2009)
• Gordon's Tech: After the Gmail hack - passwords and security (9/2010)
• Gordon's Notes: Google's two factor authentication and why you need four OpenID accounts
• Gordon's Notes: U.S. Bank's ID Shield makes me scream
• Gordon's Tech: My Google (gmail) account is hacked - by ductus.com (9/2010) - still somewhat mysterious.

The wilfull wastefulness of the Foundational Questions Institute

The "Foundational Questions Institute" (FQXi, don't ask about the acronym) recently sponsored an essay question about the nature of reality, specifically whether it is fundamentally digital or analog. Is there, for example, a smallest slice of time? Or, if you suspect time is not fundamental but is some epiphenomena of entanglement, is there a way in which the quantum world is less digital than it seems?

Sponsored essays on speculative physics! Neat idea, and seems right up my ally. Of course FQXi's mission statement must attract a wide "variety" of thinkers (emphases mine) ...

... FQXi catalyzes, supports, and disseminates research on questions at the foundations of physics and cosmology, particularly new frontiers and innovative ideas integral to a deep understanding of reality, but unlikely to be supported by conventional funding sources...

Still, the advisory council includes Guth, Bostrom, Barrow, Rees, and Smolin. So I would have tagged them as potentially eccentric, but most likely interesting.

Would have tagged them I say - but not after I actually tried to read one of the winning essays.

It's a PDF. That's bad enough, but it's a PDF of badly scanned document.

This is pure madness. The FQXi is a sad waste.

Nimbophobia: 4 more reasons to fear the cloud

It's been a gratifying week for my fellow nimbophobics. Our numbers are growing by leaps and bounds. Consider just four examples ...

• Apple MobileMe transition and iCloud: Those beautiful online albums of thousands of images and videos? Kiss them all good-bye. Your massive web site? Sayonara.
• A few weeks after being caught dissembling about their encryption keys, Dropbox accidentally removes all security for all accounts for four hours.
• Google is shuttering its hugely hyped Google Health Personal Health Record. Hope you weren't relying on those online medical records you entered.
• My longstanding and much appreciated Google Custom Search pages are, as of today, abruptly overwhelmed with copious top and side Google ads.

These stories range from appalling (Apple) to annoying (excess ads in custom search pages). The Google PHR fail would be the worst, but it's somewhat mitigated by the data exit options they provide and by the two year warning. Those options include CCR XML migration to Microsoft's HealthVault [1].

Friends don't let friends rely on the Cloud. Don't put anything in the Cloud unless you have a way to move your data to an alternative platform. That's as true for your business processes as it is for your family photos.

[1] Any health informatics students looking for a semester project or an easy publishable paper? Create a PHR in Google Health Records. Export as CCR XML. Import into Microsoft HealthVault. Write a paper on the data loss.

Quantum macro

Living in a Quantum World: Scientific American by Vlatko Vedral is the headline article for SciAm's June 2010 issue. It's behind SciAm's remarkably successful paywall, but for the moment you can find a PDF in Taiwan [1].

It's worth a read for my fellow lay fans of Quantum Mechanics. It captures the excitement of the field, where the theoretically incomprehensible is now becoming the materially incomprehensible. Some highlights of note, recognizing that this article is one physicist's personal view ...

• Until recently many popular presentations of QM, even very fine ones, confined QM to the micro realm. Decoherence, arising from "information leakage" was supposed to flip from a bizarre "binary" quantum world of entanglement to a bizarre "analog" world of gravity and black holes. Vedral and others says it's all QM from the bottom to the top. There's no "flip" between quantum and classic.
• In one experiment 10^20 atoms of lithium fluoride behaved as though, at some level, they were all entangled
• There are claims, with some evidence, that quantum effects are leveraged by navigating birds and phyotsynthesis.
• If entanglement is truly fundamental, then space and time (arrow of time) may in time be seen as side-effects of entanglement (which, I suppose, would make "spooky action at a distance" oddly easier to understand)
• Even more speculative -- gravity is not fundamental, but is emerges as a side-effect of the three (not four) fundamental forces (weak, strong, electromagnetic). Supposedly "proper" quantum treatment of those forces will yield gravity, which would explain why it's been so hard to quantize gravity.

Perhaps the most interesting bit of the article was a somewhat frustrating description of a Schrodinger Cat variant Bob and Alice thought experiment from 1961 and 1986. I've never heard this one, and I can't find it described properly on the web, so I wonder if this is partly a modern interpretation focusing on how information leakage leads to decoherence [2]. Briefly, it goes like this

• Bob, the cat, the cat poison and cesium atom are in a room. Alice is outside. The cat poison is released if the atom decays. Bob can see the cat. Alice can't.
• The cesium atom is "quantum". It is in an indeterminate state of decay or integrity. That's "rock solid" quantum physics.
• Alice puts a piece of paper under the door. She asks Bob if he can see that the cat is dead or alive, but not what state the cat is in. Just that he can tell.
• Bob writes he can tell.

So at this point the the cat is dead or alive - at least for Bob.

Here's where I don't get it at all. According to quantum theory this is all reversible. Allegedly Alice can "undo" the observation, but retain the piece of paper. If Bob remembers seeing a dead cat, but Alice makes the poison inert, he'll remember seeing a live cat. So Bob, the Cat, the poison and the Cesium ion are all entangled and indeterminate for Alice, but for Bob they're all determinate. Smells a bit like frames of reference in special and general relativity.

Unfortunately the sidebar doesn't explain how Alice can undo the observation without a bit of time travel. So I suspect the explanation has been a bit butchered, but I'll keep an eye out for a better one (Google is no help today).  Supposedly the equivalent experiment has actually been done by teams led by Blatt and Wineland, and they've shown measurement reversal in the real world (did you just feel the  universe hiccup)?

[1] If you Google on a few unique words in an article, you can usually find one copy somewhere on earth.
[2] Much of the lay physics I read these days uses an information theoretic perspective; much of physics is expressed in the language of information. Reminds me of some of my favorite mind expanding science fiction,  particularly Greg Egan's Permutation City. In that book sentient natives of a simulation with inescapably absurd physics are designed to realize that their universe must be a simulation. Except they're so brilliant they come up with a plausible "natural" explanation, and so disrupt the simulation itself ...

Studies in paywalls: Scientific American

In the many discussions of paywalls I read, everybody mentions the WSJ, but nobody mentions Scientific American.

You can read the print edition, which is discounted heavily through subscription management services, or you can pay $40 to get the online version. SciAm has been doing this since 2009.

It is frustrating for me. I'd love to write about the articles, but it's too much hassle to do that for a paper publication. The online version is costlier than a discounted print subscription, and I like the magazine, so I'm not willing to pay for digital.

It's frustrating for me, but it seems to work for them. Unlike Newsweek, which is about ready to fit a business envelope, SciAm is holding its weight.

How do they do it? Is it institutional and academic subscriptions?

Bright side: Apple's computer for the rest of us

It's not the best of times. Long Depression 2.0 grinds on. China is increasingly unsettled -- and it's sitting on one of history's great bubbles. American corporations may have decided the American middle class is finished, done in by globalization and IT enabled automation and outsourcing. Spear phishing (Chinese?) caught white house "aides" (Obama?). Core security systems have been compromised. Peak Oil. Pakistan, North Korea, Yemen. The ChromeBook costs 200% too much. Weather badness and rising CO2.

Worst of all, I can't buy a quality dehumidifier at any price.

It's a bit much, even for me. I've got to find some happier things to say -- even if I've got to dig deep.

Today's happy thought - in Fall 2011 Apple will be make my Jan 2010 prediction true ...

Gordon's Notes: Computing for the rest of us: The iPad and the ChromeBook (Jan 2010)

.. The iPad's a pretty thing, but the combination of iVOIP and the return of the Mac Plus and the keyboard and $10 iWorks apps and the $15/month no-contract 250MB limited data plan might shorten Jobs time in Limbo.

... the 2010 [3G] iPad is more than $500 - but by 2011 the device will sell for under $500 with 3G-equivalent capabilities. An additional $15 a month will provide basic VOIP phone services (uses very little bandwidth) and access to email and Facebook Lite -- even before the advertising subsidies kick in. Of course free Wifi access, such as in libraries, McDonald's, schools and so on will provide access to full internet services....

... Think about your family. If it's big enough, your extended family will have at least one person who's, you know, poor. They may have cognitive or psychiatric disabilities. Or you may have a family member who, like most of American, can't keep a modern OS running without an on call geek. These people are cut off. They can barely afford a mobile phone, and they won't have both a mobile phone and a landline. They will have little or no net access. They may have an MP3 player, but it's dang hard to use one without a computer.

By 2011 the combination of a $400 iPad (and iTouch for less) and $15/month VOIP access will start to replace a number of devices that are costly to own and acquire, while providing basic net services at a rate that other family members can subsidize. Not to mention something pretty, which, speaking as someone who grew up poor, ain't a bad thing...

Apple's iCloud [3] and iOS combination mean most families won't need an energy sucking, loud, unstable, unsupportable, malware infested winbox. They will buy a signed-code curated app library iPad with integrated backup and offline media libraries [1]. They will also, unwittingly, accept FairPlay DRM -- which is the best balanced DRM system I've lived with [2].

This will make the world a better place.

Of course there's a silky black lining to the silver cloud, but let's not go there just yet ...

See also:

• The $80 ultra-portable - in unexpected form 9/2010
• Cricket’s $149 Android and the future $4000 Dell desktop 10/2010 - Post-PC predictions
• Buying my Chrome OS (XP) Netbook 7/2009 (Except I didn't. Now I'm buying a MacBook Air.)
• Why you will live in an iOS world 12/2010 - Signed code, curated software
• Computing for the rest of us: The iPad and the ChromeBook 1/2010
• I, Cringely - iCloud’s real purpose: kill Windows

[1] If money is tight however, and a user foregoes home internet service for the $15/month iPad data plan, they really don't want to be streaming their media library. They'll want to do their iPad backup at a local cafe or library.
[2] It's so good it's silently accepted. It's freakin' brilliant and Apple gets no credit. Of course they don't want credit -- because they don't want anyone to notice it. 
[3] In all the iCloud discussions so far there's mention of Apple's prior efforts at iTools, .Mac, and MobileMe. Few remember the 1980s AppleLink (later the basis of AOL when it was interesting) and the 1990s eWorld. Sixth time lucky?

Uncertainty and Long Depression II

Today's DeLong is about bond prices, and, in part, about the effect of uncertainty on savings decisions (emphases mine) ...

Hoisted from the Archives: If You Are Looking for a Monument to John Hicks, Look Around You - Grasping Reality with Both Hands

... Let me give you the Hicksian argument about what happens in a financial crisis--a sudden flight to safety that greatly raises interest rate spreads, and as a result diminishes firms' desires to sell bonds to raise capital for expansion and at the same time leads individuals to wish to save more and spend less on consumer goods as they, too, try to hunker down...

The economic event currently known as the "Great Recession" started and ended within the past 3-4 years. The period of economic uncertainty for many of us is older than that ...

Manufacturing collapse - Karl Smith - The Washington Post

... manufacturing did collapse. Yet, it’s hard not to look at the graph above and think that the real manufacturing recession began in 1999 and simply never stopped. What’s amazing is that we had any recovery at all....

I'm in the business of producing software; I believe for the purpose of US statistics I'm in manufacturing (we can capitalize production for example). For my sector the 1990s were a golden era. Since the 1990s things have been ... different.

The times have not necessarily been bad at leasts in terms of work income [1]. Many of us have similar or even higher incomes than we had ten years ago, even adjusting for inflation. The times have not been bad, but they have been turbulent. Close calls are frequent. Even if yearly winnowings have been modest, employment options have been comparably modest. "Lean" investments means there are no reserves, no excess capacity. Business infrastructure is shaky; IT departments are cut to the bone. Predictable small disruptions have major impacts on a weakened enterprise. Projects go slowly, and are easily derailed. Productivity falls.

I suspect most sectors of the US economy have had similar experiences, even in years of relatively good GDP growth. Even growth sectors, like healthcare, face enormous regulatory uncertainty.

Uncertainty has become endemic, and the outcome is, of course, that households spend less, save more, and save more as cash. Instead of being able to go for 6 months without income, we aim for 1-2 years. When millions of us reduce consumption, the economy shrinks.

What could the US do to reverse these trends, and climb out of Long Depression II?

The US could do a lot. Health care cost and access is a major contributor to economic anxiety. Obama's ACA is better than nothing, but it left coverage tied to employment in general, and employment with large corporations in particular. This was a necessary political compromise, but it has hobbled the ACA and made it easier for the GOP to sabotage America's future.

Beyond health care, economists like DeLong, Krugman and Jared Bernstein have a wide range of ideas.

Unfortunately, nothing will happen as long as the GOP can paralyze the US government [2]. The bursting of China's bubble and the Euro crisis will make things particularly interesting leading up to Nov 2012.

See also:

• Long Depression - Wikipedia, the free encyclopedia
• Third Depression Watch - Krugman 5/25/11
• Gordon's Notes: Causes of the Great Recession: China, GPSII and RCIIIT. Now for Act III. (4/2010)
• Gordon's Notes: Employment in the Great Stagnation (8/2010)
• Gordon's Notes: Whitewater world - the insane numbers of iOS and Android (8/2010)

[1] Investments have been utterly lousy however -- and we avoided all but the real estate crash. The reason we were relatively lucky, however, is due to uncertainty making us more conservative.
[2] Though it's certainly unjust, given the sheer wrongheadedness of today's GOP it's entirely possible that a Romney or Huntsman would be able to do more than Obama -- simply because they're only pretending idiocy and today's Dems are far higher quality than today's GOP.

The kidney trade thrives

The kidney trade is growing ...

• The village of the kidneys - Nepali Times: A village near Kathmandu is the "Kidney Bank".
• Desperate Americans Buy Kidneys From Peru Poor in Fatal Trade - Bloomberg: "An American man had promised to give Picado, a 23-year-old high school dropout who worked as a construction laborer, a job and an apartment in New York if he’d donate one of his kidneys. He jumped at the deal, his mother says."
• NK defector in legal dispute over organ transplantation
• For Chinese, kidney donation is a click away - Features - Al Jazeera English
• Blood & Treasure: redundant organs: The youth and motivation (iPad) of the donor made this more prominent.

Bloomberg has had the best recent coverage of the organ trade. A long and slightly rambling article provides important background ...

Desperate Americans Buy Kidneys From Peru Poor in Fatal Trade - Bloomberg

... Every year, about 5,000 gravely ill people from countries including the U.S., Israel and Saudi Arabia pay others to donate an organ, says Francis Delmonico, a Harvard Medical School professor and surgeon. The practice is illegal in every country except Iran, Delmonico says.

Affluent, often desperately ill patients travel to countries such as Egypt, Peru and the Philippines, where poor people sell them their organs. In Latin America, the transplants are usually arranged by unlicensed brokers. They’re performed -- for fees -- by accredited surgeons, some of whom have trained at the world’s leading medical schools.

The global demand for organs far exceeds the available supply. In the U.S., 110,693 people are on waiting lists for organs, and fewer than 15,000 donors are found annually.

Americans who go abroad for illicit transplants can contract infections or HIV from unhealthy donors, posing a public health threat when they return, Delmonico says...

... Medical tourism company MedToGo LLC, based in Tempe, Arizona, says it will offer kidney transplants in Mexico and Costa Rica for about $50,000, a fifth of the cost in the U.S....

... “The poor have become a spare-parts bank for the well-to- do,” says University of California, Berkeley, anthropologist Nancy Scheper-Hughes, who specializes in organ trafficking.

The Peruvian National Prosecutor’s Office is investigating 61 transplants in seven of Lima’s top hospitals since 2004, documents in the case show. Peraldo is one of 150 brokers, doctors, nurses and others under investigation, says Jesus Asencios, the prosecutor leading the probe....

Note the key feature of MedToGo is not their cost savings, it's that they procure kidneys in ways that circumvent, and perhaps violate, US, Costa Rican and Mexican law.

If the trade cannot be stopped, then it must be regulated. If a country decides it wants an organ trade, they can set a fixed rate that's paid every donor, regardless of whether the recipient is local or foreign. They can tax foreign transplants so that every foreign transplant pays for two local transplants.

Beating the odds - a software story

For the second time in my career I had the great privilege of imagining a software product and seeing it through to release. It started out as a key toolset for another product, then ended up, at the end of its initial product cycle, being repurposed for a significantly larger role.

This isn't the kind of software story you usually read about. We weren't a startup group of young developers, we were a small group of grizzled veterans in a large publicly traded company. We didn't have to worry about VCs, but we had our own version of funding uncertainty. If 37Signals is Superman, we were Bizarro. In a mirror world of software development, we faced our own set of grim obstacles.

Frankly, I'd rather face the obstacles of my startup days but, still, we did succeed. We succeeded because we had a great team (seriously great) and, at critical times, we had close collaboration with a great customer.

Here are a few of the things I learned in the process, in no particular order. In a small way, they were a recursive version of a far more ambitious project I read about many years ago - The Data General Eagle ...

• The core of our team was local, but we had key contributors that were remote. Our collaboration technologies were phones (and teleconferences - 1970s tech) [6], email, a Sharepoint 97 wiki [2], LiveMeeting screen sharing [4], and Rally [3]. When I ran meetings with remote contributors I had everyone dial in. We developed some good techniques for managing remote discussions, including sharing MindManager maps to record and organize discussions. The main lesson here is that you can get good results out of some very limited tech tools -- if you thing hard enough about how work around their issues.
• Early on I spent a lot of thought solving problems we never got to. Some of those imaginings led to patent applications [1], but they didn't have much impact on the product. In a few cases though, those solutions were critical. It was time well spent, but it's worth remembering that the real challenges are likely to be surprises.
• What we ended up with had a lot in common with my earliest designs. I don't know if that means they were good designs, but they did persist.
• The curse and joy of software is that there are so many different ways to solve a set of problems. The trick was figuring out what compromises to accept, even when two good alternatives combine in a troublesome compromise. This wasn't hard within our group, but it was challenging when we had to fit into other models. A mediocre compromise, however, is better than a breakdown in critical partnerships. We threaded the needle.
• Our best decisions weren't coming up with solutions to tricky problems, they were deciding what to keep and what to drop. We had to choose between throwing the seats overboard, or the luggage, or dumping fuel. We couldn't afford to get those choices wrong and we mostly got them right.
• Everyone contributed everywhere. I did everything but write Java code. Our engineers did designs. We were all analysts.
• I like Agile. We couldn't do it fully for several reasons, including the world in which we lived. I think though, we stayed true to the sprit of Agile. Rally helped, though I fear the developers have been too responsive to their customers. There are quite a few rough corners left over. Nothing's perfect, but Rally is pretty good.
• I liked the Agile philosophy of just enough architecture. The key for us was deciding where we needed solid foundations and where we could put up a low cost shed that we'd happily tear down when it decayed.
• There's a trick to choosing between a range of reasonable options. It doesn't matter so much which one is chosen, only that we don't spend a lot of time choosing.
• In the absence of proper resources, a well crafted email with an edited thread attached to a Rally story can be a reasonable stand in for a requirements document.
• Inbox Zero was very important form me. When I cleared my inbox and scheduled my Rally tasks [5] into my calendar I was usually in good shape. When either fell behind I was in trouble.
• No emotion. I worked hard to stay balanced. Things were tough when we were repurposed [7].
• No death marches. We are too old for death marches, it's not an option when you're over 40. Quality goes off a cliff. We rescoped or invented easier solutions rather than pulling all nighters.
• I reported out probabilities of success to management rather than predictions like "we'll do it" or "we won't do it". Somehow that worked better.
• I kept a complexity budget in my head for everyone on the team. We spent our complexity capacity carefully, targeting high value work.
• We kept moving. When we bogged down we stopped and moved on an easier path. There were quite a few obstacles we couldn't control, rather than try to knock them down we went around.

We had fun. It's satisfying to beat the odds.

--

[1] Software patents are a curse upon civilization.
[2] Once you figure out all the traps (don't paste rich text into the rich text editor!) and the trick to embed images, you find out the search is pretty good and that SP's wiki isn't all bad. SP as a document management system is pretty bad, though less intolerable with Office 2007 than 2003. This project made me a Wiki convert. (See also: Gordon's Tech: Vermeer/FrontPage lives in Sharepoint Wiki)
[3] Also, alas, StarTeam. I try not to think about StarTeam.
[4] Only the screen sharing.
[5] I got to write the stories and the tasks then do them. A bit inbred.
[6] I tried very hard to get Google Video Chat working. It failed for us. Partly due to bugs, partly due to a remarkably poor UI, but mostly because the corporate net connectivity was overloaded. This is a more common problem than most realize.
[7] Would have been helpful if I'd done the Conversations - From emotional confrontation to dialog class a year ago, but I'm not sure it would have made that much difference. Managing genuine conflicts and power struggles by phone is less than ideal. We had no travel budget.

Aging boomers and the coming Golden Age of Cyberfraud

Just one recent example: Aggressive Social Engineering Against Consumers

As we boomers age, there will be a rich supply of weakened herd members for online predation. The Golden Age of Fraud is coming.

This is why you will live in an iOS world.

Tornados and global warming - how do we judge predictions?

We can't forecast a tornado, and we can't predict how a tornado will behave. We can, however, characterize tornadogenic climates and geographies. As CO2 accumulates and the earth warms virtually all terrestrial climates will change. Because climates will change they will all become more or less tornadogenic. This seems self-evident; I don't think there's any controversy here.

There is lots of controversy, however, when we try to understand the causes of the great American Tornados of 2011. There is controversy too, when we try to predict what will happen over the decades to come. Will, for example, geographic regions experience an increase in tornados as the earth warms, only to see a decrease when it warms still more? Will "Tornado zones" migrate north, so that Arkansas will have fewer, but Minnesota more?

Insurance companies would dearly love to know. So would homeowners contemplating installation of a basement emergency shelter. Given the purported limitations of historic data, how can insurance companies and homeowners make decisions?

Consider the case of a fair coin. Flip the coin ten times and you get this: TTTTTTTTTT - ten Tails. What's the chance of a Head on the next toss?

It's a trick question. I said it was a fair coin. The chance of Heads is 1/2, just as it was for the previous 10 tosses. Reverend Bayes does not apply.

Now consider that the coin has been altered; it's no longer a fair coin. Flip the coin ten times and you get this: TTTTTTTTH. What's our best estimate of the chance of a Head on the next toss?

It's 1/10.We don't know anything about the coin, so our best estimate of future performance is past performance.

So we can measure tornados like biased coin tosses and, in 30 years or so, we'll get some reasonable answers.

We can do better than that though. I wrote recently ...

... The process of iterating on internally consistent models that make testable predictions, and revising those models when predictions fail, has transformed human history. It is the only guide we have to developing better medicines, understanding the universe, or predicting the consequences of CO2 accumulation...

Consider our biased coin. We might speculate that a variable gravitational field is causing bias. We may predict that if gravity is varying, then local clocks should diverge from distant clocks. Clocks seem unrelated to coin toss, but if we do find clock drift, then our varying gravity explanation for both coin bias and clock drift is strengthened. We can use that new understanding to make more accurate predictions of future coin toss outcomes.

In a connected system, like a climate, a model can be validated by shorter series of multiple measures. So a model that predicted tornadogenic weather might take decades to validate, but a model that predicts summer storms, winter snow and average temperatures might be validated in a shorter time.

At least that's what insurance companies must be banking on. There's a vast amount of money at stake, a good model would be worth a lot. Particularly if it were private ...

See also

• Demography, Design, Atom Bombs and Tornado Deaths - NYT (Revkin)
• Tornadoes and Warming, from the Archives - NYT (Revkin)
• Attributing single events to background conditions — Crooked Timber
• The Facts (and Fiction) of Tornadoes - NYT FAQ
• Gordon's Notes: Bayes theorem - in a nutshell
• Gordon's Notes: Separated at birth: alternative medicine and climate change denial

Understanding democracy

In a democracy, the primary talent of the ruler is winning elections. This explains why, although winning politicians are more clever than the average citizen, they are not significantly better at governing. This also explains why juries are wiser than one would expect.

Failures of American government are failures of American citizens. We get what we deserve.

We should try drafting our political candidates by lottery, then elect from this pool after a traditional campaign. We can start with a State House and see what happens.

See also:

• Campaign Finance Reform: Voluntary Mandated Sharing

Update: Ok, could we at least insist on licensing exams for Senators and Representatives? We license barbers for Pete's sake ...

Reconciliation May 2011: The posts I won't get to

Reconciliation for May 2011 ...

• Elderly Drivers Fail a Test (NYT) and Some Older Workers, Pressured to Quit, Fight Back (NYT): Why raising the retirement age won't work. Our brains don't last.
• The Blind Use The Visual Cortex To Process Sound (SciAm): I've long wondered about how we evolved the ability to read -- to convert images into sounds. It seems an unnatural use of the brain, even for an animal that must have been good at sign language. Given these latest findings, the ability to read may be just another manifestation of astonishing brain plasticity.
• The Early Days of Eugenics: 1911 SciAm article on Eugenics. "The proper attitude to be taken toward the perpetuation of poor types is that which has been attributed to Huxley. "We are sorry for you," he is reported to have said; "we will do our best for you (and in so doing we elevate ourselves, since mercy blesses him that gives and him that takes), but we deny you the right to parentage. You may live, but you must not propagate."
• Medicaid a Big Deal Too | Talking Points Memo: The real budget problem is not social security or medicare or medicaid. The real problem is that our bodies outlive our brains. Demographers knew this day was coming back in the 1970s. That's when we should have invested in slowing the onset of dementia.
• Third Depression Watch - NYT: The good news is that Krugman is no longer talking about the Great Recession. The bad news is that he's talking about the 2nd Long Depression. (See also: Leonhardt, 2008)
• Frothy Real Estate in Manhattan: $5.5 million apartment
• Reassessing The Cost Of The Post-9/11 Era, Post Bin Laden (Froomkin): Was bin Laden the most successful investor in history? With our help he turned mere millions into a successful trillion dollar attack on America.
• Jann Wenner is my hero - Cringely and The People vs. Goldman Sachs | Rolling Stone: Corporate control of American media - more than you think
• Software Reviewing as it Should Be Done - James Fallows and Zeoli on Tinderbox: A quixotic effort to push the boundaries of Xanadu, but not a word about data lock.
• Inequality Rising Across the Developed World - NYT Rampell: Not only America.
• Quote of the Week, Last Week Edition - Jeffrey Goldberg: Obama's famous quote on birthers, spoken while he was secretly leading the bin Laden hit.

The Data General Eagle: Tom West and an old paper of mine

Tom West died recently. He was made briefly famous by a Tracy Kidder book, The Soul of a New Machine.

Reading his story, I remembered that I co-wrote* an essay on the Eagle project, drawing from Kidder's book and interviews with Data General veterans. It was written using MORE 3.1 @1996. I liked the paper and I eventually put a PDF online in 2001. It's still available from my old archived FrontPage web site: The Data General Eagle Super-minicomputer : A project management paper.

The lessons of those days still apply today. Not much has changed ...

Happiness as editing - Sunnyside edition

“No story ends happily. The happy ending is only about knowing where to end on a smile, at the very moment where fortune is still on the ascent. The open road. The wedding." Sunnyside by Glen David Gold, as quoted in Sunnyside II – Count no man lucky until he is dead — Crooked Timber.

Happiness is all about the editing. Things happen around us, we make up our own stories. If we're smart, we pick the happy moments to declare a chapter done, signed, sealed for ever. It can't be undone, the unchanging past is more eternal than the universe.

Some chapters are long, some short. The last chapter ends badly, but it's only one among many.