PLATO Notes, Microsoft Groove, and the curious history of software

This is a bit more software-centric than my usual 'Notes' postings, but it's really not about a particular technical issues, rather it's an interesting and topical ancectdote about how software evolves. Once upon a time I thought software largely came from the imagination of a few people. Sometimes it does (for better or worse), but most complex software projects have a long and often unrecognized legacy.

Groove is in the news today, it's a software solution recently acquired by Microsoft. Ray Ozzie, the CEO of Groove, will become Microsoft's Chief Technologist. Microsoft's involvement has created the recent interest in this "new" software.

Groove seems new, but it's been in development for at least six years. It's not six years old, however, because it's an offspring of Lotus Notes, which was developed in the 1980s. But it's not twenty years old, because it's really a descendant of PLATO Notes, which was developed at the University of Illinois in the 1970s atop the 1960s (1950s?) PLATO platform. So it's thirty years old. Heck, one could argue that it's really a child of the Memex (1945), so it's about sixty years old.

This is what Kapor of Lotus/spreadsheet fame wrote about the connection of Groove to PLATO:

Mitch Kapor's Weblog: Microsoft Acquires Groove
Ray has been a colleague and friend for over 20 years. He came to Lotus is 1982 with the vision of Notes already in mind, having been inspired by the PLATO system he used as an undergraduate at the University of Illinois...

Kapor's posting led me to a Google search, and thus quickly to a history of PLATO Notes, a pre-PC system for communication and collaboration. The history is well worth reading for anyone who develops or works with complex software systems, or who is just interested in the history of ideas. There are lessons there about electronic community (10 million hours!), about open source development, about the software development process, about software evolution, about software-as-platform -- and more besides.

There are also some minor personal serendipities here. I am writing this on a blog, a modern version of the kind of collaborative community that PLATO pioneered. I live in Saint Paul, and PLATO Notes was commercialized by a Minneapolis company -- Control Data. I have worked with many Control Data veterans who no doubt have connections to the CDC PLATO team, but, in addition, I have a longstanding interest in collaborative software systems (warning: old web pages). About 8 years ago my interest led me to review several alternatives and to comment on the work of David Woolley and his web conferencing guide.

David Woolley, as a young man, created PLATO Notes in 1973; he wrote the article I mention above. David is also a leader at Minnesota e-Democracy, which I've long appreciated. I shall have to send him a note of appreciation.

Update 3/15: David Woolley corrected some errors I made in dates. Thanks David!

What's worse than no privacy? Lies.

A while back I posted on the (gross) errors that my shadow medical profile is likely accumulating thanks to a persistent billing error. I thought I was making a prediction, but tomorrow is today. The data stolen from ChoicePoint (and Lexis/Nexis and everywhere else) is full of errors:

MSNBC - ChoicePoint files found riddled with errors
By Bob Sullivan MSNBC

... Pierce, a privacy advocate, obtained her report nearly two years ago, long before the current controversy. Thanks to the unknown source -- perhaps a company employee, Pierce said, but she has no way of knowing -- she got a rare privilege most consumers don't: a chance to see what ChoicePoint knows about her.

... What first caught Pierce's eye, she said, was a heading titled "possible Texas criminal history." A short paragraph suggested additional, "manual" research, because three Texas court records had been found that might be connected to her. "A manual search on PIERCE D.S." is recommended, it said.

Pierce says she's only visited Texas twice briefly, and never had any trouble with the law there.

"But if I was applying for a job, and there were other candidates, and this was on my record, the company would obviously go for another person," she said. "It raises a question in your mind."

... On ChoicePoint's Web site, the National Comprehensive Report is described as a collection of searches that glean data from "national and state databases for a summary of assets, driver licenses, professional licenses, real property, vehicles, and more. Each report offers the ability to add associates to the report, which include relatives, others linked to the same addresses as the subject and neighbors."

... Under former addresses, an ex-boyfriend's address was listed. Pierce said she never lived there, and in fact, he moved into that house after they broke up. The report also listed three automobiles she never owned and three companies listed that she never owned or worked for.

Under the relatives section, her sister's ex-husband was listed. And there are seven other people listed as relatives who Pierce doesn't know...

...Most alarming to Pierce is the fact that, with all this information, the ChoicePoint report she received had glaring omissions, too. Many of her former addresses aren't listed; and despite the host of other people listed on her report, many relatives and nearby neighbors were missing.

... Pierce's experience neatly parallels that of Richard Smith, another privacy advocate, who paid a $20 fee and received a similar report from ChoicePoint several years ago. The company offers a wide variety of reports on individuals; Smith purchased a commercial version that's sold to curious consumers.

Smith's dossier had the same kind of errors that Pierce reported. His file also suggested a manual search of Texas court records was required, and listed him as connected to 30 businesses which he knew nothing about.

Some of the mistakes on Smith's report were comical: That his wife had a child three years before they were married, that he had been married previously to another woman, and most absurd, that he had died in 1976...

It's a longish article, and pretty depressing. The quality of the data is awful, it can control your destiny, you can't see it and you can't fix it. I would have been surprised if anything else were true. Most of my blogly bloviating is pure opinion; in this particular domain I have actual expertise (shock! It's even exotic expertise). Even if there weren't inevitable and severe matching errors associated with gathering data from multiple sources, the data would only be as good as its sources. Then there's the risk of the inevitable inferences that must be performed to process the data. Lastly, there's ChoicePoint's motivations. They don't get in trouble if they label someone a "child molester" who isn't -- how would anyone find out? They get in serious trouble if they mislabel a "child molester" as "clear".

Think about it. What are they more likely to do? Err on the side of labeling a good person as bad, or a bad person as good? Which error costs them money?

Add these four things together:

1. Fundamental problems related to "matching" identities managed in different systems.
2. Mismatch between the quality needs of the acquiring systems vs. the use to which the data is put by ChoicePoint.
3. Semantic issues too complex to mention here.
4. The intense motivation to err on the dark side of life.

and it would be astounding of ChoicePoint records were not full of severe errors and prone to cause harm to the (relatively) innocent.

Alas, do Americans care? Not now they don't. They will one day.

PS. David Brin covered this topic in great depth many years ago. He wrote a book about it (The Transparent Society), but you get the main ideas here and (more recently) here. The Amazon reviews make interesting reading -- the best are pained admissions that Brin might be right.

Why going open source is not a trivial thing

Mitch Kapor's Weblog: Should Groove Have Gone Open Source?

If you work with software this is a quite important statement. Read it carefully and think about it. I believe these issues are also true if one wishes to deliver a "solution platform" rather than an application. I also have a prejudice that the same designs that enable an open source or platfoorm approach may also allow long term software growth and evolution -- but I have no proof points for that prejudice. In contrast, Kapor has lots of proof points:

... There are advantages to going open source as well as challenges. In some cases it may even be necessary to forestall a competitive threat, i.e., do it before it is done to you. When I see businesses whose strategies involve defending a class of business model which is simply going to be obsolete going forward, my heart sinks about all the wasted effort.

Caveat altert: In a transitional era like the one we are in now, it is notable that it's harder to convert a code base developed in a proprietary context to be open source than it is to start from scratch for the same reason renovating a house completely is harder than new construction. Trust me if you haven't been through this. I have. This is one of the reasons it took seven years from the day Netscape announced it was going to open source the Mozilla browser to get to Firefox 1.0.

It typically requires a complete overhaul of the code and the development process, which is much harder than starting from scratch. Typically, the existing code base is not one which is amenable to community development. There is major code re-factoring and rewriting to be done, rethinking and reworking of API's, switching to open standards, and changing of the tool set to use transparent, community-oriented tools for source code management, issue and bug tracking, build status, knowledge base, and synchronous messaging.

The Lefkow murders, mental illness, and jumping to conclusions

ABC News: Man Claims to Have Slain Judge's Family

Judge Lefkow's husband and mother were murdered by a mentally ill man unrelated to the neo-Nazi groups that had threatened her in the past.

I give credit to the police and commentators for not jumping to conclusions about the neo-Nazi connection. The articles I read, in newspapers and blogs, were generally fairly cautious. Hale and his kin were obvious suspects, but both the police and the commentators were clear that there were many possible alternatives.

I haven't seen, however, much discussion of the mental illness aspects of this case. Bart Ross sued his care providers because of what sounds like some fixed beliefs (delusions) about his care. He lived alone and had few social contacts. His suicide note suggests a very troubled and guilt-racked person, not the typical sociopath.

I hope the mainstream newspapers will delve into the deeper story, the important story. Was Bart Ross paranoid schizophrenic, perhaps with some superimposed dementia? What kind of care did he receive for his illness? What is the best way to care for mentally ill adults, particularly paranoid schizophrenics?

Paranoid schizophrenia is a terrible disease that inflicts great suffering on both ill persons and their families. It's also associated with violent and irrational acts. We need to understand it much better, and we need to treat it far, far better than we do. The mainstream newspapers feel neglected and unwanted these days, but this is the kind of story they can and should do well. Perhaps Judge Lefkow, once she has time to grieve, will use her story to avert future tragedies.

GOP theology: a return to basics

Salon.com | The gospel of the rich and powerful

Joe Conason complains about the Bush agenda. I do too, but there's not much new to say. We fought hard because we knew the stakes. We lost.

His summary of GOP theology, however, is interesting. I've been saying the same thing for a while, but Conason is a writer and I'm a hack (emphases mine):

... Appalling as these policies may be, however, they are in no sense inconsistent with the cosmology of the religious right, which melds laissez-faire economics with fundamentalist orthodoxy. Underlying these conservative attacks on the poor by professing Christians is a worldview that dates back to earlier centuries, when the church defended privilege and declared that the wealthy and powerful were God's elect. From that perspective, minimum wages, subsidized health care, and other such laws and regulations only corrupt the poor, who must earn charity by their temporal and spiritual submission.

If these ideas sound a bit old-fashioned -- or even primitive -- be assured that they represent the latest thinking on the evangelical far right, which is where "compassionate conservatism" originated. Guided by the most literal interpretation of Old Testament law, the preachers who have influenced the President are determined to undermine every modern protection enjoyed by poor and working-class Americans. Let's hope they draw the line at bringing back public whippings and debt slavery.

Conason expresses these core ideas far more succinctly that I have. The premise that wealth and power is a sign of God's blessing predates medeival christianity, it predates monontheism, heck, it probably predates language. It is the oldest and the most powerful theology. It is resurgent in America and it is the theology of George Bush.

Given human nature, it's not surprising that the old ways are back. What's odd is that there ever were a few powerful people who actually tried to follow the inhuman, illogical and unattainable teachings of Christ. What's peculiar is that the Book of Job hasn't been deleted from the Old Testament.

By comparision the theology that wealth is virtue, and that poverty and disease are a sign of God's virtue, makes perfect sense. It's good to return to a sane world where the powerful behave as they ought to.

Faughnan's Notes needs a new home - Blogger doom

Faughnan's Notes

I've given up on Google's Blogger/Blogspot. They've had periods of reliability over the past four months, but the rule has been unreliable and mediocre service. The past week has been quite dismal.

I'm looking for a new host. I expect to pay; I've never been a fan of "free" services and Blogger has reinforced my prejudices. (BTW, I think it's fascinating that even while Microsoft's desire to sell leased software has been frustrated web apps have been able to deliver the same result from a different direction.)

If you have recommendations for a host or software solution please email me at jfaughnan@spamcop.net. I'm leaning towards installing a package on my current web hosting solution (lunarpages - faughnan.com).

Patrick Leahy and privacy

Senator predicts 'overdue' changes to privacy | CNET News.com

WASHINGTON--Recent data mishaps at ChoicePoint and Reed Elsevier Group's LexisNexis service could usher in a dramatic reshaping of privacy laws, a U.S. senator predicted.

Vermont's Patrick Leahy, the top Democrat on the Judiciary committee, said Wednesday evening that a recent slew of data thefts and other leaks requires a 'comprehensive rethinking' of the laws regulating companies that compile electronic dossiers on Americans that are typically purchased by creditors, employers or police.

Howard Dean. Patrick Leahy. I want to live in Vermont.

If Paul Wellstone had not died in a plane crash prior the 2002 senate race, then Norm Coleman wouldn't be Minnesota's (wretched, awful) senator and Patrick Leahy might chair the judiciary committee. Then we'd be a happier and healthier nation.

The one bright spot is that there's always been a fair interest in privacy in conservative America. Of course this interest was stronger when Clinton was president, but it's not entirely gone. I'm not optimistic however. There just hasn't been much real outrage about the Lexis/Nexis and ChoicePoint security breaches, or the flaws in the ChoicePoint profiles. As far as I can tell, the American people don't really care.

Remember Argentina: The US trade gap and the charity of strangers

BBC NEWS | Business | US trade gap expands to $58.4bn

...The trade deficit makes up much of the red ink in the US's current account [deficit], the measure of the divergence between the US's incomings and outgoings.

Both the current account and budget deficits - the latter currently close to $500bn - are funded largely by the purchase of dollar-denominated debt by foreign central banks.

The inflow of about $2bn a day, estimated by some economists to be about 80% of the world's excess savings, also keeps the dollar from sliding further and faster.

We Americans keep spending, you foreigners keep sending us money so we can spend. What a deal! I think I'll buy another Mac. Or maybe a Mercedes.

Meanwhile Manhattan, previously sold to the Japanese and the Saudis, is now being sold to the Italians. Somehow I suspect New Yorkers came ahead on each sale.

We Yanks are partying like the boom and bust never happened. We don't pay taxes, and foreigners pay for our government's services and invasions. We buy Armani and foreigners keep our credit card rates low. Those bogeyman bond traders who are supposed to threaten irresponsible nations just wink at us. Greenspan keeps the taps runnin' and foreign money flows out. Yeah, our homes cost bazillions now, but that's not inflation, that's wealth. Right?

Ok, I'm no economist. Modern economics reminds me of modern physics -- both have far outstripped my dwindling intellectual resources. The universe is an infinite pile of infinitely expanding grapefruits, and the flow of capital between the US, China and India means we can party forever.

All the same, I'd be interested in an investment strategy that allowed me to make a goodly bet that the house of cards collapses before 2010. I'm willing to lose the bet (but not all my assets!) if the game goes on. Anyone know a legitimate place that offers such wagers to non-billionaires? (There's a name for this kind of complex financial instrument but I need more coffee to remember it. [1])

[1] Google came to my rescue. The search on "complex financial instrument" provided a nice list from DeloitteLearning (course costs money and is IE only!). The term is "Derivative and it takes one deep into the bowels of finance -- a field where mathematicians have gone to play. So all I need is a packaged derivative that won't wipe me out if it goes south, but lets me bet that the wheels come off the train.

Phil Bradley's posts on search and net services: March 7-11

Phil Bradley's Blog: March 7-11

Phil is a librarian, with a special interest in net search and design. I always enjoy following his blog, but in the past few days (3/7-11) he's had some exceptional entries. I wonder if he's been at a conference recently. I'll post separately on a few of the things he mentioned, but it's worth going through the March archives and reading from 3/7 onwards.

Blogger outage

Faughnan's Tech: Blogger was a mess today!

Blogger/blogspot has been out of order for at least the past day or so. I don't know how reliable it will be today. If anyone needed proof that Google was imperfect then they need only turn to Blogger. It's the Google equivalent of Microsoft Word.

The disadvantage of "free": no threats!

Faughnan's Tech: Blogger was a mess today!

A real mess. Sadly, Blogger is a "free" Google property. I can't threaten to deprive them of a revenue stream because I don't pay them. Blogger needs a business model.

Plagiocephaly and the law of unintended consequences

Baby's First Helmet (washingtonpost.com)

This is an intelligent and accessible discussion of plagiocephaly, aka 'funny looking baby heads'. We ask parents to place babies on their backs to reduce the incidence of sudden death (seems to work!), but this has an unintended consequence -- squished heads. The problem probably resolves on its own, but noone can guarantee full resolution. The helmets seem to speed up head rounding and they seem troublefree. They're not cheap though!

Prodigies and their futures

Boston.com / News / Boston Globe / Magazine

An interesting article on the prodigy and their outcomes.

I was never a prodigy, but I knew a few in my undergrad days. Those I knew were more or less like most bright people, but they could learn just about anything with astounding speed. A typical trait would be to goof off throughout much of a semester, then cover the course material in the last two weeks and finish with an excellent grade. They didn't necessarily have exceptional ideas or insights (though they did ok!); there doesn't seem to be a strong correlation between unusual creativity and being a prodigy. Reading between the lines in this article, it seems some of the pressure prodigies experience is caused by the false beliefs of parents, teachers and the children themselves that their extraordinary ability to learn also implies an extraordinary ability to create. If they turn out to be only as creative as other very bright people, they may feel as though they've failed. The children mentioned in the article also seemed to have somewhat intense parents.

Lastly they all seem to feel a frustrated obligation to try to fix the world. Hmm. I'm neither a genius nor a prodigy, but that I can understand.

One of the curiosities of a prodigy's life is that while they start out immensely far ahead of their peers, but by their twenties they start to slide into the 99% percentile rather than the 99.999%. To them that can feel like a big drop. A prodigy seems to be a mixture of both genius and accelerated maturation of the brain; by adulthood they lose the maturational advantage but may continue to be a genius.

Of course genius (based on IQ) doesn't translate to "success" (fame, wealth, offspring, sex - whatever) either. Glorious success is a funnny thing, a mixture of talent, luck, perseverance, and inclination. IQ helps with the talent part, but learning ability is only a portion of a quarter of the "success equation". If IQ alone is less than 1/8th of the "success equation", then it's no surprise that most of the adult prodigies are not "successful" on a national stage. Of course an enhanced ability to analyze and understand the world may also limit one's desire to sell body and soul in the pursuit of fame and power.

UK: National programme for IT

Medify Solutions

A UK vendor has put up a nice non-technical overview of the UK's ambitious (to put it mildly) national IT program.

The US put men on the moon. The UK is launching a massive healthcare IT programme. I think the US took the easier challenge. I'm hoping the UK puts their team on the "moon".

Upgrading mice

Boing Boing: Mice with human brains coming?

...the university's ethics committee approved the research, under certain conditions. Prof Henry Greely, the head of the committee, said: 'If the mouse shows human-like behaviours, like improved memory or problem-solving, it's time to stop.'

It's reassuring that these guys know when to stop, I am now confident these mice will not acquire any civil rights.

Next up: augmenting fly brains and flying jets with mouse-human neural tissue.