Wikipedia and the I-35W collapse

As expected, Wikipedia now has a good initial entry on the I-35W Mississippi River bridge collapse. This will be the place to go over the next few weeks and months to get an understanding of what happened. The Wikipedia article references the National Bridge Inventory but I was unable to find any record for the 35W bridge at this time. I will be most interested in seeing a list of bridges of similar age, design and history -- though one or more of construction error and local geology may turn out to be relatively unique contributors.

The Wikipedia article links to the Silver Bridge collapse of 1967, the outcome of the subsequent investigation suggests what may lie ahead:

... The collapse focused much needed attention on the condition of older bridges, leading to intensified inspection protocols and numerous eventual replacements. There were only three other bridges built to a similar design, one upstream at St Mary's and a longer bridge at Florianopolis, Brazil. They were both closed immediately, and the St Mary's bridge demolished in 1971....

I suspect we'll know within a few days which bridges in the US have a similar design.

Update 8/2/07: Culturally, slow to anger, but ....

Update 8/2/07: I've seen mentions of a bridge failure in 1983, the reference is to metal corrosion that caused the collapse of the Mianus River Bridge in Connecticut, killing 3 people and injuring 5 others. Wikipedia has a list of bridge disasters.

TSA Administrator on No-Fly list

I was wondering when Schneier would get to the "no-fly list" part of his interview with the TSA Administrator, Kip Hawley. It came out today.

My read is that Hawley was defensive and his answers this set of questions were weak. Here he discusses the new redress procedure ... (emphasis mine)

Schneier on Security: Conversation with Kip Hawley, TSA Administrator (Part 3)

...KH: ... if someone is either wrongly put on or kept on, the Terrorist Screening Center (TSC) removes him or her immediately. In fact, TSA worked with the TSC to review every name, and that review cut the no-fly list in half. Having said that, once someone is really on the no-fly list, I totally agree with what you said about appeal rights. This is true across the board, not just with no-flys. DHS has recently consolidated redress for all DHS activities into one process called DHS TRIP. If you are mistaken for a real no-fly, you can let TSA know and we provide your information to the airlines, who right now are responsible for identifying no-flys trying to fly. Each airline uses its own system, so some can get you cleared to use kiosks, while others still require a visit to the ticket agent. When Secure Flight is operating, we'll take that in-house at TSA and the problem should go away.

So the list was so bad an initial review dropped it by 50%. I hope Hawley is simply dissembling and that he's not so foolish as to think his answers are reasonable. Note the problem with each airline having their own procedures for dealing with "cleared" names. What a mess.

A local bridge falls

A local bridge fell into the Mississippi today. We're on the front page on the BBC World:

BBC NEWS | Americas | Six dead as US bridge collapses

...Russ Knocke, a spokesman for the Department of Homeland Security, told the AFP news agency: 'There are no indications of a nexus to terrorism at this time.'

The initial reports said "not a terrorist event", but of course that's unknowable. Later reports used "No indications", which is at least plausible.

We're a pretty wealthy state without any recent history of major infrastructure problems, and there's been no significant work on that bridge recently. Bridge engineering is supposed to be very well understood. So either our engineering textbooks need a rewrite, or there was something wrong about the initial construction or later repair of this particular bridge (and its peers?), or somebody blew it up. The last seems very unlikely but the other explanations aren't too likely either.

Journalism in transition and not-so-irrational voters: Rosenberg

Scott Ronsenberg, who's just left Salon, wrote two posts that merit a broader attention. The first is simply a response to Kristoff's recent "irrational voter" essay. Rosenberg expresses exactly what I thought as I read Kristoff's article. I think the American voter is future shocked, comatose, AWOL and derelict, but I thought Kristoff's examples of irrationality were .. umm ... irrational. Rosenberg captured my thoughts perfectly:

Those darn irrational voters.

... What are the ways in which voters are “worse than ignorant”? Kristof summarizes Caplan’s complaints of “systematic error” in voter rationality: Voters share “a suspicion of market outcomes and a desire to control markets.” They have “an anti-foreign bias,” evidenced by an unwillingness to embrace free trade wholeheartedly. They share “a neo-Luddite bias against productivity gains that come from downsizing or “creative destruction.’” And they have a “pessimistic bias, a tendency to exaggerate economic problems.”

Gee, it sounds like the real problem Caplan has with the voting public is that they don’t agree with the program of conservative economists!...

...Personally, I’m reasonably comfortable with the pro-free-trade argument. But you won’t find me sneering at those who sense that the dynamic of the global economy is not doing them or their families any good.

More significant in the longer run is Rosenberg's summary of the state of the journalist in the Murdoch-WSJ era:

Scott Rosenberg’s Wordyard » Blog Archive » Murdoch, the Journal, and the newsroom diaspora

It is no surprise that Rupert Murdoch will be the new owner of Dow Jones and the Wall Street Journal...

...The truth is that most professional journalists in the U.S. have lived in a cocoon for decades...

... I don’t trust Rupert Murdoch. He has a long and well-documented record of using his properties to further his own agenda. But I trust that there are a lot of smart writers and editors at the Journal. Either they’ll get an opportunity to reshape their paper in a way that suits the times and their own consciences — or they’ll find themselves in the great newsroom diaspora with the rest of us, helping us figure out new models for the future.

WSJ journalists have no excuse for complaining about the cold winds of capitalism, I'm sure they're too smart to expect much sympathy. I do hope that the best of them take Rosenberg's challenge and join a new world, but I also remember that when BYTE died, nothing replaced its value. (The sum of the entire tech blogsphere is probably the closest thing we have now, and that took about ten years post-BYTE to emerge.)

PS. As to the fate of the WSJ, as I've written before, I'm an optimist. At worst it will stay about the same. At best Murdoch will eliminate the editorial staff and keep the news people. Next best is to keep the editorial staff but so weaken the news function that it dies a slow death. All improvements.

The way we think: reason as an afterthought

Towards the end of an article reporting how much unnoted context alters our feelings and behaviors, a NYT article describes the implication for how we think. It's another in a long series of blows to the idea that we're fundamentally rational (emphases mine)...

The Subconcious Brain - Who's Minding the Mind? - New York Times

... an area called the ventral pallidum was particularly active whenever the participants responded.

“This area is located in what used to be called the reptilian brain, well below the conscious areas of the brain,” said the study’s senior author, Chris Frith, a professor in neuropsychology at University College London who wrote the book “Making Up The Mind: How the Brain Creates our Mental World.”

The results suggest a “bottom-up” decision-making process, in which the ventral pallidum is part of a circuit that first weighs the reward and decides, then interacts with the higher-level, conscious regions later, if at all, Dr. Frith said.

Scientists have spent years trying to pinpoint the exact neural regions that support conscious awareness, so far in vain. But there’s little doubt it involves the prefrontal cortex, the thin outer layer of brain tissue behind the forehead, and experiments like this one show that it can be one of the last neural areas to know when a decision is made.

This bottom-up order makes sense from an evolutionary perspective. The subcortical areas of the brain evolved first and would have had to help individuals fight, flee and scavenge well before conscious, distinctly human layers were added later in evolutionary history. In this sense, Dr. Bargh argues, unconscious goals can be seen as open-ended, adaptive agents acting on behalf of the broad, genetically encoded aims — automatic survival systems.

In several studies, researchers have also shown that, once covertly activated, an unconscious goal persists with the same determination that is evident in our conscious pursuits. Study participants primed to be cooperative are assiduous in their teamwork, for instance, helping others and sharing resources in games that last 20 minutes or longer. Ditto for those set up to be aggressive.

This may help explain how someone can show up at a party in good spirits and then for some unknown reason — the host’s loafers? the family portrait on the wall? some political comment? — turn a little sour, without realizing the change until later, when a friend remarks on it. “I was rude? Really? When?”

Mark Schaller, a psychologist at the University of British Columbia, in Vancouver, has done research showing that when self-protective instincts are primed — simply by turning down the lights in a room, for instance — white people who are normally tolerant become unconsciously more likely to detect hostility in the faces of black men with neutral expressions.

“Sometimes nonconscious effects can be bigger in sheer magnitude than conscious ones,” Dr. Schaller said, “because we can’t moderate stuff we don’t have conscious access to, and the goal stays active.”

..Using subtle cues for self-improvement is something like trying to tickle yourself, Dr. Bargh said: priming doesn’t work if you’re aware of it. Manipulating others, while possible, is dicey. “We know that as soon as people feel they’re being manipulated, they do the opposite; it backfires,” he said...

Really, it's amazing we do as well as we do. Our mind seems a pretty thin veneer on a heck of a lot of evolutionary programming. I am reasonably certain, however, that self-awareness varies from person to person. In other words, consciousness, like strength, speed, and wit, is a variable. Perhaps with some training we can begin to acquire second hand access to our unconscious controllers, and subvert them. So, perhaps we cannot directly detect an unconscious motivator, but perhaps we can become better at evaluating our own behaviors. When we find ourselves skeptical, or friendly, or generous, or competitive, we might then infer the presence of an unrecognized trigger, and thus infer our unconscious goals.

More in another blog on the implications for the management of persons with behavioral problems ...

Recommender systems - so that's why they've been disappointing

Recommender systems for music and books haven't lived up to initial expectations. Netflix and Amazon give me pretty decent recommendations, but among other things they get confused between things I buy for myself and things I buy for other people DeLong excerpts Slee to suggest a host of other problems -- mostly fraud related.

There's so much money riding on recommender systems even clumsy fraud is common, so it's rather likely that subtle fraud is also common. It's the same problem Google has had, since Google's original search approach was a form of recommender system.

Useful recommender systems may first require a good reputation and identity management infrastructure, or be based on data points that cost money to create. Not coincidentally, when I'm researching Amazon the first thing I look at is the sales ranks for the product domain I'm interested in. Then, for each product that's selling well, I look at the negative reviews first. I don't pay that much attention to the star rankings or the positive reviews.

A mouse model for calcineurin-type Schizophrenia: exciting news indeed

This is terribly interesting news on several fronts. Mouse models of a variant of autism emerged about 1-2 years ago, and they've radically accelerated our understanding of that mind/brain disorder. We can reasonably expect a similar revolution from this discovery ...

MIT research may lead to better schizophrenia drugs - MIT News Office

CAMBRIDGE, Mass.--MIT researchers have created a schizophrenic mouse that pinpoints a gene variation predisposing people to schizophrenia...

..Nobel laureate Susumu Tonegawa, director of the Picower Center for Learning and Memory at MIT and a Howard Hughes Medical Institute investigator, found that genetically engineered mice lacking the brain protein calcineurin exhibit a number of behavioral abnormalities shared by schizophrenic patients.

In a related study with researchers at Rockefeller University in New York, MIT scientists show that variation in a human calcineurin gene also is associated with schizophrenia. Calcineurin--part of a biochemical pathway in the brain linked to receptors for two brain chemicals, NMDA and dopamine--plays a significant role in the central nervous system.

This is the first study that uses animals who demonstrate an array of symptoms observed in schizophrenic patients to identify specific genes that predispose people to the disease...

... Tonegawa creates tools to explore the genetic underpinnings of the molecular mechanism for memory. Genetically engineered mice who are missing the brain enzyme calcineurin were previously shown to have an impairment in short-term, day-to-day memory formation, known as working memory. This kind of memory also is impaired in schizophrenia patients.

Further testing of these mice by Picower Center research scientist Tsuyoshi Miyakawa revealed that they also have attention deficits, aberrant social behavior and several other abnormalities characteristic of schizophrenia.

Picower Center research scientist David Gerber then collaborated with Rockefeller's Maria Karayiorgou to examine calcineurin genes in DNA samples from schizophrenic patients and their immediate relatives. The researchers found an association between a particular calcineurin gene and schizophrenia.

"This is an intriguing series of findings," Tonegawa said. "The combination of evidence from the genetically altered mice, together with the human gene studies, create a strong argument to link calcineurin with schizophrenia."...

... Alterations in multiple genes are believed to predispose people to schizophrenia. Tonegawa suspects that many of these genes may turn out to be components of the calcineurin pathway or to directly interact with the calcineurin pathway.

"Once we better understand exactly which genes are involved, we will know how proteins are affected, and we can set up a test to screen large numbers of compounds to identify ones that have desired effects on the activity of these proteins," Tonegawa said. "This can potentially lead to the discovery of new kinds of drugs for psychiatric conditions such as schizophrenia."

In addition to Gerber, Miyakawa, Karayiorgou and Tonegawa, co-authors include Joseph A. Gogos of Columbia University, and Diana Hall and Sandra Demars of Rockefeller University. Authors on the mouse study include research specialist Lorene M. Leiter and Hongkui Zeng of MIT, and Raul R. Gainetdinov, Tatyana D. Sotnikova and Marc G. Caron of Duke University...

If this actually works out, Tonegawa will earn another Nobel.

Age of Parody: Steven Colbert* and the Fake Steve Jobs

We've all read that the young-uns get all their news from watching Steven Colbert, a "fake" newscaster. We also know that Colbert has the highest quality news coverage anywhere, so the young-uns aren't so dim after all.

Which brings one to the media analysis of FSJ -- aka Fake Steve Jobs. One reason this geek parody site is widely read by Mac fans is that the commentary is often quite insightful, and, of course, it's well written, reasonably funny, and generally entertaining.

I think the Soviet Union was famous for high quality parody. I hope our age of Parody has no common roots.

*PS. Since I'm kind of removed from popular culture, I first typed "Stephen Coulter". Doesn't work. In the old days I've have been stuck. Now I googled on "you tube" "steven" and "parody". First hit.

Variations on the publicly traded company: two class ownership

James Fallows tells us what the New York Times and Google have in common: two tier corporate ownership (class A and B shares). I've passed through several variations of the modern corporation, and like most veterans I know the limits of both the private and public company. Private companies are capital limited and, eventually, channel limited. Public companies survive by brute force, but have all the grace and maneuverability of a steamroller.

Lately we've seen "private equity" variations, which seem to be largely a variation on the leveraged buyouts of the 80s. I hadn't recognized, however, that Google had implemented a private/public model previously known primarily in the news industry.

Fascinating.

I love blogs.

Caffeine and apoptosis

This is not necessarily good ...

Coffee and plenty of exercise could cut risk of skin cancer | Science | The Guardian

...A combination of coffee drinking and regular exercise may help to lower the risk of developing skin cancer, according to scientists in the US.

The two are thought to work together to kill off precancerous cells whose DNA has been damaged by ultraviolet-B radiation from the sun....

... Previous studies have suggested that exercise and coffee may each play a small role in protecting against skin cancer, but the latest research shows for the first time that when combined, the two may offer far more protection.

Scientists led by Allan Conney at Rutgers University, New Jersey, examined the effect of ultraviolet light on mice bred to be hairless, and so particularly vulnerable to the effects of sunlight.

Four groups of mice were exposed to UV-B radiation, but were given different diets and exercise regimes. One group drank caffeinated water, giving them a caffeine intake equivalent to one to two cups of coffee a day. A second group was fed pure water but allowed to exercise on a running wheel. The third group was given caffeine and access to a running wheel, while the fourth did no exercise and had no caffeine.

The scientists later took samples and checked for signs of UV-induced genetic damage. They also looked for evidence of a natural survival mechanism called apoptosis, in which damaged and potentially cancerous cells are forced to commit suicide before they can form tumours.

The tests showed that caffeine alone led to a 95% increase in programmed cell death and there was a 120% increase from exercise alone. But when combined, exercise and caffeine led to a four-fold increase in cell death, suggesting the body was able to rid itself of pre-cancerous cells much more effectively....

A 400% increase in programmed cell death?! Omigod, that's a lot. It turns out there's a burgeoning literature on caffeine and apoptosis.

Yech. I have both an affection (heck, addiction) to caffeine and a family history of skin cancer, so one might think I'd find this lighthearted good news. Alas, biology doesn't work that way. If this effect occurs in humans we're looking at a significant impact of caffeine on the fundamental behavior of cells. It would be surprising if that effect were always benign. Apoptosis of pre-melanoma is fine, but apoptosis of dopaminergic neurons ... maybe not so fine.

Fallows on the media: a 1996 article is even better today

Fallows, writing in his personal blog, called attention to an article he wrote in 1996 - before Bush, before 9/11, before the crash -- at the very height of our Glory Days: Why Americans Hate the Media. In brief, Fallows was merciless. Americans dislike the media, he concluded in 1996 because ... they had performed miserably. Everything he wrote then is true today. There are some great exceptions (some print journalists turned bloggers, Fallows himself, some astounding science writers) but there are lots of disasters. The punditry is an almost complete mess, and the beltway media is as bad today as it was 11 years ago.

Today Fallows writes "several times I have considered revisiting the whole what's-wrong-with-the-press question and have instead plugged on with other topics -- Iraq policy, China -- for reasons that boil down to: what's the point? The problems with the media are the same as I tried to describe 11 years ago -- just worse, and with new technology. But there's always tomorrow..."

The media didn't reach the basement by itself of course. Advertisers and bottom-line editors and owners drove many there, though some made it through egomania alone. Americans, above all, spent the money that justified those decisions. In consumer action as in American politics, the blame ultimately falls on the American citizen. Of course since newspapers are hemorrhaging money, maybe Americans have reformed a bit ...

NSA 2004: the loons were right, of course

In 2004 cynical geeks were convinced that Total Information Awareness, Poindexter's program of data mining, was continuing under an assumed name. Mainstream journalists classified this as "lunatic fringe". Now, of course, the "loons" have gone mainstream.

Which brings us to Gonzales. Why hasn't he gone? Why the intense focus on GOP election rigging strategies and telecom monitoring when there are so many other GOP/Cheney/Bush crimes to investigate? It's not unreasonable to assume that there's more going on that meets the eye. Something Gonzales has to cover up, something that will come to light too soon if he's gone ...

Data Mining Figured In Dispute Over NSA
By Dan Eggen and Joby Warrick
Washington Post Staff Writers
Sunday, July 29, 2007; A04

A fierce dispute within the Bush administration in early 2004 over a National Security Agency warrantless surveillance program was related to concerns about the NSA's searches of huge computer databases, the New York Times reported today.

The agency's data mining was also linked to a dramatic chain of events in March 2004, including threats of resignation from senior Justice Department officials and an unusual nighttime visit by White House aides to the hospital bedside of then-Attorney General John D. Ashcroft, the Times reported, citing current and former officials briefed on the program.

Attorney General Alberto R. Gonzales, one of the aides who went to the hospital, was questioned closely about that episode during a contentious Senate hearing on Tuesday. Gonzales characterized the internal debate as centering on "other intelligence activities" than the NSA's warrantless surveillance program, whose existence President Bush confirmed in December 2005.

FBI Director Robert S. Mueller III contradicted Gonzales, his boss, two days later, testifying before the House Judiciary Committee that the disagreement involved "an NSA program that has been much discussed."

Although the NSA's data mining efforts have been reported previously, neither Bush nor his aides have publicly confirmed that, in connection with the surveillance program, the agency had combed through phone and e-mail records in search of suspicious activity.

Nor have officials publicly discussed what prompted the legal dispute between the White House and the Justice Department.

The report of a data mining component to the dispute suggests that Gonzales's testimony could be correct. A group of Senate Democrats, including two who have been privy to classified briefings about the NSA program, called last week for a special prosecutor to consider perjury charges against Gonzales.

The report also provides further evidence that the NSA surveillance operation was far more extensive than has been acknowledged by the Bush administration, which has consistently sought to describe the program in narrow terms and to emphasize that the effort was legal.

The White House, the Justice Department and the Office of the Director of National Intelligence declined to comment last night. Calls placed to the NSA, which collected and analyzed the data, were not returned.

The warrantless surveillance program, which was authorized by presidential order after the Sept. 11, 2001, attacks, was first revealed publicly by the Times in December 2005. Bush confirmed aspects of the program at that time, defining it as monitoring communications between the United States and overseas in which one party was suspected of ties to al-Qaeda.

The Washington Post reported in February 2006 that the NSA targets were identified through data mining efforts and that thousands of Americans had been monitored. USA Today later reported that the government had the help of telecommunications companies in collecting millions of phone records.

The practice of sifting through mountains of privately collected data on phone calls and Internet communications raises legal issues. Although the contents of calls and e-mails are protected, courts have ruled that "metadata" -- basic records of calls and e-mails kept by phone companies -- are not...

I'll bet it wasn't merely international phone data. I suspect if they got a "hit" they ran through every database they could get their hands on -- and that they're false positive rates were significant. The media needs to read up on the original TIA plans, and assume that they were all implemented under different names.

Stories sunk without a trace: Feb 2004 bioweapon scare

I was searching my blog for a post and came across this old story ...

Gordon's Notes: WaPo: Chemical and bioweapon attacks on planes (2/1/2004)

... Intelligence indicating that al Qaeda terrorists are seeking to release a chemical or biological agent aboard an airliner, or transport a radiological device in cargo, was one of the factors that prompted the cancellation of six international flights scheduled for today and tomorrow...

I've long wished newspapers would routinely review stories written 2-5 years ago and tell us how it turned out...

Paying doctors: Room for huge cuts

The New York Times points out that physician reimbursement is one of the reasons that Amerian healthcare is more costly than european (or Canadian, Israeli, etc) health care.

I think they're mostly correct, though the discussion is necessarily simplified. They so point out that the discrepancy is far larger for specialists than primary care, but I think that when one adjusts for differing social costs and workload that primary care physicians in Europe and the US are similarly compensated. The gulf is entirely in the specialties, and particularly the procedural subspecialties (gastroenterology, radiology, etc.). The vastly greater compensation for specialty care in the US has shifted the physician workforce to being largely specialty based, amplifying the total cost to American healthcare. Of course lawyers in the US are probably paid several times as much as European lawyers, but that doesn't change the conclusion.

There's no way to figure out what one should pay physicians because there's no true market in healthcare; the best guide we have is people who pay cash and they tend to be atypically impoverished. It's likely, however, that one could slash US orthopedic reimbursement by 30-40% [1] and still get excellent orthopedic care once the dust settled (though the dust would take years to settle).

There's one point in the article where things went quite wrong however (emphases mine)...

...Dr. Goldman of RAND said that doctors are misleading themselves if they think the current system serves patients’ needs.

For example, if a diabetic patient visits a doctor, he said, “the doctor is paid to check his feet, they’re paid to check his eyes; they’re not paid to make sure he goes out and exercises and really, that may be the most important thing.”

“The whole health-care system is set up to pay for services that are rendered,” he said, “when the patient, and society, is interested in health.”...

I assume Dr. Goldman is not terminally naive, so he must be dissembling. I'm sure his job requires some creative dishonesty, but I wish he wouldn't. There's no real evidence that Americans, or anyone really, is happy shifting healthcare resources from treatment to prevention. Humans simply are not that rational. It's not even close. If Americans were spending their own dime, meaning we had a true healthcare market, I bet we'd spend even less on preventive care than we spend now.

Other than Dr. Goldman's misdirection I agree with the thrust of the article. We could slash healthcare administrative overhead by 70% chop 30% or more off drug costs, and cut procedural/subspecialty reimbursement by 30-40% [1] and still end up with better quality healthcare, by any measure, than we have now. We'd also, with no increase in primary care compensation, end up with a better supply of primary care physicians. The subspecialty offices, however, would have thinner carpets and rattier furniture.

[1] I originally wrote 70%, but that was the result of too quick math. The contrary argument is that, barring continued importation of non-citizen physicians, the US healthcare system is competing for talent with the corporate sector. If US corporate compensation is much higher than European compensation, then that might necessitate higher proceduralist compensation. I don't think that's true however.

The erratic non-progress of the personal information manager

The Personal Information Manager (PIM) has had a difficult 24 years, since Borland's "Sidekick" more or less launched the genre. We're coming up on the 25th anniversary of Sidekick, and I think it's fair to say a geek of 1983 would be shocked by how little progress we've made. The iPhone has no tasks. What more can I say?

The PIM has been a longstanding interest of mine. At various times in my life I've had a pre-web listserv dedicated to the personal information manager, a now-defunct blog dedicated to the Palm and its alternatives and an abandoned web page or two on related topics. I thought of the PIM as I cleaned out some old files, with clippings about (some of these were groupware too) the golden years from 1983-1994. It was in 1994 that the reign of Sauron began.

• Arabesque's Ecco (much mourned)
• Lotus Organizer (ok, so it wasn't too fancy)
• Act for Windows (still around I think)
• CrossTies (object oriented model)
  
• MeetingMaker (cross-platform)
  
• Lotus Agenda (a classic)
• Attain Corp's "In Control": outliner/calendar combination
• GrandView: calendar/task/outliner/spreadsheet
• Ascend
• Commence
• Arrange 2.0 (Mac - bit of an object oriented database I think)
• InfoDepot
• FullContact
• First Things First (outliner, calendar)
• NewtonOS: a PIM that was an Operating System

Outlook came later, and the combination of Outlook/Exchange crushed the genre on the PC -- and finished off Palm as well (though by then Palm's owners had shot both feet off). Reinvention continues on the Mac, with a vast array of small vendor products that have various combinations of features of all of the 1983-1994 PIMs. On the web we have Backpack and a range of Web 2.0 apps, most of which will vanish in the next few years. Along with all your data.

Fifteen years ago I thought the salvation of the PIM would be application embedding, what we then thought of as OpenDoc. We'd have applications for projects, tasks, calendars and the like, and they'd all seamlessly interoperate with one another. That was a bit before I got into the interoperability business myself, building applications that tried to talk to one another about lab studies, diseases, genetic history, procedure history, consultations, etc. In that world software is relatively easy, the hard part is "meaning" -- having a common, or at least reasonably interoperable way to represent knowledge about things between systems. It starts with being able to generate a common data model (even if it's only used for communication), but it gets much harder than that when you need to store and create bits of data. That's when you get into really painful things, like formally maintained ontologies. (Engineers love emergent ontologies, which is more like the way our minds work, but interoperability between minds requires more CPU power than we have on the desktop.)

I think the 25 years of non-progress in PIMs springs from the same roots as 25 years of very slow progress in interoperable clinical systems (whether you want them to actually be able to share data is another matter - one of which I've blogged before). The domain of the PIM is far simpler semantically than that of the clinical record, but there's far less pressure for the grindingly hard work of common semantics and mutually agreed upon data models. I think we'll be at roughly the same point in 2033 that were were in 1983 ...