Managing complexity: the lifelong data repository

Faughnan's Tech: Yahoo! Desktop (X1) is the new champion

In my tech notes blog I posted a review of X1. I've been using it for a while. It needs work, it's not as polished in some ways as Lookout, but it's pretty good. We have a lot further to go, however.

Lookout works well because Outlook content has lots of metadata and context. Email has dates, links to people, descriptive text surrounding attachments, etc. Email tends by nature to provide focal chunks of context. In contrast Google works well on the web because web pages have links that can be weighted, a robust form of metadata. Heck, web pages even have descriptive titles.

By comparison today's desktop file store is a barren desert. There's very little to go on to help search tools work. The most useful tool is probably the folder name -- pretty meager fare.

This wasn't such a big deal when we managed a few MBs of data. But what of the dataset that grows over a decade? That repository may be vast. Unfortunately, due to lack of supporting metadata, it's easier to find documents on the web than it is to find them on the desktop.

The good news is there are no lack of ideas to make things better. Heck, even as one uses today's software to search for items, one can be layering metadata atop the file system. If I do a search and open a file, then it's clearly more valuable and might earn a higher value score. The list of ways to assign value is very long; it will be fun to see how they get instantiated. Some of those ideas are 50 years old (Vannevar Bush described most of them in 1945 or so), I doubt any of them are truly new -- but the implementations will bring surprises.

PS. This is an old interest of mine.

Update 2/21/05: I've taken to appending the string [_s#] where # is 1-5 to the end of filenames to provide some crude metadata value scores. Full text search programs that index file names can then be filtered by the suffix value.

What Fates Impose: The 2004 annual lecture to the British Academy

British Academy - King: "What Fates Impose"

Mervyn King is the Governor of the Bank of England. He presented the annual lecture of the British Academy at the end of 2004. The British Academy is what is technically known as a daunting audience. Governor King, with, I assume, help from his staff, rose to the challenge. His lecture on risk and probability is a classic, with footnotes. He chooses as his working example pension reform -- a topic of some interest in America.

Everyone should read this, particularly decision makers. It is, alas, wasted on our current government.

... Whether in policies for health or transport, matters monetary or meteorological, in times of war and peace, decisions should reflect a balance of risks. Yet policy debates continue to be permeated by the ‘illusion of certainty’.

The reluctance to give adequate prominence to risks may reflect the fact that many of us feel uncomfortable with formal statements of probabilities. Probability theory is relatively recent in our intellectual history, dating back to a flowering of ideas around 1660 from Pascal, Leibniz, Huygens and others. Despite advances since then, statistical thinking remains prone to confusion and is often avoided. Television weather forecasts in Britain rarely employ the language of probabilities used by the meteorologists themselves. Professor Gigerenzer of the Max Planck Institute in Berlin has demonstrated in a series of studies how poorly doctors, lawyers, and other professionals understand probabilities. And despite Seneca’s maxim that ‘luck never made a man wise’, airport bookshops stock titles on how to become rich by successful investors and entrepreneurs who are confident that their success is the result of outstanding business acumen rather than good fortune.

Many of these misunderstandings stem from a failure to grasp basic statistical concepts. Juries are not informed that, in a country of our size, multiple cot deaths are likely to occur several times a year, that several people will have DNA that matches the incriminating sample, and that in themselves these coincidences are not evidence of guilt. Bookshops do not stock such titles as ‘I would have been a billionaire if only Lady Luck had been faithful’...

...I want to illustrate those two propositions by considering as an example public policy about pensions -- an issue, you might think, of particular interest to many of us in the Academy. When the Pensions Commission reported in October, it highlighted the financing gap in our present system. But we must not lose sight of the equally important question of what are the risks incurred in pension provision and how should they be shared among us? It is not my intention to make any recommendations. That is for the Pensions Commission next year, and the Government in its turn. But I do want to show that risk is at the heart of the issue...

...As Bertrand Russell said, ‘The whole problem of the world is that
fools and fanatics are always so certain of themselves, but wiser people so full of doubts’...

If only, if only .... Ohio. Florida.

Another big thing the media missed

The New York Times > National > Bush Nominee Wants States to Get Medicaid Flexibility

An astounding statement from the former secretary of health and human services.

Mr. Leavitt said he did not believe that the secretary should have the power to negotiate with drug manufacturers to secure lower prices for Medicare beneficiaries.

The current secretary of health and human services, Tommy G. Thompson, said last month that he wished Congress had given him that power. But Mr. Leavitt said that a healthy, competitive market was a better way to hold down drug prices.

I'm not surprised Mr. Leavitt does not want a federal formulary. A great deal of money was spent to prevent such a thing.

I'm very surprised Tommy Thompson did want one, or at least that he wanted the power to negotiate for best prices on a national level. There was not a whisper of this while he was in power. Was he silent out of fear or loyalty to Bush? Why didn't the media realize how big this alleged statement of his was? We're only talking about hundreds of billions of dollars.

What's wrong with American journalism?

Google oddities: my "top ranked" page

Home Video Editing

I'm shopping for a digital video camera. So I do a Google search on "pass-through", "video" and "quality".

A page I started in 2000, advised by my brother Brian, came out #1. Ok, so it was a good start and it did attract some email, but there's no way it's a terribly useful web page. I aborted the project in mid-2000 because I decided the software/hardware solutions weren't there yet - especially on the PC platform. Instead I went into digital photography in a big way.

Now, with iLife 2005 and a G5 iMac (pending) I think I can do what I want in a semi-tolerable way. I have to move anyway, my analog tapes are getting old. (Actually I really want the 2006 G6 dual-core dual CPU Mac with 4GB of RAM, but I can probably start now.)

So maybe I'll update the page ... eventually.

At last, the obvious begins to be discussed. It's the technology stupid.

INTEL DUMP - Be afraid... but be prepared

From an Atlantic article by a counterterrorism guru

... This 'war' will never be over, unlike the Civil War, the Vietnam War, or even the current war in Iraq. There will always be a threat that someone will blow up an airplane or a building or a container ship. Technology has changed the balance of power; it is easier for even a handful of people to threaten a community than it is for the community to defend itself. But while we have to live in danger, we don't have to live in fear...

I wrote this in October of 2001:

Over the past century technology has increased destructive power more than it has increased defensive capabilities. Technology, including communication networks and knowledge distribution, has brought to individuals and small groups (micro-powers) the capabilities once limited to nation states; the cost of acquiring and deploying nuclear and particularly biological weapons has decreased substantially. It has increased the harm potential of individuals and small groups. I sometimes call this the AIM problem, a pseudo-acronym for Affordable, Anonymous Instruments of Mass Murder. Our technologies are lowering the cost of the havoc, and the new weapons can be deployed anonymously. Anonymity means invulnerability. We cannot be anonymous, so we are are at an enormous disadvantage -- eventually contending against an invulnerable opponent with irresistible weapons.

John Kerry's petition: replace Rumsfeld

:: Donald Rumsfeld Must Resign ::

Of course I'll sign as a gesture of support to John Kerry. I feel I owe him for fighting hard against terrible odds (a special thanks to for Bruce Springsteen and the E Street band). I'm sure Bush will pay a lot of attention ...

King George

BBC NEWS | Americas | Kings in the White House

Stephen R Graubard, a historian, has a book out claiming that George Bush has the unchecked power of a king, and behaves like aone. I'm not sure that's so different from the CEO-president role Bush's admirers claim; even though CEOs of public companies in theory report to a board, in practice the board is often subverted.

It will be interesting to watch the succession. Will King George continue to reign even behind the scenes? (Assuming he does officially retire 4 years from now ...)

How you know you're middle-aged

jfaughnan's Yahoo! Profile

This is amusing. I have a profile on Yahoo that I set up years ago when I was running some Yahoo groups and needed a profile for the group. I never gave it much thought. Of course it persists, and it has my age as 42 (true way back then). Serendipitously I came across it today. When I view it I see the inevitable ads for personals (I guess that's why Yahoo does these directories -- duhhh). Only, unlike the usual personals ads one sees on the net, the models featured are clearly over 35. Maybe it's chance, but since Yahoo knows my age, there's no reason they couldn't target the ads a bit.

How sweet. I think I'll adjust my age to 85 and see what happens.

Update: At age 85 the "personals" go away. Instead I get some very generic ads that look like they're pretty low rent.

Seymour Hersh: The Next Wars

The New Yorker: Fact

Another Hersh article worth reading. If he didn't have such a good track record, and if Bush were not brutally wilfull, this would not be believable. As it is, I trust Hersh more than Bush. Where, though, will we get all the soldiers?

AuthenticGOP.com

AuthenticGOP.com

My favorite "A person of tolerance and diversity keyed my car."

This is indeed a well titled site. "Authentic GOP" indeed.

The Darwinian Society

Ownership Society

..An ownership society values responsibility, liberty, and property. Individuals are empowered by freeing them from dependence on government handouts and making them owners instead, in control of their own lives and destinies. In the ownership society, patients control their own health care, parents control their own children's education, and workers control their retirement savings.

---

Darwin was an outstanding scientist and an exceptional human being. Alas, his name is not his own any more. So, with apologies to the great scientist and humanist, let us declare that the "ownership society" is social Darwinism in the Spenglerian tradition.

In social Darwinism "excellence" is rewarded by wealth (theoretically by progeny, but that part didn't work out). Weakness is punished -- by misery and death.

The Ownership Society is about rewarding Strength and Excellence. And, conversely, by elimination of the weak.

That's the agenda underlying "social security reform". Those who are strong will do well, those who misjudge, who are frail, who are weak, will perish. Unless, of course, they have wealthy families who will save them.

Only the poor and the weak will truly perish.

Sound familiar?

I don't like the idea of living in that world. I don't want my children to grow up in that world. The saddest irony of all is that Bush was elected, in part, by those who were too "weak" to see through his agenda. They and their children will pay the price -- unless, by some miracle, we turn this back.

NYT Magazine has an in depth review of social security

The New York Times > Magazine > A Question of Numbers

... Overall, the [Bush] plan is gentler toward lower-income seniors than wealthier ones, but all seniors would be poorer than under present law. In other words, absent a sustained roaring bull market, the private accounts would not fully make up for the benefit cuts. According to the C.B.O.'s analysis, which, like all projections of this sort should be regarded as a best guess, a low-income retiree in 2035 would receive annual benefits (including the annuity from his private account) of $9,100, down from the $9,500 forecast under the present program. A median retiree would be cut severely, from $17,700 to $13,600. On the plus side, budget deficits would be lower in the future. But, because of the lengthy transition, that ''future'' is exceedingly remote -some 50 years down the road. In the interim, deficits would rise by up to 1.5 percent of the country's G.D.P....

An in depth review. How many people know that since 1997 the insolvency date has moved back 13 years -- because lifespans have not extended as much as expected. The obesity epidemic perhaps?

This detailed article is robust evidence that this is a battle about ideology, not demographics and not economics. Social security is fundamentally "socialist" (progressive) -- those that have give to those that have not. It's easy to see why extremists, like Bush, want to eliminate it.

Among all the details and surprises in the article, one impression stands above all. The politicians who did this, and the people who worked for and with them, tower over our current leadership. It's not that America was any better; the historical context is very familiar. Bile spouting morons on the radio, whacko extremists proposing radical revisions, the dark specter of fascism off to one side -- heck, the American people were then as they are now. What puzzles me is why the politicians, and the bureaucrats who built social security, were so superior to what we now endure.

PS. Social security is mandated by law to do 75 year predictions. This is pathetically funny. We may not even be human in 75 years.

My home town paper shows some spine ... callling a lie a lie

Editorial: Social Security/Blacks get more, not less, from it

Of all the lies -- let's call them by their right name -- that the Bush administration is spreading about Social Security, none is as vile as the canard Bush repeated last Tuesday, when he said, 'African-American males die sooner than other males do, which means the [Social Security] system is inherently unfair to a certain group of people. And that needs to be fixed.' That is an entirely phony assertion; it has been debunked by the Social Security Administration, by the Government Accountability Office and by other experts. Bush and those around him know that. For them to repeat what they know to be a blatant lie is despicable fear-mongering.

---

Wow. The Strib is usually as dull as dishwater. Someone must have put something in the water coolers. Good for them.

A just sentence for Graner -- but an unjust world

Graner Gets 10 Years for Abuse at Abu Ghraib (washingtonpost.com)

On the night shift at One-Alpha, Graner said, the Army assigned two low-ranking reservists to guard 80 to 100 prisoners, ranging from common criminals to veteran terrorists. He showed a picture of the guards' cellblock 'office' -- a closet-size space surrounded by sandbags to protect against the guns and grenades that he said were regularly smuggled to the prisoners.

Graner said the guards were told to 'terrorize' the inmates to make it easier for CIA agents and military intelligence officers to question them.

'They would say . . . give this prisoner 30 seconds to eat,' Graner recalled. 'It's pitch black in your cell. I shine a light in your eyes to blind you. I haul you out, naked, and I hand you the [packed lunch] and the whole time you're trying to eat I'm screaming at you. Then time's up. We gave you the opportunity to eat. You just didn't eat.' ...

...Graner named a series of Army officers, ranking from lieutenant to full colonel, who gave orders, he said, to mistreat prisoners -- particularly those described as "intelligence holds" who were believed to have information about the Iraqi insurgency that grew up after the fall of Baghdad. Those he named included Col. Thomas M. Pappas, commander of the 205th Military Intelligence Brigade in charge of the prison; Lt. Col. Steven Jordan, the senior Military Intelligence officer; Capt. Donald J. Reese, commander of the 372nd Military Police Company; Capt. Christopher Brinson, platoon leader; and 1st Lt. Lewis Raeder, platoon leader in the military police command.

Graner deserves his 10 years. So, probably, do the people he's named. Problem is, so does Rumsfeld. And maybe, so does Bush.

Confusion about medical errors, sigh

A Health Care Cost Shift (washingtonpost.com)

'So we're spending a third more than any advanced industrial country, but half of that money is wasted and people are hurting,' Darling said. 'The employer has to pay more just to undo the damage done' by medical mistakes.

---

There's no evidence that the US has a higher medical error rate than other wealthy nations. In fact, based on my experience in Canada, our error rates are probably about the same or a bit lower than elsewhere. (It's not that US physicians are more virtuous or better trained, it's that US consumers are more aggressive and have more lawyers.)

This is not a chance bit of confusion. It's a deliberate smokescreen by the insurance companies and payors. Of course journalists at the Washington Post are readily bamboozled by this kind of thing.

We know from studies in the 80s and 90s why our costs are higher than other nations. We use more subspecialists, we pay far more in overhead to payors and insurance companies, we're more aggressive in neonatal and end-of-life care, we pay our physicians and nurses more than other nations. (Note we pay our CEOs far, far more than other nations). It's no great mystery.

Reducing medical errors might reduce costs -- it depends how you define "error". In the popular sense of a "mistake that causes harm" reducing those might increase costs! Most the mortality from medical errors comes near the end of life, to vulnerable people who can't survive commonplace mistakes. Since most of those people would die even with perfect care, medical errors are reducing costs. (This is a common mistake about medical errors btw. The years-of-life-lost to medical error are far less than years-of-life-lost to other causes.)

If by "error" one means duplicate testing, "unnecessary care" (MRI for sore shoulder -- one man's luxury is another's necessity), etc. then indeed reducing these "medical errors" would reduce costs. That's in part what "managed care" is about. It's not what Leape et al were writing about in the 80s and 90s when they made their mark.

Of course there's no way any journalist will get this straight. They have been betrayed by their publishers/editors and are massively outgunned by the insurance/payor industry.