Thursday, April 10, 2008

Best explanation of the Monty Hall problem (Bayes)

John Tierney has been playing with explanations of the "Monty Hall" (Bayes theorem) problem for 17 years. That might be why he's provided the most succinct explanation I've come across ... (note: Monty knows where the car is, he can't open the door you picked, and he won't open the door for the car. That's important -- his actions provide new information. He's not picking randomly.)
Cognitive Dissonance in Monkeys - The Monty Hall Problem - New York Times

...Here’s how Monty’s deal works, in the math problem, anyway. (On the real show it was a bit messier.) He shows you three closed doors, with a car behind one and a goat behind each of the others. If you open the one with the car, you win it. You start by picking a door, but before it’s opened Monty will always open another door to reveal a goat. Then he’ll let you open either remaining door.

Suppose you start by picking Door 1, and Monty opens Door 3 to reveal a goat. Now what should you do? Stick with Door 1 or switch to Door 2?...

...You should switch doors.

... when you stick with Door 1, you’ll win only if your original choice was correct, which happens only 1 in 3 times on average. If you switch, you’ll win whenever your original choice was wrong, which happens 2 out of 3 times...
Probability problems are often asymmetric, they can be hard to solve in terms of the "correct choice", but easy to understand when considered when re-expressed in terms of the "wrong choice" (or vice-versa). That's what we see here.

Tierney's paragraph is a great example of expressing simple algebra in sentence form, but the key thing to recall is that Monty is adding new information because he doesn't choose randomly.

I'm fascinated by Bayesian probability. The mathematics is very simple, yet it can be very challenging to map correctly to the physical universe. On the other hand even a trivial understanding would greatly improve government and law enforcement. What a marvel!

Wednesday, April 09, 2008

The case of the curious silence: Counterfeit Heparin killed 62+

Counterfeit Heparin (alas, again fraud in China) killed 62 - Americans. I've not seen data on foreign deaths. The latest via ABC:

ABC News: FDA Puts Heparin Death Count at 62

... After reviewing adverse events back to January 2007, FDA said Tuesday it uncovered 103 reports of patients who died while taking heparin.

Of those deaths, 62 involved allergic reactions or hypotension, a type of dangerously low blood pressure. Those are the same side effects that caused Baxter's to pull all U.S. heparin injections from the market by February...

America's adverse event monitoring system is very unlikely to be complete. I would not be surprised if the true Heparin associated death rate were 2-3 times as high. Anaphylactic reactions are not universally fatal, so I suspect that for every death there might be ten people who were injured, some of who will not fully recover. Some will have been weakened at a critical point, and then died of other causes.

So let's make a wild guess, and estimate that 2,000 to 3,000 Americans were seriously injured or killed by a fraudulent version of an extensively used medication. Since up to 10% of our Heparin supply was affected by the fraud, that's not too shocking. In fact there's reason to suspect the practice started before January 2007, so the total could be higher.

But that's only one drug. And that's only the US. If we include all the wealthy nations of the earth (it's a cheap widely used drug but the most common uses are for expensive treatments) we can guess that 5,000 to 30,000 people have died or been badly injured as a result of fraudulent medications or fraudulent food practices over the past 2-3 years.

That's a fair total, so the curious thing in this affair is the public silence. There's a great deal of unmerited anxiety about immunizations, but very little about fraud and our food and medication supply.

So, why is America silent? Is this a variant of the social phenomena that leads to complacency about climate change?

I don't think so. Climate change complacency is relatively easy to understand -- for many Americans a warmer climate is seen as a net plus, and even some plausible experts feel our only hope is a technological breakthrough in either energy production or carbon sequestration.

This feels different. My best guess is a kind of learned helplessness, the result of 12 years of GOP destruction of government* and the obvious failure of Libertarian dreams of emergent market-driven auto-regulation.

If I'm right, matters will only improve if McCain loses the presidency and both the House and Senate stay Democrat. I don't see any other configuration that will allow the rebuilding of our government.

* 8 years of Bush, and during Clinton's last term the House and Senate were both GOP. It's much easier to destroy than to create or maintain, so control of either the presidency or the legislature is sufficient to destroy  government.

Aaronson's MIT lectures on theoretical computer science - open to all

Scott Aaronson teaches an MIT course on theoretical computer science (emphasis mine)

6.080 Great Ideas in Theoretical Computer Science

... a challenging introduction to some of the central ideas of theoretical computer science. It attempts to present a vision of "computer science beyond computers": that is, CS as a set of mathematical tools for understanding complex systems such as universes and minds. Beginning in antiquity -- with Euclid's algorithm and other ancient examples of computational thinking -- the course will progress rapidly through propositional logic, Turing m achines and computability, finite automata, Gödel's theorems, efficient algorithms and reducibility, NP-completeness, the P versus NP problem, decision trees and other concrete computational models, the power of randomness, cryptography and one-way functions, computational theories of learning, interactive proofs, and quantum computing and the physical limits of computation...

Prerequisites. This course is designed for undergraduates (both under- and upperclassmen) in computer science and related areas of science and engineering.... The only prerequisite is some facility with mathematical reasoning ... Programming experience is helpful but not essential; the course has no programming assignments.

Scott writes on his blog, Shtetl-optimized, where he's created blogthread for the course, including links to completed lectures:

A typical lecture handout is about 8 pages. I plan to print out each one and leave them by the spot where I keep all the material for five minutes of concentrated attention is practical.

Amidst the mixed news of everyday life in 2008, what a wonder it is that all of us can freely share in work like this. Thank you Scott, and thank you MIT.

Marx and Engels on the railway bubble of 1845

Brad DeLong has a great post for his econ class that includes, as an aside, a quote from Marx and Engels on a 19th century boom and bust ...

April 9 Lecture: Econ 101b: Arguments Against Lender-of-Last Resort Operations

... In the years of prosperity from 1843 to 1845, speculation was concentrated principally in railways, where it was based upon a real demand.... The extension of the English railway system... 1845... the number of bills presented for the formation of railway companies [i.e., IPOs] amounted to 1,035.... The heyday of this speculation was the summer and autumn of 1845. Stock prices rose continuously, and the speculators' profits soon sucked all social classes into the whirlpool. Dukes and earls competed with merchants and manufacturers for the lucrative honour of sitting on the boards of directors of the various companies; members of the House of Commons, the legal profession and the clergy were also represented in large numbers. Anyone who had saved a penny, anyone who had the least credit at his disposal, speculated in railway stocks...

I think something similar was going on in America, and the economic chaos is thought to have played an important role in the Civil War.

The entire quote is fascinating reading.

The railway bubble, though it burst, had very solid foundations. My Irish-Canadian grandparents and their extended family worked largely in the rail industry at the start of the 20th century -- fifty years after the great railway depression of 1848. So the historic analogy is not to the real estate bubble of 2002 - 2007, but to the Internet bubble of 1998 - 2000.

(Incidentally, DeLong is a solid neo-liberal capitalist, his reference to Marx belongs to the history of economic thought.)

Sunday, April 06, 2008

Gordon's Notes move failed

My planned migration of Gordon's Notes is on hold -- Google's Blogger failed with a typically unhelpful Blogger error message.

I'm putting the migration on hold until I hear back from Google -- so posts should continue to appear at jfaughnan.blogspot.com.

Gordon's Notes to move to notes.kateva.org: In theory, no changes required

I've been writing Gordon's Notes at the Blogger address of jfaughnan.blogspot.com since July 2003. The 4050 posts to date have a very small (but much appreciated) regular readership, but they turn up in a gratifying number of Google searches.

The writing relieves my personal frustrations, and it helps me think and learn. I enjoy it, and I intend to continue. The address, however, is about to change to notes.kateva.org, where it will join Gordon's Tech which has been tech.kateva.org for about a year. It will still be published through Blogger, kateva.org is a Google "custom domain".

There's an explanation of the name change at www.kateva.org - Welcome to Kateva.org

... I originally published under my own name, but the common business practice of searching the names of new acquaintances meant I was making a bit too much of first impression. I've since taken a few steps to move the blogs one or two steps from my "real name", including publishing under a pseudonym (John Gordon) and moving the blogs to this domain. I am not, however, too hard to find...

In theory Google will do a "301 redirect" and old feeds will continue to work. In practice, problems can occur. In particular Google doesn't redirect for some legacy Blogger (working) feeds, only for the current feeds.

So if you find this post in your Feed, and then no "it worked" post after it, you should be able to find Gordon's Notes at notes.kateva.org. The Kateva.org main page will also have news of the move.

Most blog readers will reset your "unread" count to 10 posts.

Wish me luck, I'll make the move later tonight April 6th.



Update 4/6/2008: Well, I tried the move and I got:

We're sorry, but we were unable to complete your request.
When reporting this error to Blogger Support or on the Blogger Help Group, please:
* Describe what you were doing when you got this error.
* Provide the following error code and additional information.
bX-czudlu
Additional information
blogID: 5587346
host: draft.blogger.com
uri: /blog-publishing.do
This information will help us to track down your specific problem and fix it! We apologize for the inconvenience.
Sigh. There's no real support for blogger, so I guess I'll have to see how this works out. I wonder if this blog will continue to work ...

Primary care access problems in Massachusetts -- and the strange reporting thereof

The New York Times has a longish front page article on access problems in Massachusetts that could have been replaced by three bullet points:
  1. Massachusetts' pseudo-universal coverage cut the uninsured in half, making another 340,000 eligible for non-emergent care.
  2. The average radiologist in Massachusetts makes $380,000 dollars.
  3. In the rural areas that are short of providers "...some physicians are earning as little as $70,000 after 20 years of practice...".
Gee, that's not so complicated is it?

Cut reimbursement to radiologists and other specialists by 30-40%, and increase reimbursement to rural family physicians by 60-80%, and I promise those access problems would melt away. I can even promise that overall quality of care would eventually improve across the board -- after an incredibly painful transition period.

This is roughly the income distribution that both Canada and the Mayo Clinic used to have, so it's known to work. Of course an income cut of that magnitude would put some specialists out of their homes, and a goodly number of senior people would simply retire. The transition would not be pretty, and perhaps not very fair. It would work though.

For me, the interesting thing about this story is not its unsurprising content, but its peculiar structure. The relevant information is oddly distributed, and few will read to to the meaningful paragraphs at the very end.

For example, here's the beginning:

In Massachusetts, Universal Coverage Strains Care - New York Times

...In pockets of the United States, rural and urban, a confluence of market and medical forces has been widening the gap between the supply of primary care physicians and the demand for their services. Modest pay, medical school debt, an aging population and the prevalence of chronic disease have each played a role.

Now in Massachusetts, in an unintended consequence of universal coverage, the imbalance is being exacerbated by the state’s new law requiring residents to have health insurance.

Since last year, when the landmark law took effect, about 340,000 of Massachusetts’ estimated 600,000 uninsured have gained coverage. Many are now searching for doctors and scheduling appointments for long-deferred care....

This is followed by filler, and then "page two":

... The situation may worsen as large numbers of general practitioners retire over the next decade. The incoming pool of doctors is predominantly female, and many are balancing child-rearing with part-time work. The supply is further stretched by the emergence of hospitalists — primary care physicians who practice solely in hospitals, where they can earn more and work regular hours. President Bush has proposed eliminating $48 million in federal support for primary care training programs.

Clinic administrators in western Massachusetts report extreme difficulty in recruiting primary care doctors. Dr. Timothy Soule-Regine, a co-owner of the North Quabbin practice, said it had taken at least two years and as long as five to recruit new physicians.

At the University of Massachusetts Medical School in Worcester, no more than 4 of the 28 internal medicine residents in each class are choosing primary care, down from half a decade ago, said Dr. Richard M. Forster, the program’s director. In Springfield, only one of 16 third-year residents at Baystate Medical Center, which trains physicians from Tufts University, plans to pursue primary care, said Jane Albert, a hospital spokeswoman.

The need to pay off medical school debt, which averages $120,000 at public schools and $160,000 at private schools, is cited as a major reason that graduates gravitate to higher-paying specialties and hospitalist jobs.

Then, at the very end, the two most important paragraphs ...

... Primary care doctors typically fall at the bottom of the medical income scale, with average salaries in the range of $160,000 to $175,000 (compared with $410,000 for orthopedic surgeons and $380,000 for radiologists). In rural Massachusetts, where reimbursement rates are relatively low, some physicians are earning as little as $70,000 after 20 years of practice...

Where do editors and journalists learn this obscure form of writing?

In any case, the problem is relative income of course -- it always is. Relative not only to medical specialists, but also to corporate executives, business owners, lawyers, accountants, etc.

Incidentally, I'm fine with Bush eliminating the $48 millions in subsidies for primary care programs. In 2007 42% of family practice residents came from US schools -- that's low enough to be a serious quality issue. We probably need to close half of the remaining primary care residencies, and losing the subsidy would ensure that. Of course the access problems would worsen, but subsidizing training is the wrong answer. Perhaps a sudden drop in a tight supply would concentrate minds a bit ...

Update 4/10/08: Coincidentally, today's NYT editorial also mentions the Mayo example, but fails to make the important connection to May's relatively small specialty/primary care income ratio.

AT&T vs. Sprint: one is better

I don't think there are any truly good cell phone companies [2], but three months after switching from Sprint to AT&T I can say that for us these two are not the same.

Sprint has better service and costs less.

Alas.

We switched because I wanted to buy an iPhone, which I still haven't bought [1]. I wasn't as wise as this writer:
ATPM 14.04 - Bloggable: Shallow Depth of Field

...Then, there’s writing about the iPhone. You see, I don’t have one, because I don’t have AT&T service. And until my friends who do aren’t constantly cursing dropped calls, I have no intention of switching carriers and buying an iPhone...
My friends weren't as forthcoming, but I can personally confirm that in the midwest AT&T drops calls routinely. My phone may show "four bars", but that's just a little AT&T joke. I go from four bars to no carrier in an eye-blink.

Cost? AT&T costs us about 70% more than Sprint for a similar set of services. I think this huge gap is partly an artifact of our usage patterns (two phones, moderate voice use, frequent calls to Canada, very little roaming on Sprint) but I suspect AT&T would cost most people about 20% more than Sprint.

Contracting trickery? Sprint has a nefarious history of covert contract extensions, but they've been getting better since being sued in Minnesota. AT&T has the rebate scam from Hell.

Sprint wins across the board on voice quality, cost, and contracting. I'm amazed.

The only thing AT&T has is the iPhone, but that's a very big thing. If you aren't going to buy an iPhone immediately, however, don't switch to AT&T.

As a current AT&T customer, I join the world's pleas for a great Google Android phone, and for a future iPhone free of AT&T.

--

[1] The timing of the switch was dictated by the impending death of my wife's beloved Samsung i500, I was waiting to see the shape of the SDK before committing to the iPhone. The SDK took so long to be revealed I ran into the pending iPhone 2.0 release!

[2] The contract lock-ins, switching costs, and the pricing costs all promote bad business practices.

Friday, April 04, 2008

Bush war crimes trial: exhibit A

Someday, many years from now, an aged George Bush may yet face a war crimes tribunal. If he does, the Yoo torture memo will be exhibit A:
There Were Orders to Follow - New York Times

...The March 14, 2003, memo was written by John C. Yoo, then a lawyer for the Justice Department. He earlier helped draft a memo that redefined torture to justify repugnant, clearly illegal acts against Al Qaeda and Taliban prisoners.

The purpose of the March 14 memo was equally insidious: to make sure that the policy makers who authorized those acts, or the subordinates who carried out the orders, were not convicted of any crime. The list of laws that Mr. Yoo’s memo sought to circumvent is long: federal laws against assault, maiming, interstate stalking, war crimes and torture; international laws against torture and cruel, inhuman or degrading treatment; and the Geneva Conventions.

Mr. Yoo, who, inexplicably, teaches law at the University of California, Berkeley, never directly argues that it is legal to chain prisoners to the ceiling for days, sexually abuse them or subject them to waterboarding — all things done by American jailers.

His primary argument, in which he reaches back to 19th-century legal opinions justifying the execution of Indians who rejected the reservation, is that the laws didn’t apply to Mr. Bush because he is commander in chief...

...When the abuses at Abu Ghraib became public, we were told these were the depraved actions of a few soldiers. The Yoo memo makes it chillingly apparent that senior officials authorized unspeakable acts and went to great lengths to shield themselves from prosecution.
The state of California needs to revoke John Yoo's right to practice law.

Those convicted of the Abu Grhaib war crimes should ask for a retrial, and submit the Yoo memo as evidence that they were following their leaders.

Tuesday, April 01, 2008

Over the hill

Older folk are slower at computer tasks:
Quoted : Good Morning Silicon Valley

...Does this mean that people in their 40s or 50s can’t do their jobs? Not at all. There are many other ways in which people get better with age...
We get more expensive, for one.

Oh wait, that's not a feature.

We need to move the retirement age to 40. Since many of us are in school and training to about age 30 that will give us ten years to work.

Today.

In ten years though, training will extend to age 33, and the age of incapacity will slip to 37.

I detect a problem.

How we perceive - a review of the art

Natalie Angier, while reporting on a science conference, provides a fascinating state of the art summary of visual perception.

We are again reminded that we live in our heads, and that the film that plays in there has only a loose connection to what's going on around us ...
Change Blindness - Natalie Angier - New York Times

.... Visual attentiveness is born of limited resources. “The basic problem is that far more information lands on your eyes than you can possibly analyze and still end up with a reasonable sized brain,” Dr. Wolfe said. Hence, the brain has evolved mechanisms for combating data overload, allowing large rivers of data to pass along optical and cortical corridors almost entirely unassimilated, and peeling off selected data for a close, careful view. In deciding what to focus on, the brain essentially shines a spotlight from place to place, a rapid, sweeping search that takes in maybe 30 or 40 objects per second, the survey accompanied by a multitude of body movements of which we are barely aware: the darting of the eyes, the constant tiny twists of the torso and neck. We scan and sweep and perfunctorily police, until something sticks out and brings our bouncing cones to a halt.

The mechanisms that succeed in seizing our sightline fall into two basic classes: bottom up and top down. Bottom-up attentiveness originates with the stimulus, with something in our visual field that is the optical equivalent of a shout: a wildly waving hand, a bright red object against a green field. Bottom-up stimuli seem to head straight for the brainstem and are almost impossible to ignore, said Nancy Kanwisher, a vision researcher at M.I.T., and thus they are popular in Internet ads.

Top-down attentiveness, by comparison, is a volitional act, the decision by the viewer that an item, even in the absence of flapping parts or strobe lights, is nonetheless a sight to behold. When you are looking for a specific object — say, your black suitcase on a moving baggage carousel occupied largely by black suitcases — you apply a top-down approach, the bouncing searchlights configured to specific parameters, like a smallish, scuffed black suitcase with one broken wheel. Volitional attentiveness is much trickier to study than is a simple response to a stimulus, yet scientists have made progress through improved brain-scanning technology and the ability to measure the firing patterns of specific neurons or the synchronized firing of clusters of brain cells.

Recent studies with both macaques and humans indicate that attentiveness crackles through the brain along vast, multifocal, transcortical loops, leaping to life in regions at the back of the brain, in the primary visual cortex that engages with the world, proceeding forward into frontal lobes where higher cognitive analysis occurs, and then doubling back to the primary visual centers. En route, the initial signal is amplified, italicized and annotated, and so persuasively that the boosted signal seems to emanate from the object itself. The enhancer effect explains why, if you’ve ever looked at a crowd photo and had somebody point out the face of, say, a young Franklin Roosevelt or George Clooney in the throng, the celebrity’s image will leap out at you thereafter as though lighted from behind.

Whether lured into attentiveness by a bottom-up or top-down mechanism, scientists said, the results of change blindness studies and other experiments strongly suggest that the visual system can focus on only one or very few objects at a time, and that anything lying outside a given moment’s cone of interest gets short shrift. The brain, it seems, is a master at filling gaps and making do, of compiling a cohesive portrait of reality based on a flickering view.

“Our spotlight of attention is grabbing objects at such a fast rate that introspectively it feels like you’re recognizing many things at once,” Dr. Wolfe said. “But the reality is that you are only accurately representing the state of one or a few objects at any given moment.” As for the rest of our visual experience, he said, it has been aptly called “a grand illusion.” Sit back, relax and enjoy the movie called You.

This evening my son was using a Flip Video camera to record the midst of a chaotic bout of present opening. I happened to play it back about an hour after the event, so I still had some memory of what I perceived. The video showed there was much more going on than I took in, including my oldest son reading a letter included in his brother's gift.

Truly, there are vast rivers of reality going by, we live on a few sips and a lot of extrapolation. It is a miracle, given the imaginary worlds in which we live, that we are able to communicate at all.

CBS 60 Minutes expose: In a just world, Bush would face a war crimes tribunal.

The CIA destroyed the videos they made of the interrogation of Murat Kurnaz.

When someone destroys documentation, the legal presumption is that the material was incriminating. In addition to that presumption, there are these documents:

On balance the evidence of Kurnaz's innocence is robust. In addition the destruction of evidence by the CIA and the correlation with other stories and verified evidence make him far more trustworthy than the US military.

So, I believe this CBS News 60 Minutes story of his five years of imprisonment, and years of torture: Ex-Terror Detainee Says U.S. Tortured Him, Tells 60 Minutes He Was Held Underwater, Shocked And Suspended From the Ceiling - CBS News

Note this part of the story, which took place in Kandahar Afghanistan ...

They used to beat me when my head is underwater. They beat me into my stomach and everything," he says.
"They were hitting you in the stomach while you're head was underwater so that you'd have to take a breath?" Pelley asks,
"Right. I had to drink. I had to…how you say it?" Kurnaz replies.
"Inhale. Inhale the water," Pelley says.
"I had to inhale the water. Right," Kurnaz says.
Kurnaz says the Americans used a device to shock him with electricity that made his body go numb. And he says he was hoisted up on chains suspended by his arms from the ceiling of an aircraft hangar for five days.
"Every five or six hours they came and pulled me back down. And the doctor came to watch if I can still survive to not. He looked into my eyes. He checked my heart. And when he said okay, then they pulled me back up," Kurnaz says.
"The point of the doctor's visit was not to treat you. It was to see if you could take another six hours hanging from the ceiling?" Pelley asks.
"Right," Kurnaz says.

Let us assume "the doctor" is an American MD. The number of physicians who served the US military at Kandahar Afghanistan cannot be that large. It should be possible to track him down, and bring him to justice. At the very least, his medical license can be revoked.

In turn, he will implicate others.

It is not inconceivable, though it is unlikely, that one day George Bush will be called before a war crimes tribunal. We might as well gather the evidence.

Project Virgle: See it while it's up

Virgin + Google = Virgle
Virgle: The Adventure of Many Lifetime

... starting in 2014, Virgin founder Richard Branson and Google co-founders Larry Page and Sergey Brin will be leading hundreds of users on one of the grandest adventures in human history: Project Virgle, the first permanent human colony on Mars...
It's a wonderful effort, with many links to explore. Of course it helps to know that Branson, Page and Brin really do want to go to Mars ...

For example, I wouldn't be surprised if the site of the colony were used one day.

Google Gmail and Calendar flailing

Ok, so the rest of the net seems to be working, but in the past few days I've either experienced or received direct reports of recurrent hour length outages of Gmail and Calendar.

I've not received any direct reports of problems with Google Apps or Search. I'm writing this on Blogger, so it also appears to be operational!

I have hard drives fail on my desktop machines every 6 months or so, so it's not like my desktop environments are rock solid. Drive failure are so painful that, even including ISP related network failures and personal LAN issues, my Google Gmail is still somewhat more reliable than my desktop email products. (Of course drive failure can knock out a machine used to access Gmail, but I have multiple access options.)

So, on balance, Gmail probably still squeaks by on average downtime compared to my desktop apps. This ain't good though. I wonder if it's going to merit a blog posting.

Sunday, March 30, 2008

Gwynne Dyer: four new essays

The historian and journalist Gwynne Dyer has four new essays online:
  • Pakistan: Bhutto's assassination voided any deals with Musharraff and thus ended his power. Dyer detects a faint ray of hope.
  • Tibet: China will crush the uprising just as they did in 1959, Olympics or no Olympics. China has been uncharacteristically restrained up to this point.
  • Iraq: Iraq's ethnic partitioning will aid recovery. Even Lebanon stopped fighting. Was it worth it? We'll never know.
  • Abkhazia: Dyer makes a peculiar argument that Russia actually likes International law and the UN, and therefore won't truly advance Abhkazian separation from Georgia. I have to admit it's a novel idea! That never would have occurred to me.