Prediction markets are irrational - the McCain penny stock example

Whatever the outcome tomorrow, we've extracted some useful knowledge from the process.

Prediction markets are no more rational than the stock market ...

McCain Is Now A Penny Stock

... John McCain's odds on Intrade drop to 9 cents on the dollar, a new low. (If you put a gun to our head and forced us to bet, we'd actually buy the contract at that level. We think Obama will win, but we certainly wouldn't give someone 10-to-1 odds on that.)...

I think the election is too close to call, but that might reflect my superstitious nature. There's no way, however, that Obama has a 90% certainty of winning.

So we shouldn't expect too much of prediction markets in the future.

Update 11/3/08: Ahh, but for a contrary perspective on those probabilities ...

GOP slime - it must be noted

We'll soon forget the details, so it's useful to have a reminder.

GOP and Democratic slime are not equivalent. The GOP plays in another league, a much nastier league.

If Obama wins, which I still consider unlikely, then the GOP will complain bitterly about "journalistic bias". They'll want to cow what's left of the press and network TV.

It will be good to remember these examples then.

The mission of the publicly held newspaper is ...

What is a newspaper for?

Well, if it's owned by a publicly traded company the newspaper exists to make money for shareholders. It makes this money through advertising revenue and selling papers. It is the fiduciary duty of managers to do whatever sells papers and doesn't break any important laws.

The Washington Post's Deborah Howell believes the best way to sell papers is to keep both "right" and "left" equally pleased, regardless of reality ...

The Post ... Glenn Greenwald - Salon.com

... The Washington Post's Ombudsman, Deborah Howell, today wrote a column claiming that one reason that The Post and other papers are losing money is because they are "too liberal"; have had "more favorable stories about Barack Obama than John McCain," and "conservatives are right that they often don't see their views reflected enough in the news pages." To mitigate newspapers' financial problems, Howell decrees: "the imbalance still needs to be corrected." She adds: "Neither the hard-core right nor left will ever be satisfied by Post coverage -- and that's as it should be."

What if the actual facts -- i.e., "reality" -- are consistent with the views of "the hard-core left" and contrary to the views of the "hard-core right"? What if, as has plainly been the case, the conservatives' views are wrong, false, inaccurate? What if the McCain campaign was failing and relying on pure falsehoods and sleazy attacks, and The Post's coverage simply reflected that reality? It doesn't matter. In order to sell more newspapers, according to Howell, The Post's news coverage must shape itself to the Right and ensure that "their views [are] reflected enough in the news pages" ...

... That corrupt formula is, of course, what is now meant by "journalistic balance" -- say what both sides believe and take no position about what is true -- and it is precisely that behavior which propped up this incomparably failed and deceitful presidency for so long...

Alas for the Post, there's little evidence that any print newspaper, regardless of mission, can thrive in today's world ...

The Media Equation - Mourning Old Media’s Decline - NYTimes.com

The news that Google settled two longstanding suits with book authors and publishers over its plans to digitize the world’s great libraries suggests that some level of détente could be reached between old media and new.

If true, it can’t come soon enough for the news business.

It’s been an especially rotten few days for people who type on deadline. On Tuesday, The Christian Science Monitor announced that, after a century, it would cease publishing a weekday paper. Time Inc., the Olympian home of Time magazine, Fortune, People and Sports Illustrated, announced that it was cutting 600 jobs and reorganizing its staff. And Gannett, the largest newspaper publisher in the country, compounded the grimness by announcing it was laying off 10 percent of its work force — up to 3,000 people...

... The day before, the Tribune Company had declared that it would reduce the newsroom of The Los Angeles Times by 75 more people, leaving it approximately half the size it was just seven years ago...

... two weeks ago, TV Guide, one of the famous brand names in magazines, was sold for one dollar, less than the price of a single copy.

The paradox of all these announcements is that newspapers and magazines do not have an audience problem — newspaper Web sites are a vital source of news, and growing — but they do have a consumer problem...

... For readers, the drastic diminishment of print raises an obvious question: if more people are reading newspapers and magazines, why should we care whether they are printed on paper?

The answer is that paper is not just how news is delivered; it is how it is paid for.

More than 90 percent of the newspaper industry’s revenue still derives from the print product, a legacy technology that attracts fewer consumers and advertisers every single day. A single newspaper ad might cost many thousands of dollars while an online ad might only bring in $20 for each 1,000 customers who see it...

Elsewhere we read that the NYT is running out of cash, and is having trouble borrowing. They need to sell the Boston Globe, but nobody wants to pay for it.

So how will this play out? My best guesses are:

1. We'll see a return of the privately held money-losing newspaper owned by billionaires with an agenda. The NYT may go this route. Depending on the billionaire(s) this may not be a bad thing. Of course if the billionaire is Rupert Murdoch ...
   
2. Publicly held newspapers will slim down and (of course) abandon the print editions. This will (of course) be extremely painful. At the end of the day newspapers will occupy niches including reality based, "balanced" (meaning equal praise regardless of reality like the Washington Post), party-based, and low brow entertainment. The reality based newspaper will be a premium good -- small audience, high cost. Fifteen years ago The Economist owned this space. Today the NYT would be the closest contender, but I think they'll end up being private. I wonder if a new venture will try to emulate the Old Economist.
   

Experts failing - how to get better guidance for economics, health care and everything

Somewhere in my library, I have an older book that compiled all the different ways experts have been wrong over the decades. Now that I've accumulated more personal entropy I have little need of the book; I've lived through cycles of abandoned medical fads endorsed by panels whose "expertise" was exceeded only by their egos.

Today, of course, we're specifically wondering about expert economists. The NYT proposes Groupthink (Wikipedia) as a villain ...

Economic View - Challenging the Crowd in Whispers, Not Shouts - NYTimes.com

... In his classic 1972 book, “Groupthink,” Irving L. Janis, the Yale psychologist, explained how panels of experts could make colossal mistakes. People on these panels, he said, are forever worrying about their personal relevance and effectiveness, and feel that if they deviate too far from the consensus, they will not be given a serious role. They self-censor personal doubts about the emerging group consensus if they cannot express these doubts in a formal way that conforms with apparent assumptions held by the group...

As an expert in having opinions, I'll respond. Groupthink is probably a contributing factor, but it's not the whole story. Happily we know of a fix for the bigger problem. Unhappily, Newt Gingrich blew away one of our best examples.

First some history, then the lessons.

For much of the 20th century medical consensus was achieved the way economics panels work today. A group of "experts", typically involving quite a bit of Harvard, met and pontificated. These groups lacked neither confidence nor ego, but their accuracy left a lot to be desired.

After 80 years or so of this, we evolved something that's now called "evidence based medicine", but had another label in the 80s (sorry, the labels blur together). In the 1980s we even had a federal agency that took an "evidence based" approach to recommendations, one founded in science and reason rather than reputation and ego. It was known then as the Agency for Health Care Policy and Research (AHCPR).

Problem is, the AHCPR's expert panels came up with recommendations that were very different from the old-style "experts". That was bad enough, but these recommendations affected payment. Orthopedic surgeons, in particular, were furious when an AHCPR guideline recommended far less back surgery. Gingrich (remember him) zeroed out funding for the AHCPR (great story in the link).

Yes, even then the Republican War on Reason had begun. Remember that the next time you hear Newt pontificate. For all of his intellect and pride, he began our long fall.

Of course the AHCPR panel was right. Now, fifteen years later, even orthopedic surgeons agree with the original recommendations. As far as I can tell, the IPCC took a similar approach to their recommendations.

So expert panels, done right, can be very effective. They also acquire very powerful enemies, and without a supportive political environment they will be destroyed.

The AHRQ rose from the ashes of the AHCPR, but it was never again so bold. It has largely hued to the conventional approach to "expert panels", though it's clear that the leadership would like to return to the days of reason.

If Obama wins, they will.

So history teaches us that the conventional approach to "expert consensus" is deeply flawed. Maybe it's the groupthink, maybe it's the way the "experts" are chosen, but the AHCPR lesson tells us that there's a better way. There's an approach that combines a rational process with open discussion and a certain measure of humility about what we don't know.

It would be insane to dispense with expert consensus all together. That way lies climate change denialisism, vaccine autism and other triumphs of unreason. We can, however, do far better than the old style of "medical experts" and the current practice of "economics experts".

We can study when expert panels work and when they fail, and codify and continuously improve the best practices. We can develop an empirical and informed science of expertise.

Or we could keep blowing up the world economy, bake the planet too, and leave our children wandering in the rubble.

We do have a choice. Two days from now in fact. Reason, or the dark ages. Pick the one you like ...

Iceland is first to the Great Depression II

While most of the world enters the Great Recession, Iceland has skipped directly to the Great Depression II:

Iceland, Mired in Debt, Blames Britain for Woes - NYTimes.com

... Iceland’s key interest rate now stands at 18 percent. The currency, the krona, has declined 44 percent in the last year. Mr. Danielsson, the economist, visited the country recently and found the situation grave.

“Salaries are frozen, food prices are shooting up and they are laying off people left, right and center,” he said. “Companies are going bankrupt all over the place. It’s unimaginable how bad it is.”

Ms. Gisladottir said Britain’s decision had sent Iceland back some 30 or 40 years, to a time when it was an isolated, poor country, dependent mostly on its fishing trade.

“This is a major crisis,” she said. “We haven’t been in this situation for, probably, ever. We cannot solve it alone. We need solidarity from partners, from friendly countries, and we thought the U.K. was one of them.”

Iceland's population is about 300,000, so keeping them from disaster shouldn't tax the IMF very much.

The story of Iceland's collapse has gotten some modest coverage over the past few weeks. My distant sense of the story was that Iceland had transformed itself along the lines of some of the Caribbean banking states, and that their neighbors weren't entirely delighted. I also recall that Iceland declined to join the EU a few years back, because they thought they'd do better with a freer economic hand.

So now the EU nations are inclined to let Iceland defrost. Maybe for another week or two.

Eventually the IMF will launch a rescue, and Iceland will be allowed to join the EU. They may be limited to conditional membership for a while.

I'm still expecting most of the world will stick with the Great Recession, and forego the Full Monty.

McCain's strategy reaches for the disconnected

I didn't think there was any GOP ploy inane enough to befuddle Fox TV.

I was wrong. (note: Sam Wurzelbacher is "Joe the Plumber")

Editorial - Shepard the Anchor - NYTimes.com

... Mr. Wurzelbacher has held press conferences. He was interviewed on the “CBS Evening News,” where he compared Mr. Obama to Sammy Davis Jr...

At a campaign rally this week, Mr. Wurzelbacher was filmed agreeing with another McCain supporter that electing Mr. Obama would be “death to Israel.”

For five painful live-on-Fox minutes the next day, Mr. Smith repeatedly asked Mr. Wurzelbacher what evidence he had to back up that charge. Mr. Wurzelbacher refused to answer. He said it was up to Mr. Smith’s viewers to figure out why he, Joe the Plumber, thought Mr. Obama was a menace to Israel.

Looking incredulous, Mr. Smith gave up. He read a statement from the McCain campaign praising the plumber’s “penetrating and clear analysis.”

Then Mr. Smith said: “I just want to make this 100 percent perfectly clear. Barack Obama has said repeatedly and demonstrated repeatedly that Israel will always be a friend of the United States no matter what happens once he becomes president of the United States. His words.”

“The rest of it,” he said, “man, it just gets frightening sometimes.” ...

So is McCain insane?

Sadly, no.

If we exclude the alleged 66% of people who now won't answer pollster phone calls (so what does that say about polls?), we're left with 10% of the electorate that's still in play.

How can they still be undecided?

These are Americans who don't have much contact with the wider world. They don't use reason or logic (not even the logic of those who feel abortion is legalized murder), they aren't connected to a religious community or other organization that controls their voting behavior, they are simply adrift.

Outreach to this group demands the bizarre.

That's what McCain is doing.

He doesn't have much choice. The reason-based community, from libertarian to rational conservative to liberal has deserted him. The non-fundamentalist religious community has largely deserted him. He's left with the fundamentalists, the racists, the hard-core GOP talk radio audience, and the disconnected.

The worst bit is that when you add up the hard-core GOP, the racists, the fundamentalists and the disconnected you might have a majority of the American people.

We'll find out soon enough.

Call a rational young person and compel them to vote.

The undecided explained

Gail Collins explains who the undecided voters are ..

Op-Ed Columnist - Our Election Whopper - NYTimes.com

... In The Times’s poll, the percentage of respondents who said that they weren’t totally sure who they were going to vote for was almost identical to the percentage who said that they think the economy is doing well. Are they the same people? If so, perhaps they are still undecided because they are waiting to get their marching orders from well-informed friends like Abraham Lincoln, St. Catherine of Siena or Seabiscuit...

If I were Tyrant I would identify every voter who's still undecided 10 days before an American election. I'd then remove their right to drink, vote or drive.

One of the many reasons I'll never be able to run for office!

McCain/Palin closing in Pennsylvania - what you can do

At our Halloween party our local political expert was confident Obama would do well.

I don't believe that. I feel the future teetering.

For example:

Talking Points Memo | Election Central Saturday Roundup - Today's new Rasmussen shows McCain pulling to within 4 points of Obama in Pennsylvania.

Meanwhile Bush officials illegally leaked news on an immigration investigation into an Obama paternal relative living in Boston. No surprise there, we learned years ago that Bush answers only to His God; American law need not apply. Who knows what other divine directions Bush will follow in the next few days.

Elsewhere 18-35 year olds have been under-represented in Florida's early voting.

If civilization wins it will be by the narrowest of margins and we'll be clawing and scraping our way to the finish line. Or civilization could lose, and Gaia will celebrate the coming of President Palin.

So forget about calling Florida Jewish elders. Start calling your non-GOP friends, children, nieces and nephews in Pennsylvania, Florida and everywhere. Get them to vote early if they can. Tell the kids they need to get five youngsters to the polls or they'll be eating spam for thanksgiving, living on coffee shop earnings, and inheriting the wind.

Not to mention living in the dregs of a decaying civilization ruled by President Palin.

Phone. Email. Cajole. Threaten. Bribe. Donate.

The end of the personal check - what do we do now?

Donald Knuth, a famous programmer, writes about why he's stopped using checks:

eKnuth: Recent News

... due to an unfixable security flaw in the way funds are now transferred electronically, worldwide, it is no longer safe to write personal checks. A criminal who sees the numbers that are printed at the bottom of any check that you write can use that information to withdraw all the money from your account. He or she can do this in various ways, without even knowing your name --- for example by creating an ATM card, or by impersonating a bank in some country of the world where safeguards are minimal, or by printing a document that looks like a check. The account number and routing information are all that international financial institutions look at before deciding to transfer funds from one account to another...
... More and more criminals are learning about this easy way to acquire money, and devising new schemes to conceal their identities as they steal the assets of more victims...

So what's new about this? Check forgery is an ancient crime; after Emily's check book was stolen we stopped carrying checks. That's when we learned that people who lose checks are legally responsible for all losses against the account -- and closing the account is no protection. (In practice, of course, banks grumble but eat the losses. Otherwise, checks would have died a just death years ago.)

What's new is the same process that turned credit card fraud from a small time crime into a vastly profitable criminal enterprise. Yes, I was at ground zero for that transition too.

Sometimes I feel like a punching bag for history.

The fraud isn't new, and the security systems haven't become suddenly worse. What's new is globalization, accelerated knowledge diffusion, and facilitating technologies.

Globalization means that it's easy for the criminal to operate outside of legal boundaries. Knowledge diffusion and social networking, which work as well for bad as good, mean that effective techniques evolve rapidly and spread worldwide. Facilitating technologies support knowledge diffusion, automated attacks, and cheap forgeries.

The portable checkbook died at least ten years ago. Now we've moved to the next step, and it's good-bye to the baby sitter check.

So how shall we pay for services? For now we use a fair bit of cash, but it's a bit clumsy. PayPal is the obvious alternative, but I have no love for either eBay or PayPal. I know they're trying to clean up their act, but I'll hold my grudge a bit longer.

Google Checkout isn't as close to Microsoft Wallet (Remember that from IE 3? Almost nobody does, there isn't even a good Wikipedia entry) as I thought it was, but clearly both Google and Microsoft are ready to replace the check. They should be able to buy a lot of banks very cheaply right now.

I liked Amazon payments when it first came out. My confidence was shaken when they set me up with an identity confusion bug that they couldn't fix. Still broken, two months after I reported it.

See what I mean about the punching bag bit?

So I'm not that keen about Amazon payments.

So Amazon is showing a worrisome competence gap. Google has been holding back from this domain. PayPal has a shady past. Microsoft has Monopoly baked into their DNA.

So what do we need?

We need government and the market working together. We need law, regulatory frameworks, rules to prevent monopolies, and then we need Microsoft, Google, Amazon (please slap them a few times) and even PayPal to compete.

Of course to get that kind of government, we need to do something first.

Nature itself endorses Obama, reason, and the enlightenment

Nature the journal, perhaps the world's preeminent representative of knowledge and reason, has expressed their first presidential endorsement in over 100 years. Emphasis mine.

America's choice : Article : Nature

The values of scientific enquiry, rather than any particular policy positions on science, suggest a preference for one US presidential candidate over the other.

The election of a US president almost always seems like a crossroads, but the choice to be made on 4 November feels unusual, and daunting, in its national and global significance.

... science is bound by, and committed to, a set of normative values — values that have application to political questions. Placing a disinterested view of the world as it is ahead of our views of how it should be; recognizing that ideas should be tested in as systematic a way as possible; appreciating that there are experts whose views and criticisms need to be taken seriously: these are all attributes of good science that can be usefully applied when making decisions about the world of which science is but a part. Writ larger, the core values of science are those of open debate within a free society that have come down to us from the Enlightenment in many forms, not the least of which is the constitution of the United States.

On a range of topics, science included, Obama has surrounded himself with a wider and more able cadre of advisers than McCain. ... He tends to seek a range of opinions and analyses to ensure that his own opinion, when reached, has been well considered and exposed to alternatives. He also exhibits pragmatism — for example in his proposals for health-care reform — that suggests a keen sense for the tests reality can bring to bear on policy.

Some will find strengths in McCain that they value more highly than the commitment to reasoned assessment that appeals in Obama. But all the signs are that the former seeks a narrower range of advice. Equally worrying is that he fails to educate himself on crucial matters; the attitude he has taken to economic policy over many years is at issue here. Either as a result of poor advice, or of advice inadequately considered, he frequently makes decisions that seem capricious or erratic. The most notable of these is his ill-considered choice of Sarah Palin, the Republican governor of Alaska, as running mate. Palin lacks the experience, and any outward sign of the capacity, to face the rigours of the presidency...

...This journal does not have a vote, and does not claim any particular standing from which to instruct those who do. But if it did, it would cast its vote for Barack Obama.

The human world will weep on November 5th when President-to-be Palin takes her oath of office.

What of the natural world? I can't imagine Gaia having an opinion, but if it did it might favor a planet with fewer humans. Gaia, I think, would vote for McCain/Palin.

Google is doing OCR on PDF wrapped document scans

The Google blog struggles to explain why their latest technical achievement is important...

Official Google Blog: A picture of a thousand words?

... We are now able to perform OCR on any scanned documents that we find stored in Adobe's PDF format... This is a small but important step forward in our mission of making all the world's information accessible and useful...

So why is this important, yet hard to appreciate?

The first problem is that most people think of PDF as a text container. Indexing a text container is nothing special. What's less appreciated is that PDF is the de facto standard way to package a scanned document [1].

So what's novel about doing character recognition on a scan? OCR on 600 dpi B&W document scans is no great trick. Adobe's PDF client has more or less done that for about 10 years [2], and Windows' (formerly Xerox) ancient and under-appreciated document imaging has had this ability since the dawn of time.

The trick is implementing this affordably on millions and billions of PDFs indexed on Google's servers.

That's impressive and it is going to open a vast amount of knowledge.

Good for you Google!

[1] I figured this would happen in the 90s before there was any clear answer to the scanned document representation question. There are bizarre technical issues with scanning into PDF, but it's a great format overall. (Hint: Ancient fax-style lossless compression of a "B&W" document scan is much more efficient than readable JPEG compression of a gray scale scan.)

[2] They sometimes hide or remove this feature depending on what else they're selling.

Firewalls and separation of powers: banking, government, medicine and pharmacy

When I wrote about firewalls a few weeks ago I focused on contagion ...

Gordon's Notes: Systemic failure and financial firewalls

... even if there are deeper economic and cultural failures, there are also more straightforward firewall failures in our current crisis. These are usually called "regulatory failures" but regulation can come in many forms. I think the most interesting forms are those that are designed to stop the spread of contagions.

Fires, seizures, epidemics, hurricanes and financial crises are all, famously, "chaotic". They have non-linear perturbation sensitivities, and they can roar up and die down in ways that are only loosely predictable.

Excepting hurricanes, we have firewalls for these things. In our brains are systems to dampen seizures should they arise, and, we think, to limit where they spread. In our buildings we have, well, firewalls. In public health we find immunization rings, targeted interventions, quarantine and the like...

... Firewalls don't show up, to my knowledge, in classical economics. I'm sure they show up in modern economic models of regulation and in studies of "complex adaptive systems" [1]. Maybe this latest crisis will bring models of financial system firewalls, like the mourned Glass-Steagall act, to the level of popular economics.

Glass-Steagall separated commercial and investment banking. One effect was to reduce the risk of contagion, but I think the intent was to reduce conflicts of interest.

Managing systemic conflicts of interest is one reason America's political system separates power between Congress, the Executive and the Courts. (One of the reasons Bush was able to fully leverage his incompetence was that the GOP controlled all three, and had near-control of the media as well.)

Conflict of interest is one reason, for example, that it's a very bad idea for orthopedic surgeons to own imaging facilities.

Speaking of which, there's yet another separation of powers that's waned over the past tweny years.

At one time American physicians dispensed medications and pharmacists prescribed. That's still true in many nations. Shockingly, the result was very high use of very inappropriate medications. The cure was separation of powers. Physicians would prescribe and pharmacists would dispense.

Time passed. Lessons were forgotten. Market deism and libertarian ideals joined forces. Now we have minute clinics owned by dispensing organizations, and oncologists who make a large share of their revenue by the margin on dispensed drugs.

I am very confident that we will rediscover that there was a good reason to separate prescribing and dispensing. We'll find, for example, that minute clinics dramatically increase the cost of "treating" self-limited conditions -- not to mention the sale of diet pills, supplements, and candy.

Firewalls to contain epidemic chaos. Separation of powers to manage fundamental conflicts of interest in an imperfect world of imperfect communication and incomplete knowledge. They both reduce efficiency. They are both essential. Sometimes they're the same thing.

I do wish our meta-memory wasn't so short.

Don't listen to Judith Warner. Worry.

Judith Warner tries to tell us not to worry, that things are getting better.

Of course on closer inspection she's really saying that that either Obama/Biden will win, in which case we should be happy now, or McCain/Palin will win, in which case we should celebrate Bush's end because soon we'll despair.

I admit, there's a deeper logic there. After all, relatively soon we'll all be dead, so we might as well be happy now.

In the meanwhile, don't believe the polls. The turnout of "young" (under 35) voters in early Florida voting has been lousy. Just like Kerry, who might have won if the under-35s had actually voted like they said they would.

If the "young" stay home, they will give us President Palin.

Punishment for the boomers? Well, I can understand that. Silly though, the young will live with the consequences longer than I will.

George Will?!

George Will endorses Obama.

Who's left? 

Dick Cheney?

Jesus.

I can't possibly support somebody George Will has endorsed.

My head has exploded.

Again.

The Economist's endorsement: What's surprising about it.

The Economist has been running a world "electoral college" . Last I looked 80% of their US readers voted for Obama.

80%.

That's current readers of The Economist, a journal that used to be rationalist 19th century liberal but became a pale imitation of the Wall Street Journal editorial pages in the 90s. Even this readership, the very heart of McCain's former constituency, is massively pro-Obama.

So I figured the "paper" would endorse Obama. If 80% of the US readership of a WSJed-lite publication wants Obama, they aren't going to be stupid.

Still, in 2000 they endorsed Bush. They never adequately apologized.

In 2004, they weakly, half-heartedly, with poisoned pen, "endorsed" John Kerry as "the incoherent".

So I was expecting a grudging, muttered, meaningless endorsement.

Instead we got ...

Obama on the cover, striding along. Headline "It's time".

There's nothing poisonous about this endorsement (emphasis mine):

An endorsement of Barack Obama | It's time | The Economist

... all the shortcomings of the campaign, both John McCain and Barack Obama offer hope of national redemption. Now America has to choose between them. The Economist does not have a vote, but if it did, it would cast it for Mr Obama. We do so wholeheartedly ...

"Wholeheartedly". A carefully chosen word.

They are not forgiven. They will never be forgiven for their 2000 endorsement of GWB.

Still.

It's something.