There may be atheists in foxholes, but there are few realists in the C-suite - or the White House.
Optimists rule, and they scorn realists as "pessimists" and "Cassandras" [1]. No matter than Kassandra Krugman is always right - still he is called Crow.
It smells like natural selection. In a universe where entropy rules, denial is a survival trait. Group selection, however, sprinkles a few realists about - grumpily cursed (by Apollo) to see things as they are.
Yes, the glass is half full. But that's good, because the wine is poisoned.
I think we're an oppressed minority.
No wonder realists love empiricism. Facts are our friends. We realists welcome science disguised in Harry Potter fan-fic ...
Harry Potter and the Methods of Rationality, Chapter 6: The Planning Fallacy (I added some paragraphs, emphases mine)
... "Muggle researchers have found that people are always very optimistic, like they say something will take two days and it takes ten, or they say it'll take two months and it takes over thirty-five years. Like, they asked students for times by which they were 50% sure, 75% sure, and 99% sure they'd complete their homework, and only 13%, 19%, and 45% of the students finished by those times. And they found that the reason was that when they asked people for their best-case estimates if everything went as well as possible, and their average-case estimates if everything went as normal, they got back answers that were statistically indistinguishable....
.... See, if you ask someone what they expect in the normal case, they visualize what looks like the line of maximum probability at each step along the way - namely, everything going according to plan, without any mistakes or surprises. But actually, since more than half the students didn't finish by the time they were 99% sure they'd be done, reality usually delivers results a little worse than the 'worst-case scenario'....
... It's called the planning fallacy, and the best way to fix it is to ask how long things took the last time you tried them. That's called using the outside view instead of the inside view. But when you're doing something new and can't do that, you just have to be really, really, really pessimistic. Like, so pessimistic that reality actually comes out better than you expected around as often and as much as it comes out worse. It's actually really hard to be so pessimistic that you stand a decent chance of undershooting real life. Like I make this big effort to be gloomy and I imagine one of my classmates getting bitten, but what actually happens is that the surviving Death Eaters attack the whole school to get at me...
It's music to my ears.
In my small world I see this every day. My optimist friend tells me it takes 30 minutes to enter expenses, but I track these things and I know it takes 1-2 hours. Another optimist says we'll deliver a new software feature in two months; I know that five months is optimistic and 8 months more realistic.
When we follow Agile Software Development rules, however, we base our estimates on examples from previous "sprints". We take "take the outside view". It works!
The Outside View is why Chile makes reasonable predictions about government finance, while elected officials force the CBO an artificial Planning Fallacy... (emphases mine ....).
Bias in Government Forecasts | Jeff Frankels (via Mark Thoma)
Why do so many countries so often wander far off the path of fiscal responsibility? Concern about budget deficits has become a burning political issue in the United States, has helped persuade the United Kingdom to enact stringent cuts despite a weak economy, and is the proximate cause of the Greek sovereign-debt crisis, which has grown to engulf the entire eurozone. Indeed, among industrialized countries, hardly a one is immune from fiscal woes.
Clearly, part of the blame lies with voters who don’t want to hear that budget discipline means cutting programs that matter to them, and with politicians who tell voters only what they want to hear. But another factor has attracted insufficient notice: systematically over-optimistic official forecasts.
... Over the period 1986-2009, the bias in official U.S. deficit forecasts averaged 0.4 % of GDP at the one-year horizon, 1% at two years, and 3.1% at three years. Forecasting errors were particularly damaging during the past decade. The U.S. government in 2001-03, for example, was able to enact large tax cuts and accelerated spending measures by forecasting that budget surpluses would remain strong. The Office of Management and Budget long turned out optimistic budget forecasts, no matter how many times it was proven wrong. For eight years, it never stopped forecasting that the budget would return to surplus by 2011, even though virtually every independent forecast showed that deficits would continue into the new decade unabated.
... to get optimistic fiscal forecasts out of the Congressional Budget Office a third, more extreme, strategy was required....
... To understand the third strategy, begin with the requirement that CBO’s baseline forecasts must take their tax and spending assumptions from current law. Elected officials in the last decade therefore hard-wired over-optimistic budget forecasts from CBO by excising from current law expensive policies that they had every intention of pursuing in the future. Often they were explicit about the difference between their intended future policies and the legislation that they wrote down.
Four examples: (i) the continuation of wars in Afghanistan and Iraq (which were paid for with “supplemental” budget requests when the time came, as if they were an unpredictable surprise); (ii) annual revocation of purported cuts in payments to doctors that would have driven them out of Medicare if ever allowed to go into effect; (iii) annual patches for the Alternative Minimum Tax (which otherwise threatened to expose millions of middle class families to taxes that had never been intended to apply to them); and (iv) the intended extension in 2011 of the income tax cuts and estate tax abolition that were legislated in 2001 with a sunset provision for 2010, which most lawmakers knew would be difficult to sustain...
...how can governments’ tendency to satisfy fiscal targets by wishful thinking be overcome? In 2000, Chile created structural budget institutions that may have solved the problem. Independent expert panels, insulated from political pressures, are responsible for estimating the long-run trends that determine whether a given deficit is deemed structural or cyclical.
The result is that, unlike in most industrialized countries, Chile’s official forecasts of growth and fiscal performance have not been overly optimistic, even in booms. The ultimate demonstration of the success of the country’s fiscal institutions: unlike many countries in the North, Chile took advantage of the 2002-2007 expansion to run substantial budget surpluses, which enabled it to loosen fiscal policy in the 2008-2009 recession ...
Humans are programmed to be foolishly optimistic, but group selection keeps realists around so that famines don't quite kill everyone.
If we know that our programming is defective, we can correct. Realists know we can learn, because sometimes we do learn.
[1] Update: I should add that Cassandra, was, of course, the ultimate realist. She was always right. Her Curse wasn't pessimism, it was that the Apollo made humans deaf to her warnings. The ancient Greeks apparently understood the planning fallacy. True pessimists probably exist, but they are rare enough that one should consider coexisting clinical depression.