The limits of statistical methods: health, wealth and smoking

Rich people live longer than poor people.

We used to say this was because rich people took better care of themselves and smoked less, wore seatbelts, got vaccinated, had dental care, bettery bypass surgery, better breast cancer care, etc.

Then, a few years ago, a meme developed that the gap was due largely to power relationships. There was something about being on top that made one live longer (presumably this would be true of other social animals). The statisticians claimed that they'd controlled for the effects of smoking, seat belts, etc.

Now, smoking is back [2]:

Smoking is to blame for half of the difference in male death rates between men in the top and bottom social classes, say international researchers...

Half is quite a bit considering that researchers previously thought they'd accounted for the effects of smoking on mortality gaps.

We've seen this many times in healthcare research over the past decades -- case control research is essential and suggestive, but caution is always indicated.

We can't randomize infants to being rich or poor, or switch thousands of accountants and CEOs, so there's no alternative to population research. The results become more persuasive when reinforced by other lines of inquiry [1]. So if the power=health meme is reinforced by animal studies where one can randomize status it becomes stronger, but smoking is a powerfully proven source of mortality.

Occam's razor favors smoking as the simplest explanation for mortality differences, and researchers know that, so for me this is really a story about how hard it is to draw strong conclusions from population studies.

The battle will go on, and power relationships may indeed be more important as smoking decreases, because we need data to guide policy. Is it better to put effort into immunization adherence, smoking bans, breast cancer screening or liver transplants [3]?

--

[1] Science is about consistent and reinforcing models, each supported by variable amoungs of testable predictions. Where tests are less rigorous, we rely more on integration with other parts of the knowledge model.

[2] The slimeballs haven't given up yet, btw.

[3] The transplant bit was a joke. To date reducing the rich/poor mortality gap has seemed either relatively inexpensive (seatbelt laws, immunization, smoking bans) or impossible (substance abuse, power relationships). If things like expensive biosubstances, transplants, or stem cell therapies become more important the social strains will be significant.

Income growth in 2004 and exponential gaps

DeLong channels Krugman, now hidden from us by the NYT's paywall:

Brad DeLong's Semi-Daily Journal: Krugman - The Further Derangement of the U.S. Income Distribution

... Here's what happened in 2004. The U.S. economy grew 4.2 percent, a very good number. Yet last August the Census Bureau reported that real median family income -- the purchasing power of the typical family -- actually fell.

... in 2004 the real income of the richest 1 percent of Americans surged by almost 12.5 percent. Meanwhile, the average real income of the bottom 99 percent of the population rose only 1.5 percent. In other words, a relative handful of people received most of the benefits of growth.... Even people at the 95th percentile of the income distribution -- that is, people richer than 19 out of 20 Americans -- gained only modestly. The big increases went only to people who were already in the economic stratosphere.... [T]he real earnings of the typical college graduate actually fell in 2004.

Real earnings are after inflation. So the vast majority of US productivity gains in 2004 went to the wealthiest 1%. Everyone else fought over the scraps, with college grads actually losing ground (hmmm, outsourcing influence? It's hard to outsource plumbing, easy to outsource accounting).

The top 1% is departing from the bottom 99% at an exponential rate -- ascending to a neo-medieval world of relative power.

The curious thing about this is that Americans don't seem to care. I've heard the usual explanations of this (everyone thinks they might get rich too) and I find them hard to believe. On the other hand, I've realized in my old age that I'm not much like other people [1] and I can't really model their thinking ...

[1] The fact that this came to me rather late in life says something about my perceptual limitations ...

Democrat fund raising: incompetence or dirty tricks?

We worked pretty hard for the Kerry campaign. Our reward now is relentless spam from Minnesota DFL candidates and phone calls from other groups claiming to be related to democratic candidates. I believe the political parties are exempt from the telemarketer laws, so they have free rein. It's so obnoxious my wife jokes that these must be undercover GOP initiatives, trying to drive away funding for liberal candidates.

Well, full credit to the tricksters if there are any -- it's working. I suspect, however, that this is the result of sheer, unadulterated, incompetence and stupidity rather than sabotage.

I bet the GOP isn't nearly this braindead about exploiting their donors.

If there are any grown-ups with influence in dem fund raising -- wake-up! You're going to drive the party off a cliff. Stop the spam (the 'take me off your list' links only work transiently, we get added back in periodically), stop the calls, rethink this.

Or maybe it's all Libertarian tricksters ...

PS. The Dems should take some lessons from CARE Intl. When we first sent them a donation I wrote that I'd stop the moment they bugged me for more. They put me on some kind of 'if you call this guy you die' list -- we never get anything. Every year they get their check.

Memories of a lost brother - July 12, 2002

It's that time of year again, now four years since my brother disappeared on a day hike in the wilderness around Whistler, British Columbia. There's no news, but the hiking season begins once again ...

Contextual Google Ad above my Gmail spam list

French Fry Spam Casserole - Bake 30-40 minutes.

Cute. Google has finally added a 'delete all spam' link. It took out 6500 items at once -- 30 days of spam.

Physics in crisis: the non-zero vacuum energy

When I did my BS in 1981, I bought a copy of Thorne, Wheeler and Meisner's 'Gravitation'. I figured I'd study it in retirement, with a computer aide to do the math for me. It's a marvels book, one of the most extraordinary textbooks I've seen anywhere. My copy is autographed by Kip Thorne.

I think of that book now when I read of a conference Thorne attended, a Caribbean party for celebrity cosmologists. This comes from a report of the meeting:

Edge: THE ENERGY OF EMPTY SPACE THAT ISN'T ZERO: A Talk with Lawrence Krauss

...When you apply quantum mechanics and special relativity, empty space inevitably has energy. The problem is, way too much energy. It has 120 orders of magnitude more energy than is contained in everything we see!...

...One of the greatest developments in physics in the 20th century was to realize that when you incorporate special relativity in quantum mechanics you have virtual particles that can pop in and out of existence, and they change the nature of a hydrogen atom, because a hydrogen atom isn't just a proton and electron.

That's the wrong picture, because every now and then you have an electron positron pair that pops into existence. And the electron is going to want to hang around near the proton because it's oppositely charged, the positron is going to be pushed out to the outskirts of the atom, and while they're there they change the charged distribution in the atom in a very small, but calculable, way. Feynman and others calculated that effect, which allows us to get agreement between theory and observation at the level of nine decimal places. It's the best prediction in all of science. There's no other place in science where, from fundamental principles, you can calculate a number and compare it to an experiment at nine decimal places like that.

But then when we ask, if they're there, how much should they contribute to the energy in the universe, we come up with the worst prediction in physics. It says if empty space has so much energy we shouldn't be here. And physicists like me, theoretical physicists, knew they had the answer. They didn't know how to get there. It reminds me or the Sidney Harris cartoon where you've got this big equation, and the answer, and the middle step says "And then a miracle occurs". And then one scientist says to another, "I think you have to be a little more specific at this step right here".

The answer had to be zero. The energy of empty space had to be precisely zero. Why? Because you've got these virtual particles that are apparently contributing huge amounts of energy, you can imagine in physics, how underlying symmetries in nature can produce exact cancellations — that happens all the time. Symmetries produce two numbers that are exactly equal and opposite because somewhere there's an underlying mathematical symmetry of equations. So that you can understand how symmetries could somehow cause an exact cancellation of the energy of empty space.

But what you couldn't understand was how to cancel a number to a hundred and twenty decimal places and leave something finite left over. You can't take two numbers that are very large and expect them to almost exactly cancel leaving something that's 120 orders of magnitude smaller left over. And that's what would be required to have an energy that was comparable with the observational upper limits on the energy of empty space.

We knew the answer. There was a symmetry and the number had to be exactly zero. Well, what have we discovered? There appears to be this energy of empty space that isn't zero! This flies in the face of all conventional wisdom in theoretical particle physics...

This is the best summary of the physics dilemma I've read. The same math leads to a perfect prediction and a perfectly outrageous prediction. So we know the physics is very broken, but we don't have a path to follow. The best minds in the world can't come up with experiments to guide us. No wonder physicsts are extremely frustrated! I just hope I see a breakthrough before I get to meet Einstein in person ...

PS. Virtual particles came courtesy of Richard Feynman. I have his books too, but he was a much better lecturer than textbook writer.

Apple and Blackberry: be still my heart

My marriage to the Palm has been on the rocks for years. She's never been the same since she got a color screen, and let's not talk about the many tormented assignations with Microsoft Outlook/Exchange, the binge crashing, the discarded styli ... A dysfunctional menage a quatre, bien sur.

In despair I've contemplated whatever the heck Microsoft calls their PDA platform these days. Is it Microsoft Pouch now? I've stayed away because I don't know any geeks still using one of those Microsoft thingies. As bad as Palm is, and it's awful bad, Microsoft must be even worse.

All hope abandoned, I gaze longingly at Amazon's Blackberry reviews. Wow, those are enthusiastic users. Reminds me of the Palm Pilot -- before she sold her soul for rock and roll. Alas, Blackberry is all about Exchange servers and Windows -- and I'm a Mac guy at heart (no matter that I know XP far too well).

Which is why I will throw caution to the winds and believe this rumor:

Independent Online Edition > Business News:

... Apple is also said to be closing in on a new product launch in collaboration with Research in Motion, the maker of the Blackberry...

Forget the wireless music stuff. I've got plenty of CDs to listen to, thanks. It's the Blackberry integration I want -- especially if Apple/RIM supports multi-machine sync with the ability to select what syncs on a given machine (eg. Home/Work separation). Believe.... I must believe ....

I feel like the Newts when Jobs killed the Newton, praying that the US Robotics PalmPilot would save them ...

An excellent article on sociopaths and the military

I haven't blogged on the alleged rape/murder incident in Iraq because I had nothing to say. It sounded like true criminality, whereas the Haditha incidents are more complex. I do want to point out an excellent NYT discusion however. Mr Carey provides a good background and discusses the distinctions between an elite special forces soldier and a superficially similar sociopath.

Update 7/11: FMH is more critical of the article, and brings his psychiatric expertise to bear. Good discussion.

CallerID scams: the perils of flawed trust mechanisms

As Schneier repeatedly tells us, there's a big risk to a trusted identity mechanism. For example, expedited airport security checks based on special IDs increases the risks of a flaw in the ID mechanism. That's what's happened to caller ID, with unsurprising consequences:

USATODAY.com - Caller ID scammers plan to do a number on you:

...The AARP Bulletin recently reported a scam in which people received fraudulent calls claiming they missed jury duty and asking for their Social Security number. The calls seemed legitimate because the telephone number of the localcourthouse showed up on caller ID.

In Pennsylvania, constituents of Republican Rep. Tim Murphy were flooded with bogus calls from someone purporting to be from Murphy's office.

The primary worry for consumers is that if a call appears to be coming from their bank, credit card company or a government agency, they could be persuaded to give up financial data a thief could use to open new bank accounts or apply for loans and credit cards.

'It's a new way to scam people, because people rely on caller ID,' says Sid Kirchheimer, author of 'Scam-Proof Your Life' and the AARP Bulletin's Scam Watch columnist."

Sigh. This one came out shortly after I posted my essay on the threats ahead. Not far ahead evidently. The USA today article comes with a set of recommended safeguards; they remind me of the 'duck and cover' recommendations for nuclear attack ...

All the vulnerable people: eFraud, aging and special needs

Eight years ago I wrote a web page on Fighting Spam. That was a year after I'd first suggested to an ISP (Mindspring then) that they provide spam filtering services.

Alas, the spam deluge continues. My Gmail spam filter was stable at 5500 spams/month for about a year, but now it's up to 6500 spams/month. The zombie bots are getting worse.

Spam is bad, and it's sad that we still haven't adopted relatively inexpensive fixes like reputation management of authenticated sending services. I've come to realize, however, that the problems of spam are only the leading edge, the snout in the door, of something much worse. The most dangerous spam is increasingly about fraudulent schemes; desperate corporations like Vonage, Cingular, Yahoo and Delta are only marginal contributors. The spam is spawning phishing, splogs, and VOIP supported phone fraud, combining age old scams like the Publisher's Clearinghouse parasite, state lotteries, or "low interest credit card" scams with new technologies.

These fraud strategies are merging, morphing, and evolving with extraordinary speed, fueled by the worldnet. Charles Stross writes about sentient financial instruments, but one could as easily see how fraud strategies might be an even better candidate for emergent sentience [1]. Even as this happens, the prey population is growing with the aging of the wealthy western nations and the predator population is growing as the young and the desperate come online.

It takes a fair bit of intelligence, discipline and experience to see through these schemes and to to monitor one's human frailties. My handful of readers are likely immune. Not so our aging parents, not so the 50% of our population with IQs under 100. One day, all too soon, my IQ too will drop below some magic threshhold and I will join the population of the vulnerable. Most of us will, unless we die first. An increasingly complex world will offer endless opportunities for highly refined schemes to separate the vulnerable from their assets.

We're going to have to evolve new systems of defense, trust relationships, identity management and reputation management. Developing these systems will be a major social challenge over the next few decades. In the meantime, encourage your parents, and your vulnerable family members, to consult about their financial decisions.

[1] One of the leading theories for a driving force behind the evolution of the human mind is fraud detection and fraud invention.

Update 2/1/2010: See also - Phishing with the post-Turing avatar

Exploding LiOn batteries and air travel

[see update for my second thoughts]

I hadn't paid much attention to June reports of an exploding Lithium ion battery. This happens every few years, sometimes there are product recalls, sometimes not. Today a NYT article on the impact on Dell piqued my curiousity, and I read the Inquirer article. The pictures sure are impressive.

In the story a witness is quoted: "..It is only a matter of time until such an incident breaks out on a plane...". Hmm. Good point. When I was in high school a rogue lab tech demonstrated the joy of Lithium by dropping a chunk of it in a pan of water. That was one of my more memorable learning experiences. Lithium is wonderously reactive.

So how long before LiOn batteries are banned from air travel? It must now have occurred to a large number of people that a rare accident could be engineered into a planned event.

Update: Hmm. Second thoughts. There's not that much lithium in those batteries. Was that fireball photoshopped? Was there a big pile of paper beneath the laptop? One can set fire to a bag of newspaper and get a fireball on an airplane too ... I'd like to see a 'Dan's Data' analysis, but I suspect the FAA considered this problem a while back and decided it wasn't worth worrying about ...

Update 7/10: Dan did a quick private analysis for me and pointed to this public resource. Suffice to say LiOn batteries are not a security threat. The email from Dan also caused me to reflect again on something that's become apparent over the past five years.

Most terrorists, like almost all criminals, are not very bright. If someone like Dan went bad, the threat would be far greater -- but it's evidently rare for a truly bright or imaginative person to join a group like al Qaeda. Even their very few elite agents, like Atta or Zawahiri, were/are only a bit above average. I think that's the main thing that's kept us going, but it wasn't obvious to me when I wrote this.

Update 7/17/06But on third thought ...

The Empire strikes back - An Apple employee on the dangers of tech blogs

Last week I wrote about the revenge of the consumer. Now I'll give some airtime to someone from the other side:

After Apple - We Are, I’m Afraid, Only Human

.... So some guy on a blog writing that Apple are saying that MacBook top cases are being replaced when they are not doesn’t help anyone. Not the pissed off customer who travelled thirty miles to the Store, not the Genius, no-one. So please blog writers, and yes, you know who you are, for the love of God stop writing this nonsense as if you know what you are talking about because you don’t and you are doing more damage than you could possibly imagine in your wildest nightmares. Seriously. Apple are damm good at admitting mistakes and when there is a need for a product recall everyone will know about it on the support pages of the website or if you have registered your product correctly a direct email will inform you. Even if the product is out of warranty Apple will honour the repair if it’s been officially announced. Most electronics manufacturers don’t do this sort of thing for their customers, Apple do. Most electronic manufacturers don’t have a free tech support area in their Store, if they even have a Store in the first place, Apple do. Blog writers are not the authority, Apple are, that’s not arrogance, it’s a fact.

Hmm. Food for thought. I'm biased by my experience with 3 consecutive defective 3rd generation iPods (I kept the last one rather than go for yet another return -- the usb sync doesn't work), but my iBook video fears turned out to be misplaced. Macintouch has made a valiant effort to look deeper, and they find some Apple products have unusually high defect rates and some are remarkably solid. (The G4 iBooks are a wonder of reliability, the 3rd generation iPods were somewhat lemony.) I remain suspicious about the quality of Apple's central repair services -- even the sober minded Macintouch site cautions against buying Apple refurbished laptops.

On the other hand I do believe Apple is far more responsive and supportive than Dell, and vastly better than SONY, Panasonic, etc. I'll try to give them a bit more credit than I have ...

Spyware companies: Yahoo, Delta, Cingular

Business Week has written an expose of a spyware company -- Direct Revenue. Take careful note of their paying customers and investors:

Spyware developers net huge profits, outrage - Tech News & Reviews - MSNBC.com

Spyware rakes in an estimated $2 billion a year in revenue, or about 11percent of all Internet ad business, says the research firm IT-Harvest. Direct Revenue's direct customers have included such giants as Delta Air Lines and Cingular Wireless. It has sold millions of dollars of advertising passed along by Yahoo. ..

I'll be looking to avoid these three companies in the future. Yahoo must be in worse shape than the industry has realized. What they $#@$#! was Cingular thinking?

Update: A few others to live in infamy. Emphases mine:

By early 2005, Direct Revenue had notched deals with JPMorgan Chase, Delta, and the Internet phone company Vonage, according to former sales staffers and Direct Revenue documents. Cingular Wireless spent more than $100,000 a month at the peak of its relationship with Direct Revenue, current and former employees say. Direct Revenue put Cingular pop-ups in front of other phone companies' Web sites and news sites such as the one affiliated with tech magazine Wired. Vonage, meanwhile, was billed $110 for each customer that Direct Revenue delivered, according to a sales report from July, 2005. For that month, Direct Revenue billed Vonage for 287 new customers, or $31,570.

JPMorgan Chase confirms that it advertised with a Direct Revenue unit through the middle of last year, but says it was unaware of any spyware activity. Delta and Cingular declined to comment. Vonage didn't respond to inquiries.

...Many major companies, such as Cingular and Yahoo, have severed connections with Direct Revenue. But the ads of others, including Vonage, continue to appear in Direct Revenue pop-ups. Insight and TICC remain investors.

Vonage's share price is in free fall. They're making deals with devils, but it won't help them ...

Charles Stross and the Fermi Paradox

Charles Stross is a former pharmacist, former programmer and journalist, certified geek, and current full time writer. Most people would tag him as 'science fiction' writer. From what I've read of his journals, and especially his books, he's terribly bright and very imaginative.

Accelerando is one of his commercially successful books (you can scan it for free before you buy). The amateur Amazon reviews are well done (one of the two 'professional' reviews is by someone who didn't read the book); I can't add much to them. The book does not fully succeed as a novel -- it was published as a series of short stories and it doesn't hang together all that well. There are some annoying plot holes (no security on the goggles? Did one of the lead characters flee to alpha centauri or commit suicide? Why is Pierre asking what happened - he was there?!), some dangling and overly fluid characters, and too many synopses of 'what went before'. The writing itself is professional, and that's no mean trick, but the work would have needed a harsher editor and a complete rewrite to fly as a novel.

That's ok, because it's really a series of speculative essays disguised as a novel -- and the thinking is deep and creative. I thought I was being a bit whacky when I blogged about the spanish inquisition as a corporation, and the emergent sentience of corporations in the ecosystem of economic interactions, but Stross goes much, much further. He plays with the idea that at some point the relationship between finance wizard and financial instrument might be inverted, so that souls would be traded by sentient financial instruments. That's not bad; I can just about see how it might happen ...

The embedded essay I most enjoyed reading, however, is on one of my all-time favorite topics -- the Fermi Paradox. This is one of those conumdrums that bothers a very few people a great deal and is irrelevant to most of humanity.

In short, we ought by all rights, to be overrun by little green beings. The puzzle is that we appear to have much of the galaxy to ourselves. To Fermi fan-boys this is the biggest question around, compared to which matters of theology or epistemology are merely derivative.

The answer to the Fermi Paradox is most often expressed in the terms of the Drake Equation. The best bet is that something utterly inevitable ends all technological civilizations like our own in well under a thousand years. The most popular candidate for an "inevitable fate" over the past 23 years has been the Singularity (Greg Bear's 1982 short story 'Blood Music' is the earliest version of the Singularity theory I know of, Vernor Vinge developed the ideas extensively in the early 1990s.) Stross takes these ideas and pushes the boundaries. Why might a post-singular entity find travel unappealing? Why would it be hard for entities like us to live near such a beast -- even if it didn't spend any time thinking about us?

Reading Stross is like having an extremely bright and free thinking fellow over for a beer (or something, these UK writers seem fond of a range of substances). He tracks all over the place, the narrative doesn't always hang together, but it's a heck of a lot of fun -- and where else can a geek get his Fermi fix?

Follow Me Here on Bush fatigue

Follow Me Here is the blog I read that's most like this one -- though FMH is more popular and more outward directed. Recently FMH wrote a retrospective post about years of blogging. I particularly emphathized with this portion ...

Follow Me Here... : 07/02/2006 - 07/08/2006

... Third, I have an incredible degree of Bush fatigue; it is not that I cannot get outraged anymore, but there is only a finite roster of ways in which a government can lie, cheat, steal, kill, destroy, and oppress. Bush and his minions have long since done them all; I have long since taken note of them here; nothing surprises me, and my outrage is constant and numbing.

And I have only a limited tolerance for my own frustration and despondency that a more effective movement of opposition has not arisen in response to his outrages. And I have no confidence that weblogs like FmH are change agents. (I don't know what would be effective activism these days, I guess, but I can no longer rationalize as I did for so long that FmH was an integral form of activist activity)...

I also started this blog partly because I felt compelled to do something about the terrible choice America made in 2000 -- or that was made for America. Something beyond campaign donations and volunter efforts.

After Bush was reelected, however, I couldn't pretend any longer that anything I wrote was going to make much of a difference. I, slowly and reluctantly, accepted that I didn't understand America.

Sure the media has screwed up. Sure some powerful people have used a lot of money and dirty tricks. At the root of things, however, is an American public that doesn't care, doesn't understand, and doesn't want to know. I can't fix that.

Like FMH I've mostly given up on political blogging. I'll toss in a few reminders here and there of the most egregious sins of this administration, but in general I'll just assume they're methodically dismantling our democracy and destroying our physical, economic, cultural and social world. Same old, same old. America has chosen, and I'm about ready to march off to Galt's Gulch (oh, wait, you don't let in our sort? Oops ...)

I do enjoy writing about ideas and trends though, and I'll keep doing that -- if only for my own amusement ...