Is Google winning the spam wars?

I've posted on Gmail and spam fairly often. A year ago things looked pretty bad, but then I realized that my email redirection was poisoning the domain reputation algorithms Gmail used back then.

From Sept 1996 through July 2007 Gmail's spam filtering was doing pretty well, but in July they had a serious screwup. Mercifully by August it was under control and the results have been great for three months.

It seems Google's Gmail team has also noticed things are going well, today they declared light at the end of the tunnel. Google OS followed up with a bit more detail:

... Many Google teams provide pieces of the spam-protection puzzle, from distributed computing to language detection. For example, we use optical character recognition (OCR) developed by the Google Book Search team to protect Gmail users from image spam. And machine-learning algorithms developed to merge and rank large sets of Google search results allow us to combine hundreds of factors to classify spam," explains Google. "Gmail supports multiple authentication systems, including SPF (Sender Policy Framework), DomainKeys, and DKIM (DomainKeys Identified Mail), so we can be more certain that your mail is from who it says it's from. Also, unlike many other providers that automatically let through all mail from certain senders, making it possible for their messages to bypass spam filters, Gmail puts all senders through the same rigorous checks...

For years I've written that the way to defeat spam was through differential filtering based on the managed reputation of the authenticated sending service. This little blurb is consistent with Google implementing that approach.

Today about 70% of Google's incoming mail is spam -- but that's an improvement! It used to be closer to 80%. Excluding a weird 2004 bump this is the most prolonged drop in three years.

My inbox is looking pretty good, and I hardly ever find anything in the spambox now (though I only scan about 20% of what I delete, I get a huge amount of spam).

Gee. I have something nice to say about Google!

The aviation "near miss" problem

Flaming airplane collisions are a relatively blunt instrument for aviation safety measurement.

Airlines can cut a lot of corners, stress the aviation system considerably, and still go years without a big newsworthy wreck. So the wrecks will happen faster than they ought to, but probably not fast enough to perturb the public.

It might be better to detect gross problems sooner. Which is why this AP story on NASA's suppression of aviation risk data got a brief bit of attention ...

The Associated Press: NASA Chief Regrets Agency's Statement

...Among other results, the pilots reported at least twice as many bird strikes, near mid-air collisions and runway incursions as other government monitoring systems show, according to a person familiar with the results who was not authorized to discuss them publicly.

The revelations this week prompted the House Science and Technology Committee to launch an investigation into NASA's decisions, with a public hearing scheduled for next Wednesday...

I gather all the near misses are not being reported. Perhaps airlines don't like their pilots relaying bad news, so they find ways to discourage it. We need the study results to understand the problem.

Of course the Bush administration hates bad news too, and they hate the thought of government having a job to do. So it's understandable that a NASA administrator might prefer to bury this report

Which brings me to the motivation for this post. Yesterday my wife's NWA small jet flight from Dallas got within about 25 feet of the runway at MSP before making a rather steep climb to cruising altitude.

Seems something was on the runway that shouldn't have been.

I wonder if that near miss was reported.

Oh well, the collapse in aviation safety is probably one of the lesser sins of the GOP. Eventually the market will sort things out.

Any resemblance to problems with childhood toys and the human and canine food supplies are completely coincidental.

Petraeus team plays the propaganda game

Glenn Greenwald reviews a bizarre correspondence with a military member of Petraeus staff.

There are two take aways. The first is Petraeus has some dim bulbs on this team. More importantly the correspondence shows us that the US military is playing the same propaganda game with US media that it plays in the Iraqi theater.

I recall this US-directed propaganda initiative was originally a Rumsfeld proposal; I'm not surprised it's continuing.

Proof that we can't trust what Petraeus says. We need other, more trustworthy, sources.

Some relevant old links from similar operations:

• 3/05: Bush use of fake news.
• 2/05: Bush payola operations
• 2/05: Goebbels on propaganda
  

Web 2.0, AJAX and the thin client iPhone: all overrated

Most of the time, I am quite fond of Gmail. Which is to say, the AJAX technologies that Microsoft gave us [1] were a nice improvement on what we used to call DHTML (JavaScript drive dynamic HTML). Reliable JavaScript [2] and AJAX are the technical foundations of what's still sometimes called Web 2.0 .

So far, so good.

Alas, that's as far as it goes. An infinite amount of money can't make Google Docs anything more than a pale imitation of the desktop office products of 1990. The very, very best web-based applications for creating web pages have about 2% of the power, functionality and performance of FrontPage 98. As in 1998.

Jobs much vaunted [3] iPhone web apps don't work when you're in an airplane, a tunnel, inside the bowels of an office building, in an elevator, in a remote cabin, outside the US or anywhere else that AT&T's network isn't performing at full speed and top quality.

Stop it.

Just stop it.

Give me thick client applications written in Pascal [4], C, assembler, FORTRAN, C#, Python ... even Java ... any day. I don't care how painful it is to code those suckers, that's why I pay money for the products.

Ok. I feel better now.

[1] The primary challenge to Microsoft's dominance of the desktop came from ... Microsoft. They weren't unique, but IE did set the standard.

[2] More irony. Microsoft's war with Netscape had, as a side-effect, the establishment of a standardized open JavaScript.

[3] I can only pray these are a delaying tactic to allow a true SDK to come to market -- which Jobs has sort-of promised us.

[4] I've never seen software that performed as reliably and as efficiently as the Mac Classic applications written in Pascal. I think they were even relatively immune to typical hacking measures. Just saying.

Amazon's eBook program: online and print

Amazon is now doing something similar to O'Reilly's Safari Books Online - but it's not just technical books and it's one book at a time:

How to Solve It: Modern Heuristics: Books: Zbigniew Michalewicz, David B. Fogel

Upgrade this book for $11.99 more, and you can read, search, and annotate every page online...

They've done this rather quietly. A book I ordered a year ago is now available for me to upgrade today.

I suspect it would be of most value for technical references - like this one. The advantage over O'Reilly's program is you can do it for just one book.

It's an interesting advantage to buying through Amazon, since it's only available for books on one's personal account.

Huckabee and Gail Collins

The other day I actually tried to defend something Gail Collins of the NYT wrote, even comparing her to Molly Ivins.

Today DeLong persuaded me that I was making a mistake:

Grasping Reality with Both Hands: Brad DeLong's Semi-Daily Journal

...The bottom line is that a woman is dead... because Huckabee went along with crackpot anti-Clinton conspiracy nuts and released someone with a significant history of violence and sexual assault...

Collins completely missed the back story of Hucakbee's pardon of a man who went on to murder a woman. It wasn't some act of reasoned compassion, it was a bizarre side-effect of the right's insane hatred of Bill Clinton.

That is one heck of a blunder; one that would have passed without notice in the days before blogs.

I'm sorry Molly, I won't do that again.

The Wellstone memorial and the path not taken

The family was wandering the Iron Range, exiled by yet another "school release day", when we came upon the Wellstone memorial.

We hadn't known it existed. The memorial brought back bitter memories.

It's been five years since Minnesota Senator Paul Wellstone's plane crashed as he flew north to attend the funeral of a friend. The 2002 election was only a week away.

Wellstone, his wife, his daughter, several friends and two pilots died in the crash. The pilot and copilot had made several errors, it was later determined that they were not qualified to fly the plane.

In October of 2002 America's future was balanced on the edge of a precipice. From much of 2000 to 2002 the Senate had been evenly balanced, but before the election a rational Republican had defected and the Senate went Democrat.

Wellstone's death pushed America over the edge. Minnesota and national talk radio hosts used a impolitic memorial service and the Rove playbook to bring GOP Norm Coleman to power.

Minnesota traded a Senator famed for his integrity for one of the most craven politicians ever to spring from Rove's foundry. The GOP had a one seat majority in the Senate. From 2002 through 2006 the GOP controlled the Supreme Court, the House, the Senate and the Presidency.

Everything that was to follow sprang from that poisoned moment.

On May 1, 2003 the US led an invasion of Iraq and by 2004 the name Abu Ghraib had come to symbolize the worst of America. An American era that began with the abolition of slavery and peaked with the 1947 Marshall plan ended in government approved torture and ideologically mandated incompetence.

Which is why the Wellstone Memorial is a genuinely historic site. For the moment it's is not terribly well marked and is little known outside of the area. On Google's maps it's near here -- give or take a kilometer.

When we arrived, on a cold wet Sunday morning, we were alone. During a visit limited by our children's limited patience two other cars arrived. I think the site gets regular visitors.

From The Iron Range

(click on the pictures to see larger versions, you can download a full res image from the associated album)

It is a very beautiful site, one of the loveliest memorials I've seen anywhere. Certainly it has the sweetest air and finest scents of any I've known.



The Wellstone family and friends are memorialized by ancient stones ...







In 2006 another historic Senate election turned on the the survival of another midwestern Democratic senator, Tim Johnson of South Dakota. This time the balance tipped the other way, Johnson survived his ruptured aneurysm and the Democrats took the Senate.

Now we have no choice but to hope that the worst is over. We can't go back to the days before waterboarding became a national sport, but maybe we'll start a new path.

A new American enlightenment. There's not much choice, really.

The nuclear apocalypses that should have happened

I've read of most of these before, but DI has a complete collection of publicly known nuclear attack false alarms like this one:

Damn Interesting » The Apocalypses That Might Have Been

.... Unlike the previous alerts, this event wasn't an error in the early-detection system, this missile was confirmed as real. Fearing the worst, the Russian military prepared to launch a full-scale counterattack against the United States. Planes were readied, and missiles sat waiting to launch a nuclear volley on selected targets in the United States at a moment's notice. Tensions were running so high within the Russian leadership that Russian President Boris Yeltsin activated his nuclear briefcase, enabling him to communicate with his top military advisers and review the situation online. This was the first time he had ever done so. Amidst this uncertainty, as many fingers nervously hovered over death-bringing buttons, word was received from Soviet military observers: the missile, while real, was not en route to Russia. It was a harmless research rocket headed for space...

Basically if soldiers had followed their orders properly you wouldn't be reading this and I wouldn't be writing it. In the unlikely event we were alive today, we'd be foraging for food. There certainly wouldn't be an Internet.

I really don't understand why human civilization is still around. I share the opinion of Oppenheimer and the Manhattan Project physicists -- there's no evidence to suggest that humans are capable of living with nuclear weapons.

Still. We're here. Apparently.

Odd.

Rat plagues wipe out indigenous rats

We know that European human plagues depopulated most of the Americas, and allowed the Puritans to scavenge from dead Amerindians.

I've speculated that European dog diseases (Distemper?) wiped out the indigenous new world dog (true Indian dogs became extinct long ago).

So it's interesting to read that European rats wiped out native Island rats -- or rather, their infections did:

Damn Interesting » The Crabs of Christmas:

... the [Christmas Island] rats were identified as another endemic species, the Maclear's rat. It seems that in the late 19th century, these rats were as numerous as the red crabs are today. Like the crabs they were scavenging creatures that lived in burrows on the forest floor, but the exact role they played in the ecology of the island will forever remain a mystery– for by 1903 the species was extinct, wiped out by an epidemic of trypanosome parasites introduced by ship-borne black rats...

Europe's dense urban populations cooked up lethal bioweapons, carried around the world by European animals.

The US does NOT have a problem with science education

It may have a problem with science related employment however ....

The Science Education Myth (Business Week, Vivek Wadhwa)

Political leaders, tech executives, and academics often claim that the U.S. is falling behind in math and science education. They cite poor test results, declining international rankings, and decreasing enrollment in the hard sciences. They urge us to improve our education system and to graduate more engineers and scientists to keep pace with countries such as India and China.

Yet a new report by the Urban Institute, a nonpartisan think tank, tells a different story. The report disproves many confident pronouncements about the alleged weaknesses and failures of the U.S. education system. This data will certainly be examined by both sides in the debate over highly skilled workers and immigration (BusinessWeek.com, 10/10/07). The argument by Microsoft (MSFT), Google (GOOG), Intel (INTC), and others is that there are not enough tech workers in the U.S.

The authors of the report, the Urban Institute's Hal Salzman and Georgetown University professor Lindsay Lowell, show that math, science, and reading test scores at the primary and secondary level have increased over the past two decades, and U.S. students are now close to the top of international rankings. Perhaps just as surprising, the report finds that our education system actually produces more science and engineering graduates than the market demands....

... As far as our workforce is concerned, the new report showed that from 1985 to 2000 about 435,000 U.S. citizens and permanent residents a year graduated with bachelor's, master's, and doctoral degrees in science and engineering. Over the same period, there were about 150,000 jobs added annually to the science and engineering workforce. These numbers don't include those retiring or leaving a profession but do indicate the size of the available talent pool. It seems that nearly two-thirds of bachelor's graduates and about a third of master's graduates take jobs in fields other than science and engineering...

So the data suggests we actually graduate more scientists and engineers than we have jobs for. Encouraging science education isn't going to make more scientists, any more than encouraging farming education will make more American farmers.

There's better paid work for smart American students in other domains.

Which brings us back to the farming analogy. The US is a post-agricultural nation that dose agriculture as an expensive hobby. Are we a post-science nation too?

BTW, I'm so pleased someone has done the research on this. I love to have my intuitions confirmed ...

On saving the world - Shtetl-Optimized

I've had a post bouncing around my head for a while. It's about saving the human world. (The rest of the world will do just fine - eventually. As my 8 yo says, history just keeps happening.)

I'm going to write that post - eventually. I'll try to write the main risks down (US-China conflicts, WMDs, cost of havoc, rapid environmental collapse and resulting socioeconomic disruptions, artificial minds [1], etc) and what a geek can do about them in the age of O'Reilly.

In the meantime, a post by Scott Aaronson (yes, two t, two a)...

Shtetl-Optimized » Blog Archive » Procrastinating on the sidelines of history

... So, Al Gore. Look, I don’t think it reflects any credit on him to have joined such distinguished pacifists as Henry Kissinger and Yasser Arafat. I think it reflects credit on the prize itself. This is one of the most inspired choices a Nobel Peace Prize committee ever made, even though ironically it has nothing directly to do with peace.

With the release of An Inconvenient Truth and The Assault on Reason, it’s become increasingly apparent that Gore is the tragic hero of our age: a Lisa among Cletuses, a Jeffersonian rationalist in the age of Coulter and O’Reilly. If I haven’t said so more often on this blog, it’s simply because the mention of Gore brings up such painful memories for me.

In the weeks leading up to the 2000 US election, I could almost feel the multiverse splitting into two branches of roughly equal amplitude that would never again interact. In both branches, our civilization would continue racing into an abyss, the difference being that in one branch we’d be tapping the brakes while in the other we’d be slamming the accelerator. I knew that the election would come down to Florida and one or two other swing states, that the margin in those states would be razor-thin (of course no one could’ve predicted how thin), and that, in contrast to every other election I’d lived through, in this one every horseshoe and butterfly would make a difference. I knew that if Bush got in, I’d carry a burden of guilt the rest of my life for not having done more to prevent it.

The question was, what could a 19-year-old grad student at Berkeley do with that knowledge? How could I round up tens of thousands of extra Gore votes, and thereby seize what might be my only chance in life to change the course of history? I quickly ruled out trying to convince Bush voters, assuming them beyond persuasion. (I later found out I was wrong, when I met people who’d voted for Bush in 2000 but said they now regretted their decision. To me, it was as if they’d just noticed the blueness of the sky.)...

...In the end, though, the Nadertrading movement simply failed to reach enough of its target audience. The websites put up by me and others apparently induced at least 1,400 Nader supporters in Florida to vote for Gore — but 97,000 Floridians still voted for Nader. And as we know, Bush ended up “winning” the state by 537 votes...

Ah yes. Nader. There are no words.

But. We're not alone. Not completely.

[1] Even I wince when I write that. It's just so geeky. Tough.

Fragments of history: Manchester air raid shelters in abandoned mine shafts

My mother was a girl in Manchester England during WW II. Today, after my usual stories of what the children were up to, she mentioned that as a child, early in WW II, she enjoyed the family time in the backyard bomb shelter. These were hand dug pits with concrete bottoms, metal siding and roofs, and dirt on top. It's not clear to me how much protection these things provided, but I imagine the psychological benefit was significant.

We spoke a bit of general WW II shelter design, and she mentioned that among the Manchester shelters were abandoned mine shafts. Coal mine shafts in particular. These were group shelters, a step below subways and the like.

It seems to me there ought to be some stories about taking shelter in an abandoned coal mine. Pleasant no, memorable yes. The search terms should be specific enough to find something.

Alas manchester "air raid" shelter coal "mine shaft" didn't come up with much today, though it did catch another story of a lost world.

In a day or two, of course, the search will find this post.

Sheila Cox says hello.

1940 was 67 years ago. Getting on half way to the American Civil War.

Imagine someone, or something, searching on this topic 70 years from now. It's rather hard for me to imagine what they might be like.

Financial market turmoil: The problem is obvious, the fix is hard

The current explanation of why the trillion dollar securities market is in turmoil, which may push us into recession, is that mortgage-backed security pricing was inaccurate because credit-rating agencies didn't do their job.

Gee, I wonder why they didn't do their job? It's a mystery. Needs lots of investigation.

Or not ....

Robert Reich's Blog: Why Credit-rating Agencies Blew It: Mystery Solved

Recently, Treasury Secretary Hank Paulson sharply criticized credit-rating agencies for failing to recognize the risks in hundreds of billions worth of mortgage-backed securities whose values continue to plummet as home-loan defaults grow.

... obvious why credit-rating agencies didn't blow the whistle. (They didn't blow the whistle on Enron on Worldcom before those entities collapsed, either.) You see, credit-rating agencies are paid by the same institutions that package and sell the securities the agencies are rating...

As I've noted in other contexts, it's not necessary for anyone at the credit-rating agencies to actually conspire to create false results. It's simply emergent behavior -- natural selection in action. Competent honest people will fail to make their incentives, so they'll either become incompetent, move to other businesses, or become dishonest. It probably only takes 1-2 years of this type of conflict of interest to create agencies that are a mixture of incompetent and dishonest.

So the interesting question is not why the credit-rating agencies are dishonest and incompetent. That's obvious.

The interesting question is why Congress never fixed this back in the Enron days. The division was proposed then, probably by Robert Reich but also by the usual suspects (Krugman, DeLong, etc).

Oh wait, we know the answer to that one too. Congress is incented by reelection and retirement jobs, which are paid for by the .. the credit-rating agencies and securities company ...

Hmm. So why does Congress get away with such incompetency and corruption ...

Oh wait, we know the answer to that one too.

The newspapers don't write about this and the talk shows and television news groups don't educate the public.

I'll leave the rest of the "Oh, waits" to the reader.

The problem is the American voter.

My answer to Jeff Atwood's question: Why Does Software Spoil?

Jeff Atwood writes Coding Horror, one of my favorite technical blogs. In two posts he asks familiar questions (remember, he's from the XP world): Coding Horror: Why Does Software Spoil? and "Are Features the Enemy".

I think the answer is pretty straightforward. The way we pay for software induces perverse incentives. Here's the comment I wrote to his blog:

It's the business model.

If we rented software then there'd be a steady revenue stream for developers, and a lesser feature incentive.

Since we buy software (sort of) developers have perverse incentives. They have to "break" the prior version by not maintaining it, introducing new file formats, and they have to add features to hide the fact that the updates are driven by vandalism, not value.

Of course software rental completely screws customers unless the file/data  formats are completely public and interoperability is assured. I'm sure customers will realize how important that is.

Thirty years from now.

Viruses and worms now help drive this model. Security issues make it much easier for companies that own both the OS and the software products to "break" older software, thereby forcing updates. Feature addition is simply a way to make customers accept the forced update. If Microsoft didn't add features, but forced Office updates by making old versions run less well, customers would scream.

These kinds of things don't really have to be planned. They're simply emergent. It's just the way natural selection operates on business.

Of course the only thing worse than the current business model is the combination of software leasing and proprietary data formats, including proprietary metadata models. (Ever try extracting and moving all your iPhoto data?)

Happily customers would never fall for that trick.

Oh, wait ...

Sibling Neandertal and the end of the modern human

I asked the other day "what was Homo sapiens doing for 115,000 years?"

I didn't expect John Hawks to answer me so quickly (emphases mine):

... we have undergone light-years of change since the last Neandertals lived. This is not a question of "modern human origins" anymore. We can now show that living people are much more different from early modern humans than any differences between Neandertals and other contemporary peoples. I think that "modern humans" is on its way to obsolescence. What matters is the pattern of change across all populations. Possibly that pattern was initiated by changes in one region but the subsequent changes were so vast that the beginning point hardly matters... [From John Hawks Anthropology Weblog : 2007 10]

Most of the popular books on human origins I've read emphasize how similar we are to Homo sapiens of 60,000 years ago. It appears that meme is on its last legs. We look a lot like our ancestors, but our minds are very different.

To answer my original question -- Home sapiens spent the last 115,000 years turning into an animal that could write. It wasn't easy.

It's tempting to suggest we need to name a new human "species" that was launched 15,000 years ago, but these arbitrary demarcations are increasingly unconvincing, just as misleading as the concept of "the modern human".

The more we learn about the evolution of the human mind the more fluid it seems. Our modern world has many more niches for exotic minds than the ancient world; if we live long enough we'll fill those niches.

One day we might need interlocutors -- even between speakers of the same language.

The rest of Hawks article is well worth a close read. He uses a recent discovery from the Neandertal genome to support a favorite thesis of his -- that we are part Neandertal.