NSA traffic analysis: the seduction of data and the creation of conspiracy

I heard a part of the senate hearings on the NSA intercepts today. The GOP Senator was lobbing softball questions at the Bush attorney, who answered them well and carefully. Of course none of the questions were about the more interesting issue -- how the intercepts are selected and whether that process is legal. Cringely, who's done some good summaries to date, adds a bit more to the picture:

PBS | I, Cringely . February 2, 2006

... last thought comes from an old friend of mine who is conservative in the very best sense and knows what he is writing about:

"Traffic analysis, at the NSA? I'm tempted to be sarcastic, but I won't be. As you might know, I started a company a few years ago with a former NSA guy -- somebody who was a cryptographer and Russian linguist on those submarines that snuck into Soviet harbors to tap their phone lines -- and we applied traffic analysis to Internet discussion groups to identify opinion leaders, conversation trends and so forth. We used a lot of techniques that were developed or applied to law enforcement. And we didn't use anything that violated anybody's security clearances... really!

"(My company) was acquired by a business intelligence company funded by the CIA venture capital outfit. Apparently the stuff I invented is now in the hands of a couple of intelligence agencies, including Homeland Security.

"I'll tell you what I think the most troubling thing about all this is. It's easy to see whatever pattern you're looking for. It's like curve fitting in the stock market -- looks beautiful historically and maybe even in the short run, but it's a disaster in the making. So we have these guys running the country who saw a non-existent pattern in Iraq that justified a war ... and now we're going to give them software that will make it easy to create the illusion of patterns of conspiracy.

"Your friend from the NSA was right, but it's worse than he suggests. It's not just that social network analysis casts a wide net. It's that without oversight by people who really grasp the mathematics and have some distance from the whole thing, they're going to see patterns where there aren't any.

"They have a history of that."

The history of lie detector testing is informative. Current lie detector technology is very unreliable. It falsely implicates and falsely absolves. It's only somewhat better than a random guess, and some observers do better without the technology. Nonetheless, it is immensely abused by law enforcement. We can expect the NSA, FBI, and CIA to make the same mistakes with traffic monitoring technologies. Geeks don't get promoted in these agencies, and only geeks understand the limitations of this technology.

Bush's apparatchik in NASA: obnoxious youth

A few weeks ago we read that some NASA 'public affairs' officer had been attempting to silence of NASA's senior climate researches. Apparently Dr. Hansen wasn't following the part line; he needed a handler present lest he talk too honestly to the press.

What we didn't hear, and what's buried at the back of recent NYT article, is a feature of that apparatchik that made him particularly annoying -- he's a kid, a presidential appointee with zero credentials. A classic Soviet era apparatchik...

NASA Chief Backs Agency Openness - New York Times

The Big Bang memo came from Mr. Deutsch, a 24-year-old presidential appointee in the press office at NASA headquarters whose résumé says he was an intern in the "war room" of the 2004 Bush-Cheney re-election campaign. A 2003 journalism graduate of Texas A&M, he was also the public-affairs officer who sought more control over Dr. Hansen's public statements.

In October 2005, Mr. Deutsch sent an e-mail message to Flint Wild, a NASA contractor working on a set of Web presentations about Einstein for middle-school students. The message said the word "theory" needed to be added after every mention of the Big Bang.

The Big Bang is "not proven fact; it is opinion," Mr. Deutsch wrote, adding, "It is not NASA's place, nor should it be to make a declaration such as this about the existence of the universe that discounts intelligent design by a creator."

It continued: "This is more than a science issue, it is a religious issue. And I would hate to think that young people would only be getting one-half of this debate from NASA. That would mean we had failed to properly educate the very people who rely on us for factual information the most.

... Mr. Deutsch did not respond to e-mail or phone messages. On Friday evening, repeated queries were made to the White House about how a young presidential appointee with no science background came to be supervising Web presentations on cosmology and interview requests to senior NASA scientists.

Bush's team appoints a flunkee to be their apparatchik for NASA. Said flunky starts drafting very annoying memos, implying dire consequences to those who disobeys. He's particularly concerned about affirming intelligent design and blocking talk of global climate change. An interesting story. So why did it get buried at the little read end of a NYT article?

The more I see Bush in action, the more convinced I am that he's a KGB implementation of 'The Manchurian Candidate'. It was a rogue mission by a Putin clique in the KGB, and it was thought to have failed. Their vehicle was far gone in alcohol and drug abuse. Miraculously he turns his life around and becomes president. Putin is astounded but he activates the original programming. Bush begins to transform the US into a satellite of the Soviet Union ...

How amantadine became useless

Symmetrel (amantadine) and Flumadine (rimantadine), have lost their value against this year's strain and should be shelved. Ok, but why?

That's the interesting question, and most of the media reports I've seen don't address it. NPR did, however. It turns out that over the counter cold remedies in China and Russia often contain amantadine. Sigh. That's so sad, and so stupid. Very human.

Of course the same thing is true of antimicrobials, they are widely available over the counter in many nations. The difference is that pathogenic bacteria don't travel nearly as quickly as flu viruses. The CDC expected we'd get a few more years of life out of amantadine/rimantadine, but resistance spread more quickly than expected.

The ACLU's estimate of NSA activity

So what's the NSA up to that Bush couldn't use the FISA courts? The current consensus is that the most controversial things they're doing are:

1. Using messaging metadata and algorithmic analysis to target individuals for wiretaps that would otherwise not be identified.
2. Applying wiretaps to the individuals identified via #1 based on evidence (algorithmic ratings) that, by itself, would be very insufficient to justify a wiretap.

If you listen to what Bush says, he focuses on the intercepts, not on the legality of how the intercepts were selected.

The upshot of this is they're probably monitoring a lot of journalists, and a lot of family members of the true targets. They may learn interesting things in the monitoring of journalists, such as the identity of anyone in the government, NSA, or CIA who's blabbing about the NSA's programs.

The ACLU fills in the details:

American Civil Liberties Union : Eavesdropping 101: What Can The NSA Do?

Data mining is a broad dragnet. Instead of targeting you because you once received a telephone call from a person who received a telephone call from a person who is a suspected terrorist, you might be targeted because the NSA's computers have analyzed your communications and have determined that they contain certain words or word combinations, addressing information, or other factors with a frequency that deviates from the average, and which they have decided might be an indication of suspiciousness. The NSA has no prior reason to suspect you, and you are in no way tied to any other suspicious individuals %u2013 you have just been plucked out of the crowd by a computer algorithm's analysis of your behavior.

If we don't put a stop to this, we will pay a very high price. I really do believe that, at the moment, the Bush administration is a greater threat to our future than al Qaeda (in large part, of course, because the non-Iraq part of the 'War on Terror' did make sense and al Qaeda appears to be both weak and to have a very thin bench team).

Bush on cutting imports: really, it was a joke

Rarely does the leed of an newspaper article cut so deeply (via Shrillblog):

KR Washington Bureau | 02/01/2006 | Administration backs off Bush's vow to reduce Mideast oil imports

WASHINGTON - One day after President Bush vowed to reduce America's dependence on Middle East oil by cutting imports from there 75 percent by 2025, his energy secretary and national economic adviser said Wednesday that the president didn't mean it literally...

Bush's people feel they've proved beyond a reasonable doubt that 20% [1] of the voting American public are blithering idiots. Maybe they give him lines like these to test the resilience of this hard-core non-sentience. Perhaps they're trying to tell us "stop me before I kill again".

[1] Bush's approval rating is usually about 42%. About half of his hard core base feel he is serving messianic duties. I don't agree with that, but if that's one's belief then it's not idiotic to support him. That leaves my 20% estimate.

How often will sentience evolve on earth? From anthropology to the Fermi Pardox - via the Drake Equation

Hawks reviews research that suggests that "modern" evolutionary innovations are more likely to be repeatable (non-singular) than "ancient" innovations. He connects this to the Drake equation, the companion to the Fermi Paradox that attempts to estimate the prevalence of technological civilizations in our galaxy. He points out that since the only instance of sentience we know of is quite recent, it is likely that sentience is not a particularly singular innovation.

Personally, I'd bet we're not the first. Stephen Baxter wrote an immensely underappreciated science fiction novel (Evolution) that painted a rather persuasive picture of how sentience might come and go across the history of life on earth -- only once and briefly producing interplanetary technology (after this transiently spacefaring species passes, there's not much left for others to work from).

Hawks is inclined to think we're the first, but most likely not the last. Indeed, if we live out this century, I think it's likely we'll create other sentiences, both biological and otherwise. If we don't make it, the biological ones will still emerge some day, some place. Maybe they'll do a better job that us.

Back to the Drake Equation. The more we start to shift our estimates for the terms of the Drake Equation, the more the Omega term, L, looms larger (sorry). This term is often estimated based on the Fermi Paradox. Wikipedia (currently) has an excellent discussion of this relationship:

The remarkable thing about the Drake equation is that by plugging in apparently fairly plausible values for each of the parameters above, the resultant expectant value of N is generally often >> 1. This has provided considerable motivation for the SETI movement. However, this conflicts with the currently observed value of N = 1 — one observed civilization in the entire universe. Other assumptions give values of N that are <<>

This conflict is often called the Fermi paradox, after Enrico Fermi who first publicised the subject, and suggests that our understanding of what is a "conservative" value for some of the parameters may be overly optimistic or that some other factor is involved to suppress the development of intelligent space-faring life...

... L = the expected lifetime of such a civilization

Estimated by Drake as 10 years.

The value of L can be estimated from the lifetime of our current civilization from the advent of radio astronomy in 1938 (dated from Grote Reber's parabolic dish radio telescope) to the current date. In 2005, this gives an L of 67 years.

In an article in Scientific American, Michael Shermer estimated L as 420 years, based on compiling the durations of sixty historical civilizations. Using twenty-eight civilizations more recent than the Roman Empire he calculates a figure of 304 years for "modern" civilizations. Note, however, that the fall of most of these civilizations did not destroy their technology, and they were succeeded by later civilizations which carried on those technologies, so Shermer's estimates should be regarded as pessimistic.

The Wikipedia article estimates a low value for "safe" earth like planets, I read the most recent findings as much more encouraging but I'm far out of my expertise range.

If "safe" planets turn out not to be rare, then we're back to Drake's solution to the Fermi Paradox -- a 10 year lifespan for a technological civilization. My bet is that the small number is not so much L, as it is fc*L, so even if L is not short something happens to technological civilizations that causes them to lose interest in both physical exploration and communication with the likes of us. Something that produces a communicative sentience for no more than 10-40 years.

The passing of the Telegram

Western Union no longer sends telegrams. If I'd known they still sent them, I'd have tried to send one just for history's sake. Typewriters, telegrams, carbon paper ... these are some of the things I used to know.

The secret to better crime results: don't record the data

The St. Paul Police department is practising the time-honored method of improving one's results -- don't record troublesome data points. It works as well for crime measurement as it does for Texas schools.

In this case some low life smashed our rear van window -- in bright daylight outside our local library. When they were called, the police said they don't do police reports on this kind of crime.

What is not measured -- did not happen. It works for Bush, and it works for the local police.

Ampulex and the cockroach slave: another Zimmer zinger

Karl Zimmer is a great help to those on diets. Read his articles just prior to lunch.

The Wisdom of Parasites. The Loom: A blog about life, past and future

... As an adult, Ampulex compressa seems like your normal wasp, buzzing about and mating. But things get weird when it's time for a female to lay an egg. She finds a cockroach to make her egg's host, and proceeds to deliver two precise stings. The first she delivers to the roach's mid-section, causing its front legs [to] buckle. The brief paralysis caused by the first sting gives the wasp the luxury of time to deliver a more precise sting to the head.

The wasp slips her stinger through the roach's exoskeleton and directly into the cockroach's brain. She apparently using sensors along the sides of the stinger to guide it through the brain, a bit like a surgeon snaking his way to an appendix with a laparoscope. She continues to probe the roach's brain until she reaches one particular spot that appears to control the escape reflex. She injects a second venom that influences these neurons in such a way that the escape reflex disappears.

From the outside, the effect is surreal. The wasp does not paralyze the cockroach. In fact, the roach is able to lift up its front legs again and walk. But now it cannot move of its own accord. The wasp takes hold of one of the roach's antennae and leads it--in the words of Israeli scientists who study Ampulex--like a dog on a leash...

We have insufficient respect for the Emergent Designer (ED) -- the pseudo-deity of evolution. The ED is not a cuddly sort. Its humor is bleak indeed.

How many human psychiatric disorders (think OCD) are the result of adaptive mutations that compensate for a parasitic influence, but subsequently become disorders when the parasite is absent?

Google's feet of clay: Gmail and spam

Google's share price had a minor hit the other day when they "disappointed" on earnings. I can't make sense of their valuation, even though I do think they're a great company.

I can, however, point out that one of their flagship products, Gmail, has serious issues. For historical reasons I get to see how five different spam filtering systems work: Yahoo, Earthlink, Spamcop, the open source systems used by many smaller ISPs, and Gmail's system.

Gmail is not just slightly inferior. It is qualitatively inferior. It is so bad it's mindboggling. The other four all work quite well, making relatively few false positive or false negative errors. Gmail errs in both directions, misclassifying spam as mail and mail as spam.

This isn't new. They've had the same problem for over a year. The only reason I stick with them is their fantastic UI and amazing search capabilities, but if Yahoo ever updates me to their new UI I may switch (I can redirect my mail flows fairly easily since I control the routing domains).

Why doesn't Google invest in the open source systems that work for everyone else? The scale they work on is rather different from that of a small ISP, so they may face impossible scalability challenges. I wonder though, if arrogance plays a role -- the belief that their algorithms will devise a better solution. If it's really arrogance, then their share price may fall more than 10% over the next year.

The best commentary on the state of the union address

Fafblog is marvelous ...

Fafblog! The State of the Universe Address

...Would you take all that away by letting isolationist courts tax the Jesus fetus? Never! Because history is written in courage, and courage will remember us in the future how we were today: pandering, desperate and barely coherent!

My prediction on the Bush health care plan ...

The smart Republican (they're all dead now, but they once existed) thing to do would be to eliminate the corporate tax break for health insurance and apply the funds to a mixture of subsidies and individual tax breaks. That would be a Reagan type of move -- the guy who brought us the Earned Income Tax Credit. (I used to think Reagan was an idiot. True, he was demented during his second term, but compared to Bush II he was a brilliant pillar of light.)

Instead, Bush the Incoherent will introduce a meaningless tax credit that will complicate the tax code, transfer wealth from the Weak to the Strong, worsen the deficit, and have no material impact on healthcare.

The New York Times finds a spine?

It won't last, but today the shrunken residue of a once great newspaper shows a tiny hint of a vertebral column:

Spies, Lies and Wiretaps - New York Times

... Mr. Bush made himself the judge of the proper balance between national security and Americans' rights, between the law and presidential power. He wants Americans to accept, on faith, that he is doing it right. But even if the United States had a government based on the good character of elected officials rather than law, Mr. Bush would not have earned that kind of trust. The domestic spying program is part of a well-established pattern: when Mr. Bush doesn't like the rules, he just changes them, as he has done for the detention and treatment of prisoners and has threatened to do in other areas, like the confirmation of his judicial nominees. He has consistently shown a lack of regard for privacy, civil liberties and judicial due process in claiming his sweeping powers. The founders of our country created the system of checks and balances to avert just this sort of imperial arrogance.

They've done this before -- threatened to show some courage. In each case they've collapsed into equivocation.

If the NYT dedicates itself to exposing Bush/Cheney, and limiting the damage they're doing, and if they pound home time and again the fundamental issues without being misdirected -- then I'll say they have a spine. Until then, they're spineless.

Idiot America

David Brin led on this one, but Pharyngula is taking up the flag: Idiot America.

I don't think it's just America, though we've led the way in the western world. The rejection of the enlightenment, and the ridicule of expertise is pretty much universal -- from Washington to Tehran to Moscow. Beijing possibly being the exception.

Too bad the original Esquire article is behing a Paywall, but Pharyngula has excerpted the juicy parts. I hope he connects up with David Brin.

A creek in the congo: Future Shock

Sometimes I think the world isn't changing very quickly. Mostly 2006 seems much more like 1986 than I'd expected back then.

But then I enter our dog's name (Kateva) and come up with a map of a creek in the Congo and a link to a Google Earth image. This is the result of 3 separate web services.

Suddenly 2006 seems much less familiar.

DeLong on Franco

Odd that we should be thinking of Francesco Franco these days. DeLong ends a posting on Spain's tyrant with a guide to spotting fascism:

Brad DeLong's Semi-Daily Journal

German philospher Ernst Nolte's classic Fascism in Its Epoch set out six key characteristics of fascism:

1. Strong belief that--through social darwinism--morality is ultimately tied to blood and race, understood as descent and genetic relationship.

2. Strong rejection of the classical "liberal" belief that individuals have rights that any legitimate state is bound to respect

3. In its place, an assertion that individuals have duties to the state, seen as the decision-making organ of the collectivity.

4. A rejection of parliamentary democracy and other bottom-up institutions to assess the general will.

5. The assertion that the general will is formed by the decrees of the leader.

6. A strong fear of twentieth-century Communism, and an eagerness to adapt and use its weapons--suspension of parliaments, mass propaganda, rallies, street violence, and so forth--to fight it.

In China, it is common to criticize the regime by allusions to historical figures. DeLong was posting on Chinese New Years. Coincidence or subtlety?