Data mining and you: giving a damn might be wise

I do get bored returning to the same old story but I think it's a social obligation. More on how the FBI does data mining and why you should care ...(emphases mine)

Civil liberties: surveillance and privacy | Learning to live with Big Brother | Economist.com

... Two days after the attacks on New York and Washington, Frank Asher, a drug dealer turned technology entrepreneur, decided to examine the data amassed on 450m people by his private data-service company, Seisint, to see if he could identify possible terrorists. After giving each person a risk score based on name, religion, travel history, reading preferences and so on, Mr Asher came up with a list of 1,200 “suspicious” individuals, which he handed to the FBI. Unknown to him, five of the terrorist hijackers were on his list.

The FBI was impressed. Rebranded the Multistate Anti-Terrorism Information Exchange, or Matrix, Mr Asher's programme, now taken over by the FBI, could soon access 20 billion pieces of information, all of them churned and sorted and analysed to predict who might one day turn into a terrorist. A new version, called the System to Assess Risk, or STAR, has just been launched using information drawn from both private and public databases. As most of the data have already been disclosed to third parties—airline tickets, job records, car rentals and the like—they are not covered by the American constitution's Fourth Amendment, so no court warrant is required.

In an age of global terror, when governments are desperately trying to pre-empt future attacks, such profiling has become a favourite tool. But although it can predict the behaviour of large groups, this technique is “incredibly inaccurate” when it comes to individuals, says Simon Wessely, a professor of psychiatry at King's College London. Bruce Schneier, an American security guru, agrees. Mining vast amounts of data for well-established behaviour patterns, such as credit-card fraud, works very well, he says. But it is “extraordinarily unreliable” when sniffing out terrorist plots, which are uncommon and rarely have a well-defined profile.

By way of example, Mr Schneier points to the Automated Targeting System, operated by the American Customs and Border Protection, which assigns a terrorist risk-assessment score to anyone entering or leaving the United States. In 2005 some 431m people were processed. Assuming an unrealistically accurate model able to identify terrorists (and innocent people) with 99.9% accuracy, that means some 431,000 false alarms annually, all of which presumably need checking. Given the unreliability of passenger data, the real number is likely to be far higher, he says.

Those caught up in terrorist-profiling systems are not allowed to know their scores or challenge the data. Yet their profiles, which may be shared with federal, state and even foreign governments, could damage their chances of getting a state job, a student grant, a public contract or a visa. It could even prevent them from ever being able to fly again....

Ok, you can go back to sleep now.

Your "do not call" registration is expiring!

Follow Me Here reminds us we need to renew our "do not call" registration. It's hard to remember, but once upon a time telemarketers called us night and day. Then Congress passed a law that, astonishingly, worked!

I registered the first day the opt-out program became available, so I had to re-register today. You can do 3 phone calls at once, you'll get 3 confirmation emails and you need to click on each one.

Politicians and non-profits can still call, we do get those. I tell the non-profits that if they call again they'll never get a penny from me, but my wife is more forgiving. They always ask for her ...

Medical Algorithms: a web site devoted to "depression scales" and the like

I've long thought it would be interesting to have a single source for all the scales, instruments, algorithms and the like one finds in medical journals. Someone has built this source:

Medical Algorithms

...More than 11,000 Scales, Tools, Assessments, Scoring Systems, and other Algorithms useful in Medicine and Biomedical Research...

I wonder if they have any copyright issues, but they may be covered by fair use as long as they don't try to make money form the content. Most of these scales are, unfortunately, owned by the journal publishers.

Losing the roots: the Dems may have to chance a spine

I'm not yet as disgusted with the Dems as Greenwald is. Judging by the past ten years it's dangerous to assume we, the People, are thinking rationally about anything. Politicians need to, every day, calculate how rational the voter is likely to be.

On the other hand, at some point a too craven Congress risks alienating their core supporters:

Glenn Greenwald - Political Blogs and Opinions - Salon

... If the Democratic Congress capitulates yet again, there will be plenty of time and opportunity for all sorts of recriminations. I think it is quite encouraging that much of the 'netroots' is now devoting its energies and resources not to supporting Democrats, but to opposing Congressional Democrats who merit defeat.

...Matt Stoller and Open Left, for instance, are devoting most of their energies to figuring out how to surmount the obstacles to waging effective primary challenges against Bush-supporting Democrats. The fund-raising entity run by FDL, C&L and others has begun targeting worthless Democrats, funding and running robocalls against Bush-enabling Democratic incumbents in their districts (those inclined can help fund those efforts here). MoveOn is actively considering spending large sums of money to support primary challenges against war-enabling Democrats. Obviously, there is no point in working to empower Democrats who enable and support virtually all of the worst aspects of the Bush agenda...

It's a tough balancing act. I believe that the American brain is still in intensive care; the patient should be moved too quickly. Still, the loss of the core, and the inevitable return of Nader [1], requires some calculated risks. Congress may have to start resisting -- even though the American people aren't quite ready.

I wish we were a healthier People, but we're not.

[1] I think George Bush and Ralph Nader have a great deal in common. Neither has much of a learning curve, both are convinced of their righteousness. A just world would condemn them to share a house for the rest of their natural lives.

Netflix: are broken DVDs a sign of stress?

Netflix is not cheap, but it worked well for our family -- for a year or so.

Since the kids don't watch TV (much less cable), each child gets to pick one DVD a week. Some of the DVDs are of children's TV shows, I'm fond of Jimmy Neutron myself. Fred Flintstone is still funny, and now has added sociologic interest.

We receive about 10 DVDs a month on average. Recently, at least one of the 10 will be unplayable.

Ten percent is not an acceptable failure rate, though I'm sure it's significantly lower for adult DVDs. It used to be under 5% for the children's DVDs, so something is going wrong.

NFLX share price is pretty steady, and that's bad news for a growth company. I bet the CEO is feeling some options pain. I suspect they're cutting costs, probably by extending the lifespan of the circulating DVDs.

I'm not impressed with their no-Mac "free" electronic distribution, but I can live with that - for now. I can't be bothered phoning Netflix about a broken DVD once a month. The next time we call, we're going to make them an offer: we get a free month, or we see what Blockbusters failure rate is.

Bill Clinton as Hillary's attack dog

Every candidate needs someone who can rip the throat out of the opposition. Hillary has Bill, who recently demonstrated a throatectomy on CNN/YouTube. The target was the GOP's feigned MoveOn "shock and horror".

He's good, he's very good. I suspect he can turn the outrage on at will, so there's an odd sincerity to his performance.

The Economist this month proclaimed Hilary president-in-waiting. I don't think they're really conspiring against her with the ridiculously premature proclamation, I suspect they're trying to prepare their increasingly conservative readers for a future Hilary endorsement.

I don't share The Economist's supposed certainty, but watching Bill at work it's possible to imagine the GOP's snakes deciding to save their venom for softer prey. They've fanged Hilary so often that she's probably immune by now, and their incessant attacks may be boring their partisans. It will be nicely ironic if the GOP has stoked Hillary outrage so long and hard that, at the time they most need their rabid partisans, they've all become bored with the meme.

No bad bridge maps: Bridge ratings are now top secret

After our local bridge fell, I wondered if Google could add a "bridge rating" icon to their map layers:

Gordon's Notes: Bridges: 77,000 deficient, 750 have I-35W design

... I thought a bit more about how Google could accelerate the infrastructure review. A 'route around risky bridge' option for Google Map directions would concentrate minds wonderfully. One can readily imagine icons for bridges with the I-35W design and risk designation...

Recently, our Republican governor managed to have all MN bridge ratings declared "top secret". The ratings could help the terrorists, you see.

I have to hand it to Pawlenty. He waited a few weeks to give the public time to completely forget about the bridge, then he made it impossible for anyone to track the state of MN bridges.

That should help block a gasoline tax. Our local media have been playing dead of course, they know almost no-one is paying attention.

Smooth.

We've identified the problem with the US media

A research breakthrough reveals the true cause of the dysfunctional US media and, indirectly, the reelection of George W. Bush:

Britney Spears sucks Blackwater / Showdown! White-trash diva versus mercenary war thugs. Who rapes your spirit more?

... the Blackwater story is indeed fascinating and brutal and bespeaks of a particularly nasty, largely unreported aspect of Bush's bleak war machine, despite how only a handful of (very serious) readers bothered to click on that story to read beyond the headline as it sat atop the SFGate home page for much of the day, craving attention.

But then there's this: The moment the "Britney loses custody of her kids" story went up on our site, the mad swarm of clicks it instantly received made the Blackwater story seem like, well, like a Home&Garden feature on do-it-yourself mulch. Which is to say, the damnable thing was so insanely popular, so flooded with clicks, it nearly crashed our servers...

Once upon a time the media was an elitist enterprise. Now the media is customer-driven.

The customer is us.

Problem.

Fair Use defined

Coding Horror has been one of my favorite blogs for many months, but lately he's hitting new heights. Almost every post is worth a comment, I've a bunch stacked up now.

His post on Copyright, Fair Use and YouTube is a classic. On the one hand he says aloud what everyone knows but speaks not: YouTube's entire value proposition comes from breaking copyright law.

One take away is that either the law should be changed, YouTube should reform (come up with a revenue sharing model), or YouTube should go away.

My other take away is to try to memorize the following:

... It's completely glossed over on the YouTube copyright page in favor of 100% original content, but the loophole in copyright is fair use. Under the banner of fair use, you could legally upload a video without the copyright holder's permission. Anyone who contributes anything to the web should have the four factors of fair use committed to memory by now:

1. the purpose of the use [commentary, research, parody vs... theft]
2. the nature of the copyrighted work [esp. whether it's a public good]
3. the relative amount of the portion used
4. the market effect of the use on the copyrighted work

Why does Apple have a better quality reputation than Porsche?

I created a trivial Automator script today. It's supposed to add any images added to a folder to iPhoto, it's part of setting up a scanning work environment for my 8 yo. He wants more another Lego model, so he needs the money.

I opened the script to edit it, while noting that OS X doesn't provide an intuitive way to edit or remove Automator scripts -- one must read the manual.

Automator crashed.

Sigh.

I wasn't surprised. Whenever I touch parts of OS X that aren't routinely used by millions I find big chunks of dysfunction. OS X Services are another large example, though they tend to simply be unsupported.

XP and Office are no different of course, but Microsoft doesn't have a reputation for quality. Strangely, Apple does.

So, given Apple's Quality Problems how do they keep their rep? It's long puzzled me. Apple is inventive and elegant, but not reliable. Sort of like Porsche, but Porsche doesn't have a reputation for reliability -- quite the opposite.

Perhaps it's just a matter of time before reality settles in. Personally, I'd trade a significant chunk of elegance for more reliability, but no Product Manager is going to build anything for my "N of One" market.

Really, most people would be better off if I ran the world ...

Update 10/8/07: More on Automator abandonware. I like the term "abandonware" to describe OS X services, AppleScript and Automator.

Star Wars: A weird and dark story

I have seen too much Star Wars. I'd seen every movie once, before our children became fans. I've now seen episodes I, IV through VI, and the two "Clone Wars" animations many times. The children are not allowed to see episodes II and III, so my eldest is obsessed with them. He studies related picture books intensely.

I am, then, an expert in the story the movies tell. I can thus confidently declare that the story is deeply weird, dark, and disturbing. Do not be deceived by the words, observe the actions.

The Jedi enslave clones and droids alike, tear apart families, abandon their failures, never show remorse or admit error, treat non-Jedi as pawns, and deny the bonds of love. They are deeply flawed. The Sith aren't much better, thought they at least know themselves.

R2D2 knows far too much, he's involved in every intrigue. C3PO has his memory wiped (another crime), but nobody remembers the harmless little tin can that chirps and beeps. R2D2 must have learned everything C3PO knew, and everything the cunning droid has been personally involved in. He can penetrate and control any AI system, even that of the Death Star. He's hundreds, if not thousands, of years old.

The case for R2D2's power is irrefutable. Taking the movies at face value, any neutral observer would have to wonder if the real manipulator is R2D2, not Palpatine and Yoda.

Luke forgets the family that raised him with barely a second thought, is tempted by the Sith, and joins the morally bankrupt Jedi. Chewbacca never tells Han what he knows about Yoda, and pretends to know nothing about the Jedi. Leia's world is destroyed, but she shrugs it off as an acceptable sacrifice. She's perhaps not quite as flawed as her brother, but, really, it's only a matter of time.

Anakin is a child who becomes a sociopathic killer (episode II) -- long before he joins the Sith. It's a horrid story; it's another indicator of the immorality of the Jedi that he "redeems himself" by merely offing the emperor.

Han Solo is the only remotely moral character, a decent, albeit simple, man surrounded by amoral plotters and schemers.

I wonder how subversive, or even diabolical, Lucas really is ....

Give me Spielberg any day.

Microsoft's HealthVault: Only one question worth asking

There's only one question worth asking about Microsoft's HealthVault.

Microsoft, what are you doing to ensure I can move all of my HealthVault data seamlessly, with no loss of any meaning or value, to Google's "HealthVault", or that of US Bank, or Dr. Frankenstein's [1] HealthVault or ... ?

Well, you get the idea.

That's it. Every other question is a meaningless distraction.

[1] I met a Dr. Frankenstein many years ago. She was a charming family physician with a good sense of humor. I have also met Microsoft - often. I have no doubt who I'd prefer to operate my family's "HealthVault".

Living the lie: OJ Simpson, George Bush and Marion Jones

I am a lousy liar.

That was a lie.

No, it wasn't. I don't get a buzz from deception, and I'm bad at lying -- though my kids are forcing me to learn a few tricks of the trade. So one might think I'd have a hard time disbelieving OJ Simpson, Marion Jones, Floyd Landis, George Bush, Bill Clinton, Lance Armstrong, Donald Rumsfeld, Dick Cheney* ... well, you get the idea. It's a long list.

In fact I'm a cynical wretch. I recall being a very credulous child, so I presumably learned to compensate. In all of the above cases I assumed the individuals were lying. When questions of integrity arise I have learned to largely discount what a person says, how they say it, how often they say it, how intensely they say it, and how "honest" their face. I'll take the evidence, thank you.

I suspect that people who are merely good, but not superb, at both lying and detecting lies are most likely to be fooled -- they put too much weight on their ability to detect a lie. There's always a better liar.

Deception is not entirely a bad thing. If my son insists in believing in Santa Claus or the integrity of the President I'll play along. The mass of humanity would keel over in shock if their politicians were honest with them. The nature of the world is best understood slowly.

On the other hand, failure to learn can be a bad thing. People who are surprised to learn that Jones used massive amounts of drugs to win, or who believe her when she lies about "flaxseed", or who believe Bush when he denies his torture policy, need to rethink their learning curve.

Stop listening. Start thinking. Look to your inner autistic ...

* Cheney is actually a lousy liar, but I thought I'd throw him in for good luck.

Pogue loves the XO, I remember the Commodore 64

Lots of people have been poking fun at Negroponte and the "XO" laptop. Pogue has been using one ...

Laptop With a Mission Widens Its Audience - New York Times

... There’s no CD/DVD drive at all, no hard drive and only a 7.5-inch screen. The Linux operating system doesn’t run Microsoft Office, Photoshop or any other standard Mac or Windows programs. The membrane-sealed, spillproof keyboard is too small for touch-typing by an adult.

And then there’s the look of this thing. It’s made of shiny green and white plastic, like a Fisher-Price toy, complete with a handle. With its two earlike antennas raised, it could be Shrek’s little robot friend.

And sure enough, the bloggers and the ignorant have already begun to spit on the XO laptop. “Dude, for $400, I can buy a real Windows laptop,” they say.

Clearly, the XO’s mission has sailed over these people’s heads like a 747.

The truth is, the XO laptop, now in final testing, is absolutely amazing, and in my limited tests, a total kid magnet. Both the hardware and the software exhibit breakthrough after breakthrough — some of them not available on any other laptop, for $400 or $4,000...

It uses almost no power. I think I might buy a pair in November. The program deserves a chance to succeed. If Pogue is right, and he usually is, I think they'll reconsider their decision not to sell it for $250 into the US market. I remember a time that people could buy a Commodore 64 for about that price, there's been nothing comparable since.

We know what we are: Torture and the end of the American exception

Today the NYT revealed that the Bush administration lied when they said they'd cut back on torture. I won't bother quoting the original article because I assume only the most dimwitted are surprised. Cheney, Bush, Gonzales and their cronies are snakes. The reactions are more interesting, here's a very edited version of Greenwald's response:

Glenn Greenwald - Political Blogs and Opinions - Salon

Much outrage has been provoked by the generally excellent New York Times article this morning revealing the Bush administration's recent violations of legal restrictions on the use of torture and other "severe interrogation techniques." And, in one sense, the outrage is both understandable and appropriate. Today's revelations involve the now-familiar, defining attributes of this administration -- claims of limitless presidential power, operating in total secrecy and with no oversight, breaking of laws at will, serial misleading of the Congress and the country and, most of all, the shattering of every previous moral and legal constraint on our national behavior.

But in another, more important, sense, this story reveals nothing new. As a country, we've known undeniably for almost two years now that we have a lawless government and a President who routinely orders our laws to be violated. His top officials have been repeatedly caught lying outright to Congress on the most critical questions we face. They have argued out in the open that the "constitutional duty" to defend the country means that nothing -- including our "laws" -- can limit what the President does.

It has long been known that we are torturing, holding detainees in secret prisons beyond the reach of law and civilization, sending detainees to the worst human rights abusers to be tortured, and subjecting them ourselves to all sorts of treatment which both our own laws and the treaties to which we are a party plainly prohibit. None of this is new.

And we have decided, collectively as a country, to do nothing about that. Quite the contrary, with regard to most of the revelations of lawbreaking and abuse, our political elite almost in unison has declared that such behavior is understandable, if not justifiable. And our elected representatives have chosen to remain largely in the dark about what was done and, when forced by court rulings or media revelations to act at all, they have endorsed and legalized this behavior -- not investigated, outlawed or punished it...

... We all know what has happened to our standing in the world, to our national character and our core political values, as a result of the previously unthinkable policies the Bush administration has relentlessly pursued. Ignorance or incredulity can no longer explain our acquiescence. Accommodating and protecting the lawbreaking of high Bush officials is widely seen by our Beltway elite as a duty of bipartisanship, a hallmark of Seriousness...

... The NYT article today reveals new facts about the administration's lawbreaking, lying, and pursuit of torture policies which we had decided, with futility, to outlaw. The Congress could aggressively investigate. Criminal prosecutions could be commenced. Our opinion-making elite could sound the alarm. New laws could be passed, reversing the prior endorsements and imposing new restrictions, along with the will to enforce those laws. We still have the ability to vindicate the rule of law and enforce our basic constitutional framework.

But does anyone actually believe any of that will be the result of these new revelations? We always possess the choice -- still -- to take a stand for the rule of law and our basic national values, but with every new day that we choose not to, those Bush policies become increasingly normalized, increasingly the symbol not only of "Bushism" but of America.

Glenn, we know what we are, we've proven that beyond a reasonable doubt. Nobody on the right, left, or center says "it can't happen here" any longer. We're probably a bit too heterogeneous to make great Nazis, but we could do some noisier variant thereof. We are a flawed nation with a nasty past and a weak moral core -- like Japan, China, Russia, Germany, France, Mexico, Brazil, the UK, Zimbabwe, Uganda and, really, most of the world excepting the Scandinavians and maybe Canada.

American "exceptionalism" was always a fraud, but now nobody believes it.

We can still aspire to a higher ground, but it will take a generational effort. The boomers have failed America's better self, we can only hope future generations are made of better stuff.