Saturday, August 07, 2004

Email is truly broken - so don't be upset if I don't reply!

The New York Times > Technology > Circuits > Delete: Bathwater. Undelete: Baby.
Several months ago, Dr. Kim and Mrs. Crasco were at a meeting when they ran into a program director they knew from the American Association for the Advancement of Science. She greeted them coolly. Puzzled, Dr. Kim and Mrs. Crasco asked what they might have done to offend her.

As it turned out, she had sent Dr. Kim and Mrs. Crasco an e-mail message suggesting that they work together on a grant application. The application deadline had since passed, and the acquaintance was more than a little miffed that she had gotten no response from them.

The two entrepreneurs were flabbergasted. Not only did they have no idea the e-mail had been sent, they had no idea that it had been snuffed out as junk.

Yes, email is broken. It's been obviously broken for a couple of years. I have over 4000 intercepted spams a week, and about 70 that aren't intercepted. Among the > 4000 intercepted spams are probably 1-3 legitimate emails. There's no way ANY spam filter can be accurate enough to eliminate 99.5% of all spam and NOT also eliminate 1 or more legitimate emails. I have done test analysis as part of my worklife, and no test is perfect. Spam filtering is a testing procedure. (If spam filtering services made some trivial UI changes it would be MUCH easier to hunt for the legitimate email in the mounds of digipoop -- the authors of those products clearly don't get much email.)

I proposed, years ago, a series of technical approaches to spam. Indeed, many years ago, I proposed to Mindspring that they start offering spam filtering as a service -- and months later they were the first to provide ISP based filtering.

I am sure none of my suggestions were original, though I've never seen a complete presentation of my primary recommendation: "differential filtering based on the managed reputation of an authenticated sending service". In any case, there are many technical fixes that will work. All of them will involve some form of sender authentication (authenticating the sending service and then filtering based on sending service reputation, as I propose, pushes the authentication obligation to the sending service and allows at least potential identity protection).

In general a lot of things done by email (file transfer, broadcast communication, collaborative groups, receiving notification of changed content) can be better done through other technologies, esp. webDav and RSS/Atom syndication. (Finding a unified workflow engine is a related challenge, however.) Sending messages, however, requires email or something fundamentally identical; instant messaging is no replacement for email.

Email must be fixed. It can be fixed. It will be fixed. It's just a question of whether the fix is a trojan horse for the RIAA/Palladium/DRM/Ashcroft agenda, or a more modest fix that addresses our messaging needs alone.

In the meantime, if I don't reply to your message, don't assume I got it.

No comments: