Information (and data) Visualization: The future always arrives late and unexpectedly

I've intermittently taught data visualization to grad students. It's been the same old thing for years -- poor quality scans of examples from 20-30 year old experiments. Nothing seemed to make it out of the lab. So I was surprised when a colleague (Andrew) pointed me to an unexpected reference: 

Data Visualization: Modern Approaches

This article gives a nice overview of various visualization technologies that people are experimenting with.  Some seem like eye candy, some seem genuinely useful and could be applicable to clinical applications.  The articles and resources section gives some good links to other resources on the web, such as visual complexity which has hundreds of different ideas for visual representation of topics....

Andrew also pointed me to the interactive Flash based BBC British History Timeline. Lovely, and a very handy reference to use with "In Our Time" podcasts. These kinds of visualizations do make me hope Apple is able to create a decent Flash client for OS X (and thus for the iPhone). (since Adobe can't).

I expect the data visualization post will be widely used by anyone lecturing on information/ knowledge/ data visualization. It's also a golden example of how much power Blogs (Smashing Magazine is really a blog in drag) provide. Once upon a time, this would have had to be a book, with an enormous barrier to publication. Today this unattributed article has "Views: 70392 by 52501 users". As near as I can tell this is the author information:

Smashing Magazine is maintained by Sven Lennartz, the owner of the Dr. Web Magazine and Marketing Tricks and Vitaly Friedman, the creator of The Web Developer’s Handbook - with a little help from Christiane Rosenberger.

Of course I've added SM to my bloglines subscription list. Anyone who can do this must have more to offer in the archives and the future ...

Hedge funds and private equity: a hedge fund manager's perspective

Freakonomics gets some great responses from a hedge fund manager. No respectable senator can claim any longer (though really, they never could) that there's a good reason to exempt hedge fund managers from taxation.

Excerpts below ...

Freakonomics Blog � Your Hedge Fund Questions, Answered

Q: (1) Do you think the Senate proposal [to raise taxes] really could cause a flight of the industry overseas, as Ben Bernanke hinted?

A: First, let me state the obvious: there is no public policy reason for hedge fund and private equity managers to pay a lower tax rate than teachers, doctors, or lawyers. ..

... let’s not kid ourselves — the tax code is a gift to the industry...

... from a fairness perspective, it is a no-brainer. The notion that the most profitable industry in the history of mankind (I hyperbolize, but on a per-person basis, this might in fact be true) requires a lower tax rate to take risk and make investments, simply does not square with logic. I know of no manager who would stop working or stop investing as a result...

Q: I’m curious as to what degree the emergence of hedge funds has changed business strategy....

.... Often, companies go private when activists chase management into the arms of LBO shops willing to put a level of debt on corporate balance sheets that public investors would find imprudent. As the economy slows or credit markets tighten, these companies will have trouble making their debt payments, and — just as the LBO boom in the 1980s burst — lenders and possibly private equity investors will be left holding the bag...

Q: Do you believe that hedge funds provide a social good? ...

.... What specific benefits to society as a whole are provided by hedge funds? I do believe that, starting with the 1980s bull market, we have seen a society-wide reallocation of human resources. And it is unfortunate that the “best and the brightest” no longer seem to want to go into government or medicine or teaching (I did say I’d be generalizing here), and instead seem to gravitate to the best-paying professions. In other eras, I believe that was not the case to the degree it is today. Hedge funds didn’t make this happen — back in the 1980s and 1990s it was the investment banks that siphoned off the talent — but the industry has certainly made matters worse...

I chose the quotes that I found most interesting, but our manager also says that hedge funds, once they become available to retail investors, will be competitive alternatives to conventional mutual funds.

Why didn't more people die when 35W fell?

It is no comfort to the loved ones of the 13 who appear to have died, or to those facing lifelong disability from the fall of the I-35W bridge, but the greatest puzzle of the disaster is that so many survived. What are Minnesotans made of?

How did so many survive?

It's hard to imagine anyone surviving a six-story drop into the Mississippi River.

But it's now apparent that the vast majority of those who were on the Interstate 35W bridge when it collapsed Wednesday escaped with relatively minor injuries.

Although the final death toll is still unknown, doctors and safety experts say that a combination of factors, from physics to shock absorbers, probably helped cushion the blow for those plunging from the bridge in their vehicles.

In general, they say, the cars and the bridge itself helped absorb some of the impact that would have killed someone free-falling from that height.

"I would say over two-thirds of the people walked away," said Dr. Marc Conterato, an emergency room physician at North Memorial Medical Center in Robbinsdale, who was at the site. "Believe me, the human body can absorb a lot of trauma."

As of Saturday, the death toll stood at five, and 24 people remained hospitalized, five in critical condition and four in serious. About 75 others were treated and released. About eight people are missing and presumed drowned.

As tragic as that is, it's a far cry from what some at the scene expected.

"I figured we'd probably have a couple of hundred injured, and 25 or 50 fatalities," said Dr. John Hick, an emergency doctor and disaster coordinator at Hennepin County Medical Center.

Many of those treated at hospitals had broken bones, fractures or back injuries from the vertical fall, according to physicians.

"I've certainly seen many worse injuries in car crashes," said Dr. Jeffrey Chipman, a trauma surgeon at the University of Minnesota Medical Center, Fairview.

In some cases, the vehicles dropped straight down on top of a portion of the bridge as it hit the water. That "would have created some kind of cushion when they landed," Chipman said.

James Kakalios, a physics professor at the University of Minnesota, agreed. "Some of them were able to ride parts of the expressway down," he said. "And that helped distribute the force and save the individual cars, as opposed to a car just falling 50 feet on its own."

Vehicle safety designs probably also played a role, said Lanny Berke, a mechanical engineer and safety specialist in Plymouth who is a frequent expert witness in accident cases.

"Let's start with the school bus," he said, referring to the bus carrying dozens of children. Because of federal safety rules, he said, it had an emergency door at the rear through which they could escape. "So the federally mandated design features for school buses saved those kids' lives," he said. "Because there's no way in hell they could have gotten out the front."

At the same time, he said, seat belts and airbags could have helped some survive, as well.

And the survivors had another thing going for them, the experts agree. "We were lucky," said Conterato. "We were actually in an area that was very well populated. People had relatively easy access to the area. Plus, we were relatively close to two large medical facilities, the university and Hennepin County. So we were able to put people on the scene relatively quickly.

I'm sure the cars were helpful, but the construction workers were standing on the bridge. They lost one person. Did they fly?

I suspect a lot of incidental bystanders took heroic risks to help people, risks that by rights should have killed them as well. Somehow even the rescuers survived. The rush hour group is also probably younger and more vigorous than a random population sample, Minnesotans are relatively healthy by national standards, the river is warm right now, and we obviously have a lot of competent swimmers ...

Fake rabies vaccine

10,000 doses.

China Blacklists 400 Exporters - Forbes.com

... Earlier this week, the official People's Daily reported, police detained a further 17 members of a gang in Heilongjiang province producing counterfeit medicines. In an earlier raid on the gang, authorities seized 10,000 doses of bogus rabies vaccine, 20,250 bottles of a fake version of a medicine used to treat cardiovascular disease and 211 bottles of blood protein. In all, the paper said, police confiscated fake versions of 67 medicines produced by 53 companies...

Maybe I need to reconsider my principled opposition to Hell.

Solving the china import problem; $1 million prize for each recall

Want to solve the China problem?

Why lead-tainted Chinese goods slip through despite U.S. recalls -- chicagotribune.com

...Three decades after the federal government significantly toughened regulations on lead in children's products, American companies have yet to find a way to successfully screen the flood of imported products for the toxic metal.

The federal watchdog charged with ensuring they do so is overwhelmed and often ineffective. And the growing list of lead recalls of children's products underscores how the metal, slathered on with paint or mixed in with other raw materials, is more pervasive than many American consumers ever imagined...

It's not so hard.

Offer a $1 million prize to each person who detects a problem that triggers a product recall. I'll bet most of the prizes are won by Chinese entrepreneurs. Within 3 years the recalls would stop.

Next, I'll bring peace to the middle east.

The Center of it All

The center of it All.

2011 comes early: Ms. Carbon Tax, meet Mr. Bridge

I figured American life would change when gas hit $5/gallon in 2011. I picked that date because I figured we won't get a carbon tax through before 2013. That was before one of my local bridges took a tumble and I started to do the math on 77,000 "deficient" (rating =4) bridges and wonder how much it costs to (re)build a bridge. If we assume an average cost of, say, $30 million each, and we assume we replace/rebuild only 40,000 deficient bridges, that's $1.2 trillion dollars.

Now, perhaps we'll decide that the I-35W collapse was basically in line with expectations. We all understand that bridge ratings are probabilistic -- the yearly risk of catastrophic collapse is not 0% at the highest rating (10) or 100% at the lowest rating (0). Maybe it's something like a 1/100,000 risk at the rating of 4, with a substantially higher risk of non-catastrophic failure. With 77,000 bridges at that rating we should expect one to collapse every few years somewhere in the US and approximately 50 people to die or be injured by bridge collapse every 1-2 years.

That's a risk that a Vulcan would probably find quite reasonable. It's probably significantly less than the risk we assume with current food imports; riding a motorcycle is probably a hundred to a thousand times higher risk. Logically we may decide to just accept that and stick with our current bridge replacement/repair policies.

Humans are not logical. There's a good chance we'll run up a trillion dollar infrastructure repair bill rather than have yearly bridge collapse headlines.

We could simply borrow to build. It's a much better investment than the Iraq war, for example. I have a hunch though that Bush has busted our budget already. So we need to raise a trillion dollars.

We also need to reduce CO2 emissions.

We also need to decrease the load on our old infrastructure.

Gee, I wonder how we could do all of these things ...

$5/gallon by 2009?

PS. If I were in a business related to bridge construction, I'd be hiring today ...

Update 9/17/07: This is looking more likely ...

Update 9/21/07: It occurs to me that this is the cure for the AMT problem. More on that in a f/u post.

Bridges: 77,000 deficient, 750 have I-35W design

At Bridge Site, Search of River Moves Slowly - New York Times

... Dan Dorgan, a state bridge engineer for the Minnesota Department of Transportation, said a “deficient” designation did not mean a bridge needed to be immediately replaced; 77,000 bridges across the country, he said, have a similar designation...

Elsewhere I read that 750 bridges have the same design as our fallen I-35W bridge. I wonder what percentage of those are rated as deficient?

Maybe Google will start attaching design and deficiency rations to Google Earth/Maps. I bet that would be a good way to attract users.

This accident may end up costing billions ...

8/3/07 Update: I thought a bit more about how Google could accelerate the infrastructure review. A "route around risky bridge" option for Google Map directions would concentrate minds wonderfully. One can readily imagine icons for bridges with the I-35W design and risk designation. Did I say "billions"? Sorry, I meant tens to hundreds of billions ...

8/9/07 Update: My wife points out that as much as we may despise Pawlenty, it was the Ventura administration, albeit as part of a GOP initiative that returned a state budget surplus to the taxpayer rather than, say, use it to repair bridges. So Jesse Ventura deserves a chunk of the I-35W bridge for his desk.

Macworld - what the iPhone needs

I've seen a lot of lists of what the iPhone needs, but this MacWorld list most resembled my own: Macworld: First Look: iPhone fixes we want to see. I'd break the list into two parts: one part that's highly desirable and the other part that's so critical I can't buy the phone for fear Apple doesn't intend to address the problems. The latter list includes third party applications, tasks, work/home/family calendar management, search, cut and paste, etc. I could live with the crummy AT&T network and the lack of GPS if I had those features.

My experience with iPhoto and Aperture has taught me that Apple can take an incredibly long time to add features I consider critical, such as Library merge in iPhoto and date value edits in Aperture (neither exist). Given that experience, I can't buy the darned phone hoping Apple will fix the deal killers. I'm not Apple's target market, and won't necessarily listen to my concerns.

Wikipedia and the I-35W collapse

As expected, Wikipedia now has a good initial entry on the I-35W Mississippi River bridge collapse. This will be the place to go over the next few weeks and months to get an understanding of what happened. The Wikipedia article references the National Bridge Inventory but I was unable to find any record for the 35W bridge at this time. I will be most interested in seeing a list of bridges of similar age, design and history -- though one or more of construction error and local geology may turn out to be relatively unique contributors.

The Wikipedia article links to the Silver Bridge collapse of 1967, the outcome of the subsequent investigation suggests what may lie ahead:

... The collapse focused much needed attention on the condition of older bridges, leading to intensified inspection protocols and numerous eventual replacements. There were only three other bridges built to a similar design, one upstream at St Mary's and a longer bridge at Florianopolis, Brazil. They were both closed immediately, and the St Mary's bridge demolished in 1971....

I suspect we'll know within a few days which bridges in the US have a similar design.

Update 8/2/07: Culturally, slow to anger, but ....

Update 8/2/07: I've seen mentions of a bridge failure in 1983, the reference is to metal corrosion that caused the collapse of the Mianus River Bridge in Connecticut, killing 3 people and injuring 5 others. Wikipedia has a list of bridge disasters.

TSA Administrator on No-Fly list

I was wondering when Schneier would get to the "no-fly list" part of his interview with the TSA Administrator, Kip Hawley. It came out today.

My read is that Hawley was defensive and his answers this set of questions were weak. Here he discusses the new redress procedure ... (emphasis mine)

Schneier on Security: Conversation with Kip Hawley, TSA Administrator (Part 3)

...KH: ... if someone is either wrongly put on or kept on, the Terrorist Screening Center (TSC) removes him or her immediately. In fact, TSA worked with the TSC to review every name, and that review cut the no-fly list in half. Having said that, once someone is really on the no-fly list, I totally agree with what you said about appeal rights. This is true across the board, not just with no-flys. DHS has recently consolidated redress for all DHS activities into one process called DHS TRIP. If you are mistaken for a real no-fly, you can let TSA know and we provide your information to the airlines, who right now are responsible for identifying no-flys trying to fly. Each airline uses its own system, so some can get you cleared to use kiosks, while others still require a visit to the ticket agent. When Secure Flight is operating, we'll take that in-house at TSA and the problem should go away.

So the list was so bad an initial review dropped it by 50%. I hope Hawley is simply dissembling and that he's not so foolish as to think his answers are reasonable. Note the problem with each airline having their own procedures for dealing with "cleared" names. What a mess.

A local bridge falls

A local bridge fell into the Mississippi today. We're on the front page on the BBC World:

BBC NEWS | Americas | Six dead as US bridge collapses

...Russ Knocke, a spokesman for the Department of Homeland Security, told the AFP news agency: 'There are no indications of a nexus to terrorism at this time.'

The initial reports said "not a terrorist event", but of course that's unknowable. Later reports used "No indications", which is at least plausible.

We're a pretty wealthy state without any recent history of major infrastructure problems, and there's been no significant work on that bridge recently. Bridge engineering is supposed to be very well understood. So either our engineering textbooks need a rewrite, or there was something wrong about the initial construction or later repair of this particular bridge (and its peers?), or somebody blew it up. The last seems very unlikely but the other explanations aren't too likely either.

Journalism in transition and not-so-irrational voters: Rosenberg

Scott Ronsenberg, who's just left Salon, wrote two posts that merit a broader attention. The first is simply a response to Kristoff's recent "irrational voter" essay. Rosenberg expresses exactly what I thought as I read Kristoff's article. I think the American voter is future shocked, comatose, AWOL and derelict, but I thought Kristoff's examples of irrationality were .. umm ... irrational. Rosenberg captured my thoughts perfectly:

Those darn irrational voters.

... What are the ways in which voters are “worse than ignorant”? Kristof summarizes Caplan’s complaints of “systematic error” in voter rationality: Voters share “a suspicion of market outcomes and a desire to control markets.” They have “an anti-foreign bias,” evidenced by an unwillingness to embrace free trade wholeheartedly. They share “a neo-Luddite bias against productivity gains that come from downsizing or “creative destruction.’” And they have a “pessimistic bias, a tendency to exaggerate economic problems.”

Gee, it sounds like the real problem Caplan has with the voting public is that they don’t agree with the program of conservative economists!...

...Personally, I’m reasonably comfortable with the pro-free-trade argument. But you won’t find me sneering at those who sense that the dynamic of the global economy is not doing them or their families any good.

More significant in the longer run is Rosenberg's summary of the state of the journalist in the Murdoch-WSJ era:

Scott Rosenberg’s Wordyard » Blog Archive » Murdoch, the Journal, and the newsroom diaspora

It is no surprise that Rupert Murdoch will be the new owner of Dow Jones and the Wall Street Journal...

...The truth is that most professional journalists in the U.S. have lived in a cocoon for decades...

... I don’t trust Rupert Murdoch. He has a long and well-documented record of using his properties to further his own agenda. But I trust that there are a lot of smart writers and editors at the Journal. Either they’ll get an opportunity to reshape their paper in a way that suits the times and their own consciences — or they’ll find themselves in the great newsroom diaspora with the rest of us, helping us figure out new models for the future.

WSJ journalists have no excuse for complaining about the cold winds of capitalism, I'm sure they're too smart to expect much sympathy. I do hope that the best of them take Rosenberg's challenge and join a new world, but I also remember that when BYTE died, nothing replaced its value. (The sum of the entire tech blogsphere is probably the closest thing we have now, and that took about ten years post-BYTE to emerge.)

PS. As to the fate of the WSJ, as I've written before, I'm an optimist. At worst it will stay about the same. At best Murdoch will eliminate the editorial staff and keep the news people. Next best is to keep the editorial staff but so weaken the news function that it dies a slow death. All improvements.

The way we think: reason as an afterthought

Towards the end of an article reporting how much unnoted context alters our feelings and behaviors, a NYT article describes the implication for how we think. It's another in a long series of blows to the idea that we're fundamentally rational (emphases mine)...

The Subconcious Brain - Who's Minding the Mind? - New York Times

... an area called the ventral pallidum was particularly active whenever the participants responded.

“This area is located in what used to be called the reptilian brain, well below the conscious areas of the brain,” said the study’s senior author, Chris Frith, a professor in neuropsychology at University College London who wrote the book “Making Up The Mind: How the Brain Creates our Mental World.”

The results suggest a “bottom-up” decision-making process, in which the ventral pallidum is part of a circuit that first weighs the reward and decides, then interacts with the higher-level, conscious regions later, if at all, Dr. Frith said.

Scientists have spent years trying to pinpoint the exact neural regions that support conscious awareness, so far in vain. But there’s little doubt it involves the prefrontal cortex, the thin outer layer of brain tissue behind the forehead, and experiments like this one show that it can be one of the last neural areas to know when a decision is made.

This bottom-up order makes sense from an evolutionary perspective. The subcortical areas of the brain evolved first and would have had to help individuals fight, flee and scavenge well before conscious, distinctly human layers were added later in evolutionary history. In this sense, Dr. Bargh argues, unconscious goals can be seen as open-ended, adaptive agents acting on behalf of the broad, genetically encoded aims — automatic survival systems.

In several studies, researchers have also shown that, once covertly activated, an unconscious goal persists with the same determination that is evident in our conscious pursuits. Study participants primed to be cooperative are assiduous in their teamwork, for instance, helping others and sharing resources in games that last 20 minutes or longer. Ditto for those set up to be aggressive.

This may help explain how someone can show up at a party in good spirits and then for some unknown reason — the host’s loafers? the family portrait on the wall? some political comment? — turn a little sour, without realizing the change until later, when a friend remarks on it. “I was rude? Really? When?”

Mark Schaller, a psychologist at the University of British Columbia, in Vancouver, has done research showing that when self-protective instincts are primed — simply by turning down the lights in a room, for instance — white people who are normally tolerant become unconsciously more likely to detect hostility in the faces of black men with neutral expressions.

“Sometimes nonconscious effects can be bigger in sheer magnitude than conscious ones,” Dr. Schaller said, “because we can’t moderate stuff we don’t have conscious access to, and the goal stays active.”

..Using subtle cues for self-improvement is something like trying to tickle yourself, Dr. Bargh said: priming doesn’t work if you’re aware of it. Manipulating others, while possible, is dicey. “We know that as soon as people feel they’re being manipulated, they do the opposite; it backfires,” he said...

Really, it's amazing we do as well as we do. Our mind seems a pretty thin veneer on a heck of a lot of evolutionary programming. I am reasonably certain, however, that self-awareness varies from person to person. In other words, consciousness, like strength, speed, and wit, is a variable. Perhaps with some training we can begin to acquire second hand access to our unconscious controllers, and subvert them. So, perhaps we cannot directly detect an unconscious motivator, but perhaps we can become better at evaluating our own behaviors. When we find ourselves skeptical, or friendly, or generous, or competitive, we might then infer the presence of an unrecognized trigger, and thus infer our unconscious goals.

More in another blog on the implications for the management of persons with behavioral problems ...

Recommender systems - so that's why they've been disappointing

Recommender systems for music and books haven't lived up to initial expectations. Netflix and Amazon give me pretty decent recommendations, but among other things they get confused between things I buy for myself and things I buy for other people DeLong excerpts Slee to suggest a host of other problems -- mostly fraud related.

There's so much money riding on recommender systems even clumsy fraud is common, so it's rather likely that subtle fraud is also common. It's the same problem Google has had, since Google's original search approach was a form of recommender system.

Useful recommender systems may first require a good reputation and identity management infrastructure, or be based on data points that cost money to create. Not coincidentally, when I'm researching Amazon the first thing I look at is the sales ranks for the product domain I'm interested in. Then, for each product that's selling well, I look at the negative reviews first. I don't pay that much attention to the star rankings or the positive reviews.