Physics in crisis: the non-zero vacuum energy

When I did my BS in 1981, I bought a copy of Thorne, Wheeler and Meisner's 'Gravitation'. I figured I'd study it in retirement, with a computer aide to do the math for me. It's a marvels book, one of the most extraordinary textbooks I've seen anywhere. My copy is autographed by Kip Thorne.

I think of that book now when I read of a conference Thorne attended, a Caribbean party for celebrity cosmologists. This comes from a report of the meeting:

Edge: THE ENERGY OF EMPTY SPACE THAT ISN'T ZERO: A Talk with Lawrence Krauss

...When you apply quantum mechanics and special relativity, empty space inevitably has energy. The problem is, way too much energy. It has 120 orders of magnitude more energy than is contained in everything we see!...

...One of the greatest developments in physics in the 20th century was to realize that when you incorporate special relativity in quantum mechanics you have virtual particles that can pop in and out of existence, and they change the nature of a hydrogen atom, because a hydrogen atom isn't just a proton and electron.

That's the wrong picture, because every now and then you have an electron positron pair that pops into existence. And the electron is going to want to hang around near the proton because it's oppositely charged, the positron is going to be pushed out to the outskirts of the atom, and while they're there they change the charged distribution in the atom in a very small, but calculable, way. Feynman and others calculated that effect, which allows us to get agreement between theory and observation at the level of nine decimal places. It's the best prediction in all of science. There's no other place in science where, from fundamental principles, you can calculate a number and compare it to an experiment at nine decimal places like that.

But then when we ask, if they're there, how much should they contribute to the energy in the universe, we come up with the worst prediction in physics. It says if empty space has so much energy we shouldn't be here. And physicists like me, theoretical physicists, knew they had the answer. They didn't know how to get there. It reminds me or the Sidney Harris cartoon where you've got this big equation, and the answer, and the middle step says "And then a miracle occurs". And then one scientist says to another, "I think you have to be a little more specific at this step right here".

The answer had to be zero. The energy of empty space had to be precisely zero. Why? Because you've got these virtual particles that are apparently contributing huge amounts of energy, you can imagine in physics, how underlying symmetries in nature can produce exact cancellations — that happens all the time. Symmetries produce two numbers that are exactly equal and opposite because somewhere there's an underlying mathematical symmetry of equations. So that you can understand how symmetries could somehow cause an exact cancellation of the energy of empty space.

But what you couldn't understand was how to cancel a number to a hundred and twenty decimal places and leave something finite left over. You can't take two numbers that are very large and expect them to almost exactly cancel leaving something that's 120 orders of magnitude smaller left over. And that's what would be required to have an energy that was comparable with the observational upper limits on the energy of empty space.

We knew the answer. There was a symmetry and the number had to be exactly zero. Well, what have we discovered? There appears to be this energy of empty space that isn't zero! This flies in the face of all conventional wisdom in theoretical particle physics...

This is the best summary of the physics dilemma I've read. The same math leads to a perfect prediction and a perfectly outrageous prediction. So we know the physics is very broken, but we don't have a path to follow. The best minds in the world can't come up with experiments to guide us. No wonder physicsts are extremely frustrated! I just hope I see a breakthrough before I get to meet Einstein in person ...

PS. Virtual particles came courtesy of Richard Feynman. I have his books too, but he was a much better lecturer than textbook writer.