Dark horizons of knowledge

Supernovae sent Robert P. Kirshner a startling message he did not want to hear - that the universe was expanding rapidly. So maybe Einstein's 'bad idea' was correct after all.

It was a warm winter day in Pasadena, California, but I had the shivers. If the data I had were right, we had just discovered two-thirds of the universe. And it was in a form that had been on the Index of Bad Ideas since 1931. If our observations were right, we were living in an accelerating universe dominated by dark energy.

I was visiting the California Institute of Technology in the first days of 1998, on sabbatical from Harvard University. As a Caltech graduate student and beyond, I had absorbed the lesson that Einstein's "cosmological constant", which he invented in 1917 to make the universe static, was a very bad idea. Pasadena was ground zero for cosmology, the science of the universe. Edwin Hubble had used the telescopes of Mount Wilson, looming over Pasadena, to discover that we live in an expanding universe.

That was in 1929, and since then, the cosmological constant, usually symbolised by the Greek letter lambda, was regarded as a form of scientific poison ivy - something to be scrupulously avoided. Legend held that Einstein, full of hindsight and rue, regarded the cosmological constant as "his greatest blunder". In fact, what he said in 1932 was less dramatic and more sensible. "An increase in the precision of data derived from observations will enable us in the future to fix its sign and determine its value." I was in the future, and I was looking at the data.

Supernovae halfway across the universe were sending us a message that we did not want to hear - the universe was not slowing down due to gravity, instead its expansion was accelerating. This pointed to the cosmological constant, or something more modern sounding with the same effect: dark energy. I was part of an international team using distant exploding stars to judge cosmic distances. Hubble had shown that the more distant objects in the universe are receding from us more rapidly. We were interested in seeing far enough into the past to measure how the mass in the universe had slowed cosmic expansion. When you look at very faint supernovae, they tell you the relation between motion and distance over a time span of 7 billion years. That was enough to expect to see the signature of gravity.

A competing group from the Lawrence Berkeley Laboratory (LBL) had published a paper in July 1997 claiming that its supernova data pointed toward a dense, decelerating universe dominated by dark matter. Our team had started later, but we had independent data, and Adam Riess, once my student at Harvard but then a Miller fellow at Berkeley (now a staff member at the Space Telescope Science Institute), was doing the heavy lifting on the analysis of our data. Every time he did the mathematical fit for the amount of matter implied by our measurements, he got a negative mass. Negative mass? What was that? If you stand on a scale and it reads a value below zero, you might suspect that there was something else going on. In this case, it was the accelerating effect of the cosmological constant.

I didn't like this result. I didn't think all our team put together was smarter than Einstein - and he had lived to regret the cosmological constant. I did not want to make the same mistake.

"Adam," I said, "the punishment for being wrong should be as big as the reward for being right."

"I'm going to get a reward?" he asked.

On January 12 1998 (at 10:18:31am), I sent an email to the group:

"I am worried that the (data) looks like you might need some lambda. In you heart, you know this is wrongI" Adam wrote back: "I feel like tortoise racing the hare. Every day I see the LBL guys running around, but I think if I keep quiet I can sneak up... shhh... The data require a nonzero cosmological constant! Approach these results not with your heart but with your eyes. We are observers after all!"

In the end, we let the facts guide us. The universe does not care if you like it. Physical scientists express their confidence in a result by comparing the measurement to the uncertainty in the measurement, sigma. If we believed our statistics, we should be willing to bet 370 to 1 on a result like ours. You wouldn't bet your house, you wouldn't bet your dog, but you would be willing to say something at a scientific meeting. So Alex Filippenko of our High-Z Supernova Team spoke at the February 1998 Dark Matter meeting in Marina Del Rey, California. He said the universe was not slowing down due to gravitation. It was speeding up over time, as if driven by something just like the cosmological constant. We submitted our result to The Astronomical Journal on March 13. It was refereed, accepted in May and out in September. The LBL team reached the same conclusion independently - submitting a paper to The Astrophysical Journal on September 8 1998 that came out in June 1999.

What does this mean? If you combine the supernova result with measurements of the glow from the big bang, the cosmic microwave background, you conclude that the universe is one-third dark matter (which has ordinary gravity) and two-thirds dark energy (which makes things fly apart). If that is right, then we discovered two-thirds of the mass-energy in the universe when we decided to trust the statistics of the data. Not a bad day's work.

Does anyone understand the dark energy? No. If it is Einstein's cosmological constant in modern form, the value we find from observation is 10 120 times larger than the value you would get from treating gravity like the other forces of th eworld using fundamental theory. This suggests that something is missing from our physics. The deep thinkers suggest it might be that the universe is 11-dimensional and that gravity spreads into those hidden dimensions to produce the effect we have measured. Personally, I hardly know what those words mean. As a humble observer, I feel like Hubble, who wasn't sure he was really seeing an expanding universe in his data. In 1931, he wrote to Einstein's collaborator, Willem de Sitter: "The interpretation, we feel, should be left to you and the very few others who are competent to discuss the matter with authority."

Is it right? Since 1998, our High-Z team and the LBL group have been lucubrating at full speed. So far, all signs are good - new data confirms the earlier results, and extending the range of the measurements with the Hubble Space Telescope seems to show that the universe has switched from decelerating to accelerating within the range of our measurements. This "stop and go" universe fits well with the theory. The new camera brought up to the Hubble Space Telescope by shuttle astronauts this year promises to reveal more of the very distant supernovae that will make this result more secure.

Does it matter? I think so. Most people want to know where we came from and where we are going. If this work is right, we came from a hot, dense big bang 14 billion years ago, whose subtle density variations grew to become stars, galaxies, planets and people. We are headed toward an unbounded future of ever more rapid cosmic expansion in which distant galaxies will fade away as they recede more rapidly. Astronomy shows us a world that is stranger than we dared imagine. It nourishes our deep desire to learn what the world is and how it works.

Robert P. Kirshner is Clowes professor of science at Harvard University. His book The Extravagant Universe is published by Princeton University Press, £19.95.