Prepare to be super-crunched

From banking to the wine industry to medicine, professional expertise and intuition are giving way to the more reliable outcomes of mass data analysis. The next target sector? Quite possibly yours. Ian Ayres explains. In the 1840s the physician Ignaz Semmelweis caused a stir when he noticed that maternity patients had a high probability of dying from childbirth (puerperal) fever when they were treated by doctors who had just performed autopsies. Yet mortality rates dropped from 12 to 2 per cent if doctors and nurses washed their hands in chlorinated lime before seeing each patient.

This result, which ultimately gave rise to the germ theory of disease, was fiercely resisted. Semmelweis was ridiculed by other physicians who refused to believe that they were causing their patients' deaths. And they complained that hand-washing several times a day was a waste of their valuable time. Semmelweis was eventually fired. After a nervous breakdown he ended up in a mental hospital, where he died at the age of 47.

The tragedy of Semmelweis's death and the needless deaths of thousands of women is ancient history. Doctors now, of course, know the importance of cleanliness. Medical dramas show them meticulously scrubbing up for operations. But the Semmelweis story remains relevant. Even today physicians' resistance to hand-washing is a deadly problem. While doctors carefully wash for surgery, they too often move from room to room without washing. "Clean hands" projects, which try to make cleaning routine every time medical personnel pass certain points, and periodically applying antiseptic barriers to things that doctors touch have been shown to reduce hospital infection and ultimately save lives.

But doctors still often resist changing their modus operandi just because a statistical study says so. Much has been written about "evidence-based medicine" (EBM), which calls on doctors to ground their treatment decisions on the best available statistical evidence. One of the reasons why doctors are resistant is that EBM has had the impertinence to ask them to change what they do with their time and to carry out research several times a day.

Of course, doctors shouldn't do specific research for every patient. It would be a huge waste of time to hit the books when someone presents with the classic symptoms of a common cold. But studies based on "shadowing" practising physicians reveal that about one in three new hospital patients poses challenges that would benefit from a review of current research.

Until EBM came along, doctors rarely researched the problems of individual patients. In sharp contrast to law practices, most doctors' offices didn't have libraries. If a physician didn't know the answer he might consult a specialist, but neither the physician nor the specialist would be very likely to pick up a journal. Doctors' offices still don't have physical libraries, but because of the web every examination room can now have a virtual one. EBM is starting to force physicians to change the way they do business.

Patients also have access to the web and are increasingly engaging in self-diagnosis. Physicians report that patients now often treat them merely as alternative sources of statistical information. They will say: "Show me the study" or demand the research that indicates chemotherapy is better than radiation for stage 3 lung cancer. Savvy patients are treating their doctors less like sage advisers and more like a human substitute for a web portal. People are looking past the MDs, who merely disseminate information, to the PhDs, who create the database to generate information.

At first it sounds as if this trend is going to boost the status and discretion of researchers in higher education. But the contraction in professional discretion experienced by physicians is part of a much larger phenomenon that may ultimately reduce the discretion of academics as well.

We are in a historic moment of horse-versus-locomotive competition, where intuitive and experiential expertise is losing out time and time again to a new kind of statistical analysis. The same principles apply, as I demonstrate in my book Super Crunchers , to economic forecasters, wine connoisseurs predicting the future prices of vintages and Hollywood producers deciding whether to "green-light" a script. Huge data sets of digital information are allowing a new breed of number cruncher to discover empirical correlations between seemingly unrelated things.

In field after field, the rise of "super crunching" threatens the status and respectability of many traditional jobs. Once, being a loan officer for a bank was a moderately high-status position. Today, such people are glorified secretaries who tend to just input applicant data into computers. The real loan decisions are made at a central office on the basis of a statistical algorithm. Giving discretion to low-level loan officers is bad business. It turns out that looking a customer in the eye and establishing a relationship doesn't help predict whether or not he or she will repay the loan.

The tendency towards reduced discretion is impacting on skilled workers and professionals as well. Even lawyers are feeling the pinch. You would think that advice from elite lawyers would be essential in predicting the outcome of cases, but tests have found that a statistical model could predict more accurately whether justices in the US Supreme Court would vote to affirm or reverse a verdict.

Similar debates about discretion versus statistically validated routines are beginning to impact on education. At primary-school level in the US, a controversial method called "direct instruction" (DI) has generated a ferocious battle about how best to teach schoolchildren. DI is controversial because it is completely scripted. The entire lesson - the instructions ("Put your finger under the first word"); the questions ("What does that comma mean?"); and the prompts ("Go on") - is written out in the teacher's instruction manual. The idea is to force the teacher to present information in easily standardised bite-sized chunks and to make sure that it is actually digested.

It seems incredibly dehumanising. Multiple studies, however, show that direct instruction does a better job at teaching reading and maths than a multitude of alternative "whole school" reforms.

But oh, the humanity. Could you imagine having to follow a script most of your working day, repeating ad nauseam stale words of encouragement and correction? It is a stock movie genre to show teachers getting through to kids with unusual and idiosyncratic techniques (I remember crying when I first saw To Sir, with Love ). No one's going to make a motivational drama about DI.

The creator and driving force behind DI, Siegfried Engelmann, was refreshingly frank about his views on teacher independence in an interview with The New Yorker . "We don't give a damn what the teacher thinks, what the teacher feels," he said. "On the teachers' own time they can hate it. We don't care, as long as they do it."

There are signs in the US that the scripting of education is also beginning to impact on the way older children are taught. An increasing number of schools are spending hours a week "teaching to the test" in an effort to raise student scores on high-stakes standardised tests. Indeed, some of the highest-level high-school classes - so-called AP (advance placement) classes - give teachers little room to innovate as they prepare students for tests in calculus, history and a variety of other subjects. Test prep classes are largely scripted. Why? Because students who are exposed to scripted materials systematically score higher.

Higher education has so far been immune from these forces of scripted pedagogy. But beware: the benefits of reduced teaching discretion may soon extend to higher education. Truly great teachers will always be able to beat a script. But I tend to think that about half of teachers are below average. Even some university-level material might be better taught by statistically validated scripts. In a world with super crunching, dispersed discretion is not all it was once cracked up to be.

Ian Ayres is the William K. Townsend professor of law at Yale Law School and a professor at Yale's School of Management. Super Crunchers: How Anything Can Be Predicted is published by John Murray, £16.99.