It's curtains for the gadfly of the piece...

Theory is dead, rejoices Mark Bauerlein, but it has dragged higher learning with it to the grave after years of dismembering our most prized literature. The Times Higher, though, finds that most scholars do not agree

For many decades, academics envisioned the life of the mind as a quest. They yearned to solve the mysteries of the past and to explore complex artworks, honing the tools of scholarship to understand better the human condition. Teaching was a noble vocation of transmitting wisdom and knowledge to rising generations.

In the 1960s and 1970s, however, a small revolution took place.

Tradition, history and art were subordinated to a collection of thinkers and arguments that went under the name of "theory". They provided abstract rules and explanations for how human events unfolded and artistic creation happened. Theory had all the attractions of being conjecture-clean, clever, overarching - but it squeezed the vitality and unpredictability from human achievement like juice from a lemon. Instead of reading classic poems and novels, scholars mastered theories of literature. Instead of learning the details of a historical record, they acquired a theory of historical change. Forty years on, the results are in. Learning has declined and the humanities are an impoverished field. The outcome could have been foreseen, for what is the theory of history and art, or of love, gardening or health, for that matter, compared with present and past realities? But the enthusiasms of the moment were too strong.

I entered graduate school in the early 1980s for the same reason others did. During my teens, literature, history and philosophy grabbed me enough to keep me in school beyond the undergraduate round of survey classes, short papers and exams. The books that stood out were too important to leave behind after graduation. So many passages and characters had got under my skin, from Edmund in King Lear avowing "Thou, Nature, art my goddess" to Ivan Karamazov struggling with the moral meaning of a godless world. I had no ambition for an academic career, but the call of Big Ideas and brilliant writers was irresistible.

It didn't take long to realise that other idols ruled the graduate programmes. Yes, we read Shakespeare, Hume, Austen and Lovejoy, but what we did with them depended on an entirely different group: the theorists.

Derrida, Foucault, Lacan, Adorno, Rorty, Paul de Man... they set a powerful agenda for humanistic study. Their work was complex and diverse, but what made all of them theorists was a focus on method. Instead of studying directly the contents of history and thought, they said we should examine the tools of study - terms, evidence, values and practices. Biographers, for instance, aim to record a human life; theorists step back and ponder the narrative structure of that life (or any life), the nature of historical evidence and so on. The sceptical tenor was spreading in schools from Aberdeen to Berkeley to Sydney, and those of us who enlisted out of inspiration had to change our attitude. Theory was hyper-analytical and against common sense, leaving no ordinary enjoyment untouched. The beauty of Keats's verse, the truth of Nietzsche on herd morality, the heroism of Lincoln... well, you couldn't esteem such things any more. By their own declaration, the theorists probed the basic elements of language, culture and ego, and to affirm something as conventional as beauty was to be pitiably square and naive.

Some were turned off, but many were intoxicated by the approach. Indeed, it is hard for non-academics to grasp how heady those conversations in the seminars and the student lounge could be. The classic writers were still essential, but the theorists were daring and radical, and the mention of them made the energy level in class discussions jump. If a fellow student spoke about Gulliver's Travels by borrowing from Swift's life, one could cite Derrida on how outside materials don't reveal the meaning of a work but close off multiple meanings to "privilege" just one. Or one could steer the talk towards Lacan on aggression, then apply it to the world of the Houynhnms and Swift's portrayal of mankind as yahoos. Or one could take a postcolonialist tack and recount Gulliver's efforts to "go native" (ridiculous, to be sure, but one heard much worse). At that point, the colloquy would turn theoretical, with people taking sides.

Usually, the theorists would win. Traditional scholars fell back on custom and textual evidence, while theorists and their disciples enjoyed the thrill of roguish poses and weighty topics - Derrida on Western thought, Foucault on madness and civilisation, de Man on irony and death. They made our lives in the library seem adventurous and superior. Think, for instance, how Foucault flattered the student ego. In a series of books, he argued that the freedoms we cherish in bourgeois society, along with the liberal reforms of the Enlightenment, were in fact subtle forms of social control working through heightened surveillance and low-intensity coercions. The compliment this outlook paid to weary junior scholars struggling to find a place in the world was hard to withstand. While the rest of society accepted modern life and muddled through, the clear-eyed minds we fancied ourselves to be understood what was really going on.

Other sallies were just as empowering, at least in the rarefied confines of the campus. Derrida made interpretation into an escapade, with deconstruction venturing to dismantle the hallowed notions of philosophy.

Edward Said argued that the ideals of European civilisation depended on stereotypes of "orientalism" - the progress, rationality and vigour of the West confirmed by the backwardness, irrationality and femininity of the East. Such theories gave us a line of attack, a presumption of deeper understanding. The more adept we were in theorising the ordinary questions of study - how to interpret a poem, how to write a history - the more advanced we appeared.

Terry Eagleton's Literary Theory: An Introduction was published in 1983 and became one of the all-time academic bestsellers. In 1986, the president of the Modern Language Association delivered an address titled "The Triumph of Theory" and nobody quibbled. But as the years passed and theory spread through the ranks something paradoxical happened. Theory lost its cachet.

When it was a minority endeavour, it functioned as a gadfly, obnoxious sometimes, but useful for testing assumptions. When theory became a dominant habit, it lost its rationale. With nobody around to defend untheoretical positions, it had nothing more to say, no more bunk to debunk. As the numbers of old-fashioned scholars dwindled, theoretical interventions became pointless and predictable. A recent book by another president of the MLA spent pages blaming the poor reading habits of students on the New Critics, figures whose influence waned back in the 1960s. The antiquated target shows how empty theory's victory was. How many times could you "call into question" a basic assumption or "problematise" a term without sounding like a cliche?

Some thought the answer lay in a new mission for theory, so they applied it to race and gender questions. Theory's focus on power and underlying meanings seemed eminently suited to social relations, and journals and conferences seized on the connection as a logical, and just, next step. An American theorist caught the attitude when she announced to the National Endowment for the Humanities in a guest lecture in the mid-1990s: "English departments are leading a social revolution in this country." Soon enough, this, too, wore thin. Theorists might root out the subtlest forms of racism and sexism in books and cultures, but it was still an academic enterprise.

They might aim for social change, but after a few years the lectures and essays hadn't made much difference in society at all.

The test of time was undeniable. The simple truth was that the accomplishments of theory mocked its claims. Derrida and the rest spoke in grandiose terms about the implications of theoretical acumen, and their votaries echoed the tone in portentous statements. When de Man declared that "the linguistics of literariness is a powerful and indispensable tool in the unmasking of ideological aberrations", his followers repeated it as if it marked a leap in the course of human intelligence. But nobody appeared to benefit from the insight except its practitioners. Theory infiltrated the humanities, theorists found jobs and changed the curriculum, new journals and programmes were founded. But the effect beyond the campus was negligible. A few psychiatrists remained Lacanians, and some architects practised a version of deconstruction, but the influences were scattered. To proclaim theory's social impact was nothing more than a pretence.

Still, theory's influence in the university has been enormous. Even among people who've pulled away, certain axioms remain a matter of principle - for instance, the notion that sexual identity is a social construct with no biological determinants. In the absence of support from the outside world, theory has become an insider activity. And with the anti-theorists routed long since, all theory can do is rehearse the arguments made 40 years ago, the same interpretations and same conclusions. The pretexts change - Milton yesterday, Buffy the Vampire Slayer today - but the outlook doesn't.

Professors have profited from theory for a long time, and they're too comfortable and invested to have second thoughts.

But what about the legacy for which they are held responsible? That's the decisive question. How did theory fare as an education? Recent PhDs are an apt measure. It used to be that anybody with a doctorate in the humanities was expected to be familiar with the giants of Western literature and thought, from Homer and Plato to Freud and the Modernists. Specialisation came late in their training, after they'd reached a suitable level of erudition. Theory changed that, and it shows in the development of younger scholars. Instead of building a reservoir of learning, they assimilate anthologies of theory and criticism, along with a narrow set of texts to interpret. They argue endlessly over the merits of cultural studies, but ask them about Dante and they go blank. In the lectures they give and articles they write, the range of allusion, social context and intellectual history narrows by the year. They have acquired an interpretative facility - one theorist calls it "arguespeak" - but they know little about Baroque art, classical liberalism or the symbolists. They can barely teach a survey course in their own field.

It isn't their fault. They've been reared by a generation of scholars who forgot their duty as stewards of great books and great art. When we put theory ahead of tradition, we lost our sense of continuity. Theory puffed us up, cast us as an intellectual vanguard and the modest role of passing along the hallmarks of civilisation could no longer satisfy.

The arrogance was self-defeating, of course. Theory couldn't sustain the humanities by itself, and the exhilaration that brought us into the habit struck outsiders as a self-congratulatory joy carried out in an affected tongue. With the public estranged from our practice and with younger scholars not replenishing the reserves of knowledge, the humanities are a guild imploding. Theory is dead, but it has taken something much more valuable with it: higher learning.

Mark Bauerlein is professor of English at Emory University, US, and author of Literary Criticism: An Autopsy , University of Pennsylvania Press, £13.00.