A field such as physics or economics has its own vocabulary, technical terms used to describe ideas. One problem in talking about such a field or teaching it is that the terms often sound self-explanatory. I am sure there are lots of people who believe that they really understand the theory of relativity, except for the mathematical bits. “The theory of relativity says that everything is relative; I understand that.”
The theory of relativity does not say that everything is relative. On the contrary, it says that the speed of light is so absolute that it appears the same to every observer, however fast he is moving relative to the light source. It was that simple and apparently impossible fact, demonstrated by the Michelson-Morley experiment, on which the theory was built.
The same problem exists in economics. “Competition” sounds as though it should describe something like a chess game or a horse race, with each competitor trying to beat out his rivals. Such a situation exists in fields with only a small number of companies: Microsoft vs Apple, Apple vs Google, Ford vs GM. Economists call it “imperfect competition.” What we call perfect competition is a market with many firms, each so small that it can ignore the effect of any other firm on it and its own effect on the market. The wheat market is perfectly competitive, the auto market or the cell phone market is not.
The same problem exists with “market failure.” It sounds as though it means a situation in which the market fails, for whatever reason, to do something we would like done. But that is not what it means, or at least not what it should mean.
Consider a simple example. In 1908 something, possibly a large meteorite, struck Siberia, producing the equivalent of a multi-megaton nuclear explosion. The failure to evacuate the target area was a failure. Since it could have been done by the voluntary acts of those living there it could be described as a failure of the market. But it was not a market failure as the term ought to be used, since there was no way at the time of anticipating the event.
Not all failures of the market count as market failure. Equally confusing, not all market failures occur on the free market. Conservatives and libertarians sometimes describe the political equivalent of market failure as government failure. But where a failure of the political system is due to precisely the same logic as a corresponding failure of the economic system, it makes more sense to describe it as market failure on the political market. There is no central authority for defining technical terms and if there were I would not be running it. But I hope to convince you, in the course of this chapter, that my definition of market failure is more useful, leads to clearer understanding, than any definition limited to what we usually call the market.
A market failure is a situation where individual rationality does not lead to group rationality. If each individual makes the right decision, the group makes the wrong decision. In the extreme case, every individual ends up worse off than if each of them had made a different decision.
For one of my favorite examples, imagine that it is a thousand years ago somewhere in Europe. I am one of five thousand men with spears, lined up facing south. The reason we are facing south is that another army, also with spears but on horseback, is coming at us from that direction.
I do a very quick cost benefit calculation.
If we all stand our ground some will die but, with luck, we will break their charge and most of us will live. If I run, horses run faster than I do. It looks as though I should stand.
I have just made a mistake. I only control me, not we. If I stand and everyone else runs, I die. If everyone else stands and I run, reducing our army by one is unlikely to make much difference to the odds and, if their charge is stopped, I won’t be one of the ones who dies stopping it. If the line does break and run I will at least be in the lead. Whatever the rest of the army does, I am better off running than standing. Everyone else makes the same calculation, we all run, and most of us die.
Welcome to the dark side of rationality.
Readers who have studied economics or game theory have probably already encountered the simplest version of market failure, a two person game called Prisoner’s Dilemma. Two criminals, Bill and Joe, have been arrested for a crime jointly committed. The D.A. has enough evidence to convict them of a minor offense but not of a serious crime.
He makes Bill the following offer:
“If you confess and Joe does not, I will let you off with a slap on the wrist—three months in prison, a suspended sentence for the rest—and send him up for five years. If he confesses and you don’t, he will get the suspended sentence and you will be the one who gets five years.
If you both confess, you will end up with three years each. If you both refuse I will give up on the major charge and get you for a year on the minor.”
He makes the same offer to Joe.
If Joe confesses, Bill gets three years in jail if he too confesses, five if he does not. If Joe does not confess, Bill gets a year if he remains silent, three months if he confesses. Either way, he is better off confessing. The same logic applies to Joe; whatever Bill does, Joe is better off confessing. Both confess and they get three years each. If they had both remained silent it would have been only one.
Yet neither of them has made a mistake.
For a third example, consider a market failure I observed repeatedly when I was teaching at UCLA. Just south of campus was the intersection of Wilshire and Westwood, about ten lanes each, claimed by the locals to be the busiest intersection in the world. At rush hour, cars on Wilshire trying to get across Westwood and not quite making it filled the intersection, blocking the cars on Westwood trying to cross the other way. Gradually the cars in the intersection filtered out, just in time to let enough cars on Westwood into the intersection to block it the other way. If everyone had followed a policy of not pulling into the intersection unless he was sure he would be able to get through it, everyone would have gotten home sooner. But any individual who acted that way would as a result have gotten home later.
The problem in all three cases is the same. Someone is making a decision that affects both himself and other people. He makes the decision on the basis of the effect on him; everyone else acts similarly. He gains from his decision, loses from theirs. The loss is larger than the gain, so on net he, and everyone else, is worse off than if they had all acted differently. Economists describe such a situation as an inefficient outcome due to an externality. The actor ignores the external cost his act imposes on others and so takes an action which, if all costs are taken into account, ought not to be taken. If everyone makes the opposite choice everyone, at least in my examples, is better off, but each individual is better off if he makes the choice that is in his interest.
A different example of the same logic was discussed back in Chapter 34. Something could be produced, in that example a dam for flood control, that would benefit all the members of a pre-existing group of people. The producer had no way of controlling who got the benefit so could not, as with ordinary goods, make getting the benefit conditional on paying for it. The result was that the dam might not get built even if its value was greater than its cost.
Market failure, as I hope I have convinced you, is a real problem. Although imperfect solutions can sometimes be found—I described some back in Chapter 34 and could have described others—there is no guarantee that goods worth producing will get produced, that armies will stand and fight instead of running away and being slaughtered, that traffic intersections will not jam. I hope I have also convinced you, with my choice of examples, that it is not a problem limited to the free market. It can occur in any situation where individuals are making decisions that affect both themselves and others.
Considered as an Argument For Government
The central assumption of economics is rationality, that individual behavior can best be predicted by assuming that each individual takes those actions that best achieve his objectives. Obviously the prediction will not always be correct. Not only do I observe other people sometimes making mistakes, I observe myself making mistakes; even though I know I am overweight, bowls of potato chips in my near vicinity tend to mysteriously empty. I know myself well enough to predict my irrationality and try to deal with it by not having bowls of potato chips too near. But although rationality is not a perfect description of human behavior, it may be the best assumption available for predicting the behavior of large numbers of strangers. 
Rationality looks like a convincing argument in favor of institutions in which individuals are free to make their own choices, yet most economists are not anarchists or even libertarians. One reason they are not is that, even if each individual correctly acts in his own interest, the result may be worse than if each was compelled to do something else.
For economists, that is the standard justification for many, perhaps all, government actions. Public goods are underproduced, so tax people to pay to produce them—not only national defense, as discussed back in Chapter 34, but scientific research as well. Negative externalities are overproduced, so regulate pollution, tax the production of carbon dioxide to reduce global warming. Positive externalities are underproduced, so subsidize education. In some specific cases the argument is weaker than it at first appears, a point I will return to in Chapter 64. But the underlying logic is correct. It follows that even rational individuals can sometimes be made better off by restricting the choices they are permitted to make.
One response sometimes offered by libertarians is to deny the existence of market failure. There is a better one.
Market Failure as an Argument Against Government
Back in Chapter 39, I discussed the problem of rationally ignorant voters. Someone who takes the time and trouble to figure out which candidate will be best for the country and vote for him is producing a benefit that will be shared with all his fellow citizens, a public good. Rational ignorance is the underproduction of that public good.
In chapter 38, I described various government activities as legalized theft, benefiting one actor at the expense of another, and used the concept of rent seeking to show that the result might well be to make everyone, including the politicians, worse off. When you work to pass a law that benefits yourself at the cost of other people, you are producing a negative externality. Negative externalities are overproduced.
Market failure exists on the ordinary private market. Its existence implies that we could sometimes be made better if a sufficiently wise and benevolent authority made some of our decisions for us. But, in the real world, the alternative to laissez-faire is not rule by a benevolent and supremely competent dictator, it is having decisions made on the political market instead of the private market.
Market failure exists because individuals are making decisions much of whose cost or benefit goes to someone else. That situation sometimes occurs on the private market, but there it is the exception, not the rule. Most goods are ordinary private goods, so the producer can convert much of the benefit to the buyer into a benefit to himself via the price he charges. Most production uses inputs—labor, raw materials, capital, land—that the producer can only use if he compensates their owners for what they give up by letting him use them. In the standard model of perfect competition, which assumes away problems such as public goods and externalities, what the producer is paid for a good for turns out to be just what it is worth to the purchaser, what he buys inputs for to be just what they are worth to the seller, hence his private benefit is precisely equal to the social benefit, the total effect of his actions summed over everyone.
That is a simplified model of a market economy, but at least it is a first approximation. Individual actors usually receive most of the benefit and pay most of the cost of their actions, making market failure the exception, not the rule. On the political market individual actors—voters, politicians, lobbyists, judges, policemen—almost never bear much of the cost of their actions or receive much of the benefit. Hence market failure, the exception on the private market, is the rule on the political market.
Which suggests that the existence of market failure is, on net, an argument against government, not for it.
 For a more detailed analysis of the argument for the rationality assumption, see Chapter 1 of my Price Theory: An Intermediate text. The question is also discussed in my Hidden Order. The best and most interesting challenge to the rationality assumption that I have seen is Thinking Fast and Slow by Daniel Kahneman, briefly discussed in the appendix.