Thoughts on Daniel Klein's studies of economic literacy

A little while back economist Daniel Klein, a professor at George Mason University and editor of Econ Journal Watch, and Zeljka Buturovic, a researcher at Zogby International, published an article entitled "Economic Enlightenment in Relation to College-going, Ideology, and Other Variables" which presented data from a survey about how people of different ideological stances view economic questions that, Klein argued, inidcated that people who identify as politically left-wing showed less undertsanding of basic economics than those identifying as libertarians or conservatives. He's now calling his own prior claims "partially vitiated" in a new article by  Buturovic and himself, "Economic Enlightenment Revisited," based on the results of another survey which he believes shows that people of different ideologies are in fact about equally likely to believe falsehoods about economics, depending on how comfortably the answer to a particular question fits with their other beliefs. He attributes his prior results to a survey that was biased by including too many questions on issues the Left tends to be bad on and his own eagerness to believe something that supported his own assumptions. (Klein is himself a libertarian.)

I greatly admire Klein's willingness to publicly state that he believes himself to have been mistaken on a matter that he had been quite outspoken about. However, after reading his new article, I don't think he actually was wrong; at any rate, I don't think the additional data he presents tells against his own prior conclusion in the way he believes.

In each study, Klein gave the test subjects a list of statements about economics, such as "free trade causes unemployment." The survey subjects, all of whom were American adults, would then write an answer stating that they (strongly or somewhat ) agreed or disagreed with it, or weren't sure. Klein then compiled the figures for how often respondents of different political persuasions agreed with false statements or disagreed with true ones to get a sense of how well people of different beliefs understood economic issues. Conservatives and libertarians did better than liberals on the first survey. Concerned about the possibility that this result might be caused by an ideological slant to the survey items (cherry-picking subject matter so that most or all of the statements concerned areas people on the Left were especially likely to get wrong, for instance) rather than an actual difference in knowledge, he did a second survey, this time with questions intentionally designed to poke at potential conservative or libertarian blind spots. This time the results were reversed, with conservatives and libertarians scoring much lower than on the first survey.

I agree that the first survey is imperfect. In particular, the survey item "Third-world workers working for American companies overseas are being exploited" is problematic because, depending on how one interprets the word “exploited, ” agreement can be either a sign of economic ignorance (i.e. you believe that the workers would benefit economically if they had to fall back on whatever second-best option they had themselves judged inferior to the sweatshop) or an answer to a question- in this case, a moral one- other than the one the survey meant to ask and is intended to assess. (Though if you changed it to something more economics-specific, such as whether sweatshop make their workers worse off than they would be in their absence, I imagine the percentage of wrong answers would still be pretty high.) I also like the fact that the second survey includes an item about the economic effects of immigrant workers, an important and heated subject that was absent the first time around.

However, the first and second sets of questions differ in some important ways that are likely to make the first set more genuinely revealing than the second, because there were serious problems with the second set that make it questionable whether it measures what it is supposed to be measuring.

The first issue is that with the exception of the statement about sweatshops noted above the first survey consists of fairly clear, unambiguous statements like "Rent-control laws lead to housing shortages." This isn't the case with several statements on the second survey, and the result is that several statements on the second survey can't be trusted to measure what they're supposed to.

If some of my points here seem like semantic hair-splitting, it's because such hair-splitting is incredibly important in order to make sure that a survey is actually measuring what it's supposed to. Klein and Buturovic's interpretation of the new data hinges on certain quite specific assumptions about how survey takers interpreted the survey statements. If some of the survey statements are worded in a way that allows for multiple plausible interpretations of what's being asked, some of which could entail a different answer from the one treated as correct by the survey for reasons unrelated to economic ignorance, then responses to that statement on the survey can't be presumed to measure what it's trying to measure, especially if members of some of the the different groups being assessed are especially likely to take a particular interpretation. This is, I believe, the case on at least three of the survey items. One could argue about whether the word meanings I think were most likely being used by the typical respondent are the best way of using those words or not, but like the interpretation of “exploit” in the first survey that's not a question of economics.

Two statements to which the majority of libertarians gave what was judged to be a wrong response response were "Drug prohibition fails to reduce people’s access to drugs" and "Gun control fails to reduce people’s access to guns." The survey treats these statements as false, since prohibition makes the prohibited item more costly, so agreeing with them is counted as an incorrect answer in the respondent's score is.

But is it false? It depends on what "reducing access" means.

It could mean increasing the overall cost of getting drugs or guns, which is presumably the intended interpretation- but it could also be taken to mean, and in typical language is probably more likely to be taken to mean, eliminating the ability to acquire something at all or at least making it so insurmountably difficult that it is effectively impossible. Klein and Buturovic do acknowledge this problem, somewhat, but argue in defense of their interpretation of the responses that “it is reasonable to include price effects as a dimension of 'access'.”

It is reasonable, but it's also quite reasonable not to and instead treat access as basically binary, and I think it very likely that this is what most respondents- of all ideological types- were doing. In typical usage the latter meaning is probably more common- it would sound odd to most people if I said that an increase in the price of a particular class of products that I had previously been able to buy had lowered my “access” to them, unless the increase was so large that paying it was now completely out of reach. Gun and drug laws in the United States have done no such thing. Some illegal guns can be bought for less than $100, sometimes significantly less, and even people who literally own nothing but the clothes on their backs can and frequently do support illegal drug habits. The non-monetary cost of is high, due to prohibition- you have to be willing to break the law and possibly associate with dangerous characters- but it's also one that anybody can pay if they choose.

Klein and Buturovic also state they believe that their interpretation would be sound even if one doesn't interpret “access” in the way they had in mind. They don't specify what they mean by this. My assumption is that they're referring to the marginal gun owner or dug user in the absence of legal restrictions- even if “access” is treated as purely binary the total number of people with access in that sense would be decreased by prohibition because there would be some people for whom drugs or guns are now completely inaccessible who would have had access if the laws were less restrictive. Again, this is a perfectly valid way of interpreting the statement, but it's not the only valid way or the typical way. In contexts like this the word “people” is almost always means, and will be taken to mean, people in general or on average or the great majority of people- almost no one will interpret “reducing people's access” to mean depriving a few extreme outliers of access or slightly lowering the total sum total of people who have it.

Guns are readily accessible even in jurisdictions that forbid them, some of them at prices so low that anyone who can't afford one now likely couldn't have afforded one even in the total absence of gun control, and illegal drugs are so ubiquitous that the government can't even keep them out of its own prisons. In everyday English the statements "Drug prohibition fails to reduce people’s access to drugs" and “Gun control fails to reduce people’s access to guns” are true, even if there are other valid ways of interpreting the statement under which it would be false. In the absence of any indication that the respondents were assuming the definition the creators of the survey had in mind, there's no reason to assume that choosing “agree” indicates economic ignorance.

Another statement on which the study indicates libertarians were more likely than liberals or progressives to give the wrong answer was "A dollar means more to a poor person than it does to a rich person,” which the study treats as true.

If we take this to be basically a statement about the declining marginal utility of money, then this is true, but in that case it would be better to phrase such a question in terms of a single person at different levels of wealth rather than two different people. As is, the statement entangles declining marginal utility, a pretty fundamental concept that can rightly be considered a matter of basic economic understanding with the more complicated and contested subject of the validity of interpersonal utility comparisons. People who would choose “disagree” because the question uses two different men and they don't think such comparisons are possible or meaningful may or may not be correct, but even if they're mistaken being on the wrong side of a complex philosophical dispute is quite a different thing from not understanding basic economic concepts.

The other and probably bigger problem is, again, one of ambiguity: There are common uses of “means more” that do not entail that the statement is true. To say that "a dollar means more" to person A than to person B can just as easily be taken to mean that A is more frugal than B, or drives a harder bargain, or is more miserly, or makes wealth a greater priority in his life, none of which necessarily suggest that A has less money. My suspicion is that the differences in answers between people of different political persuasions boils down to philosophical differences, with libertarians being more likely to think of people in terms of their actions and liberals/progressives more likely to think of them in terms of their needs. In any case, agreement and disagreement are both defensible answers to the question as written.

Aside from ambiguous language, there's another problem with comparing some of the statements from the first survey with the second one. The statements about economics in the first survey were, with the exception already noted, quite straightforward claims on the effects of economic policies, and "agree" and "disagree" each represented distinct , dichotomous positions. You believe that minimum wages or free trade cause unemployment, or that they do not, for instance, and disagreeing with a statement from the first survey is a pretty clear statement of what you do believe. Thus, disagreeing with a wrong statement is a pretty solid indication that your beliefs on the subject are in fact correct, while agreeing with a wrong statement means that you are solidly wrong, as opposed to largely but not entirely correct, and that those who disagreed are closer to the truth than you are. This is not the case with some of the second survey.

After the gun control and drug prohibition questions, the statement which the greatest number of libertarians gave a response counted as wrong to was “When two people complete a voluntary transaction, they both necessarily come away better off.” This is something that had no analog of the first survey- it is an incorrect statement designed to closely mimic a correct one. All voluntary transactions take place because both participants believe they will come away better off from it, and- since they have direct knowledge of their own preferences, usually know more about their own situation than anyone else, and have more incentive to figure out whether the transaction is a good idea or not than anyone else- they are usually correct. The great majority of voluntary exchanges do benefit the participants- but not all, since sometimes people do things that, with the benefit of hindsight, were not a good idea even according to their own preferences and values at the time. Agreeing with the statement from the survey is thus pushing things too far.

Disagreeing with the statement, on the other hand, can mean believing anything from "They almost always come away better off" to "they never come away better off." We have no reason to believe that the typical person who correctly chose "disagree" did so because their own beliefs about the benefits of voluntary exchange are more accurate than people who chose "agree." Indeed, I would be surprised if this were the case. The claim that all voluntary exchanges turn out to be mutually beneficial is too strong, but it comes much closer to the truth than the belief common among statists, particularly of a leftist bent, that people frequently or routinely enter into voluntary agreements that cause them harm.

To a lesser extent I would make the same criticism of another item, "When two people complete a voluntary transaction, it is necessarily the case that everyone else is unaffected by their transaction." This is false, but there's no way to tell whether a person who answered "disagree' and got marked as correct was merely thinking "No, transactions can have externalities," or believes that externalities not only exist but are so common and so large that most or all seemingly private agreements are actually a public/government concern,, or chose "agree" because they believe that they or society as a whole is harmed in some moral or spiritual sense if voluntary interactions they don't like are allowed to occur.

The fact that at least one of the questions on the second survey was, by design, a "trap" set for people of a particular ideological bent in a way none of the first survey's items were is a serious problem. This is especially the case when the format is a essentially a series of true/false questions where a correct answer of "false" encompasses a broad spectrum of widely held possible answers, a great many of which are even less accurate than the false statement in the survey. Such a survey item is worse than useless for assessing how knowledgeable different groups are, because the best way for a particular group of people to be rated as highly enlightened on the subject by getting the correct answer is abject ignorance- it's only when you're in the ballpark of being right that you're at risk of agreeing with the not-quite-right claim and being marked as wrong in the survey results. It's like a test of scientific knowledge where respondents have to agree or disagree with the statement “The Earth is 5 billion years old,” which overstates the Earth's age a bit- giving the wrong answer requires knowing that the Earth is several billion years old, while someone who believes that the entire universe is only a few thousand years old will give the correct answer by disagreeing.

I think Klein and Buturovic's concern about possible bias against people on the political Left in the original survey led them to overcompensate in the second survey, resulting in the problems described. If anything, the fact that Klein and Buturovic's attempts to balance against possible bias against people left of center required them to downplay the first survey's focus on straightforward claims about the effects of economic policy in favor of more abstract and/or ambiguously worded questions and come up with questions that were specifically designed as traps for other ideologies only seems to underscore the results of the original survey; at any rate, it doesn't make Klein's original conclusion that people on the Left knew less about economics seem less plausible.

So I think Klein is wrong now about being wrong before. He attributes what he believes to be his error to confirmation bias, pouncing too eagerly and uncritically on data that seemed to reinforce his existing beliefs. Based on the two survey articles, my own impression is just the opposite. Klein goes so far to guard against his own possible bias against liberals and progressives that he ends up biasing things in their favor and against his own (at the time) interpretation of the first survey. Given Klein's demonstrated willingness to scrutinize his own position and even publicly criticize his own past conclusions, this is unsurprising. Monitoring yourself for bias in your own favor involves the same trade-off as detecting other things: the more vigilant you are to ensure that nothing gets past you and the more sensitive you are to possible signs of your quarry, the greater the chance of a false positive. The direction in which I believe Klein has actually erred is just the sort you'd want in a person involved with a publication like Econ Journal Watch- would that more people's mistakes were the result of going too far to be fair to their opponents!- but in this case his desire to ensure a level playing field has backfired.