Statistics, stories, and their tensions

The New York Times's Opinionator featured an article, John Allen Paulos' "Stories vs. Statistics", that caught my attention. I'll quote at length, if you'll forgive me for that.

[T]he notions of probability and statistics are not alien to storytelling. From the earliest of recorded histories there were glimmerings of these concepts, which were reflected in everyday words and stories. Consider the notions of central tendency — average, median, mode, to name a few. They most certainly grew out of workaday activities and led to words such as (in English) “usual,” “typical.” “customary,” “most,” “standard,” “expected,” “normal,” “ordinary,” “medium,” “commonplace,” “so-so,” and so on. The same is true about the notions of statistical variation — standard deviation, variance, and the like. Words such as “unusual,” “peculiar,” “strange,” “original,” “extreme,” “special,” “unlike,” “deviant,” “dissimilar” and “different” come to mind. It is hard to imagine even prehistoric humans not possessing some sort of rudimentary idea of the typical or of the unusual. Any situation or entity — storms, animals, rocks — that recurred again and again would, it seems, lead naturally to these notions. These and other fundamentally scientific concepts have in one way or another been embedded in the very idea of what a story is — an event distinctive enough to merit retelling — from cave paintings to “Gilgamesh” to “The Canterbury Tales,” onward.

With regard to informal statistics we’re a bit like Moliere’s character, who was shocked to find that he’d been speaking prose his whole life.

The idea of probability itself is present in such words as “chance,” “likelihood,” “fate,” “odds,” “gods,” “fortune,” “luck,” “happenstance,” “random,” and many others. A mere acceptance of the idea of alternative possibilities almost entails some notion of probability, since some alternatives will be come to be judged more likely than others. Likewise, the idea of sampling is implicit in words like “instance,” “case,” “example,” “cross-section,” “specimen” and “swatch,” and that of correlation is reflected in “connection,” “relation,” “linkage,” “conjunction,” “dependence” and the ever too ready “cause.” Even hypothesis testing and Bayesian analysis possess linguistic echoes in common phrases and ideas that are an integral part of human cognition and storytelling. With regard to informal statistics we’re a bit like Moliere’s character who was shocked to find that he’d been speaking prose his whole life.

Despite the naturalness of these notions, however, there is a tension between stories and statistics, and one under-appreciated contrast between them is simply the mindset with which we approach them. In listening to stories we tend to suspend disbelief in order to be entertained, whereas in evaluating statistics we generally have an opposite inclination to suspend belief in order not to be beguiled. A drily named distinction from formal statistics is relevant: we’re said to commit a Type I error when we observe something that is not really there and a Type II error when we fail to observe something that is there. There is no way to always avoid both types, and we have different error thresholds in different endeavors, but the type of error people feel more comfortable may be telling. It gives some indication of their intellectual personality type, on which side of the two cultures (or maybe two coutures) divide they’re most comfortable.

People who love to be entertained and beguiled or who particularly wish to avoid making a Type II error might be more apt to prefer stories to statistics. Those who don’t particularly like being entertained or beguiled or who fear the prospect of making a Type I error might be more apt to prefer statistics to stories. The distinction is not unrelated to that between those (61.389% of us) who view numbers in a story as providing rhetorical decoration and those who view them as providing clarifying information.

The so-called “conjunction fallacy” suggests another difference between stories and statistics. After reading a novel, it can sometimes seem odd to say that the characters in it don’t exist. The more details there are about them in a story, the more plausible the account often seems. More plausible, but less probable. In fact, the more details there are in a story, the less likely it is that the conjunction of all of them is true. Congressman Smith is known to be cash-strapped and lecherous. Which is more likely? Smith took a bribe from a lobbyist or Smith took a bribe from a lobbyist, has taken money before, and spends it on luxurious “fact-finding” trips with various pretty young interns. Despite the coherent story the second alternative begins to flesh out, the first alternative is more likely. For any statements, A, B, and C, the probability of A is always greater than the probability of A, B, and C together since whenever A, B, and C all occur, A occurs, but not vice versa.

It's worth noting that Demography Matters' bloggers are concerned with bridging the gap between statistics and narratives, trying to produce narratives of what's going on in our world that are firmly based on statistics--anecdotes illuminate and animate, yes, but we try to limit anecdotes to those highly specific roles. With my specific educational background--English and Anthropology were my two majors--I may be more inclined to emphasize the power of the narrative. Stories provide useful and--I'd argue--necessary framing, very often inspiring statistical research, explaining it, and showing how and why these statistics are used. Certainly Eurabia or hyperaged societies or explanations for low fertility can't be examined--proven, disproven, whatever--without these narratives. But then, I quite agree you also need empirical evidence. Obviously.

It can be a difficult thing to do. What are your thoughts on the issue?

On the problems of an Iraqi census

I've recently written about how the gutting of Statistics Canada's long-form census is an act of little value. Just recently, I came across an argument that made something of a case that it was a very good thing Iraq's unable to undertake a census at all. Reuters described the situation this January.

The long-delayed count, which may shut down the country for two days in October, is also expected to determine how many Iraqis live abroad and how many have been forced to move within Iraq in seven years of war, census chief Mehdi al-Alak said.

The census was postponed for a year over worries it was being politicized. Ethnic groups in contested areas like the northern city of Kirkuk, home to Arabs, Kurds, Turkmen and a valuable part of Iraq's oil fields, opposed it because it might reveal demographics that would undermine political ambitions.

The count could provide answers or create more squabbles in a diverse nation riven by sectarian violence following the U.S. invasion in 2003 and now trying to bolster fragile security gains while deciding how to share out its vast oil wealth. Iraq has the world's 3rd largest crude oil reserves.

The autonomous Kurdish region in the north claims Kirkuk as its own. The census will determine whether Kurds are the biggest ethnic bloc in the city, which could bolster that claim.

It will also find out how many people live in Iraqi Kurdistan, which will define its slice of central government revenues, currently 17 percent. If the census finds Kurds are a greater percentage of the total population, the constitution says the region gets more money, and retroactive payments.

What it won't do, Alak said, is attempt to determine which of the hotly disputed areas belong to whom.

"It is not our business to decide their destiny," Alak, the head of the Central Organization for Statistics and Information Technology (COSIT), said in an interview this week. "We count the people in the province where they live. Deciding the destiny of the areas is the business of the politicians."

The census will be the first to include the Kurdish region since 1987. A 1997 census counted 19 million Iraqis and officials estimated there were another 3 million in the Kurdish north.

The current national population is believed to be "not less than 30 million," Alak said.

The census didn't happen, actually, for the reasons of ethnopolitical rivalry that Joel Wing described at his blog as well as the state's bureaucracy. Foreign Policy's Joost Hiltermann argued that, notwithstanding the census' importance in national planning, the census' failure was good inasmuch questions on ethnicity and language would be too polarizing.

The Iraqi census stands to play a critical role in the country's development. Its data will help in drawing electoral districts, allocating funds, projecting future population growth, and planning education, public health, housing, transportation, and other essential elements of a well-regulated state. Particularly in Iraq, which has witnessed several false starts in reconstruction following the 2003 invasion, having accurate socioeconomic data will be indispensable to sound economic planning.

But there's reason to believe that this census, as it is currently designed, will polarize rather than unify Iraqi society. The problem lies in a question that asks Iraqis to define their ethnicity, aiming to get a sense of how big the country's various ethnic groups are. Although such a question will no doubt provide interesting information for academics and analysts, it is not in Iraq's national interest and risks destabilizing some of Iraq's most sensitive hot spots.

The ethnicity question is particularly likely to inflame passions in areas that Kurdish leaders have said they want to incorporate into the federal Kurdistan region in northern Iraq. Along with Kurds, these areas are home to a diverse population of Arabs, Turkmens, and smaller minorities, all of which have been engaged in a tense standoff over Kurdish aspirations, which they resist almost unanimously. The situation holds the potential for violent conflict. Several incidents in these disputed areas over the past two years required U.S. commanders to establish joint military checkpoints along the so-called trigger line dividing Iraqi Army troops from Kurdish regional guards. Finding a negotiated solution to the tug of war over these areas, with the city of Kirkuk at their center, will be critical for Iraq's future.

All sides see the census's ethnicity question as a proto-referendum on these areas status. Everyone assumes that in a referendum Kurds would vote in favor of accession to the Kurdistan region while the vast majority of non-Kurds would vote against. If the population in a given area is found to be majority Kurdish, the political case for linking this area to the Kurdistan region will be greatly strengthened -- regardless of the wishes of the area's non-Kurdish population, whatever its size. The census, in other words, would increase the momentum toward a non-negotiated solution of these areas' status via an ethnically driven, zero-sum-game plebiscite. Going forward with the ethnicity question intact, then, would almost certainly lead to an Arab and Turkmen boycott, as well as popular protests in disputed territories, likely culminating in violence.

Even questions of language, obviously relevant for the provision of government services like education, should be dropped else the exercise create too many disputes for Iraq to survive.

There may be something to this argument. In Yugoslavia, the various republican and national censuses in the Communist era provided ample data for disputes over the direction of demographic changes and their import, helping to fuel various conflicts until, fittingly, the outbreak of war made the 1991 census a
partial one that excluded Kosovo and other parts of Yugoslavia on account of boycotts and conflict. I can't help but feel this parallel doesn't say good things about the survivability of Iraq, mind.

On the necessity of immigrants

Anti-immigrant sentiments aside in immigrant-receiving countries at all levels of economic development, the necessary existence of 3D jobs--alternatively Dirty, Dangerous and Demeaning, Dirty, Dangerous and Demanding or Dirty, Dangerous and Difficult) and the reluctance of natives to take these jobs ensures that immigration will continue at some level. Gregory Viscusi's Bloomberg BusinessWeek article "Immigrants in the West Aren't Going Away" makes this point effectively.

Since arriving in France from Mali six years ago, Youba Soumbounou has sorted trash at factories and warehouses near Paris. He got his job by using fake papers and has been told five times by police to leave the country.

His employer, which he asked not be identified, didn't fire him. Instead, it has joined other French waste-treatment companies such as Veolia Environnement (VE) and Derichebourg in seeking legal residency for laborers they need. "If we didn't have access to foreign workers, we simply wouldn't be able to do our work," says Pascal Decary, head of human resources at Paris-based Veolia Propreté. The company found out last year that 18 of its workers have phony documents, Decary says. "There was never any question of us abandoning people who have worked hard for us all these years, doing jobs that natives don't want to do."

[. . .]

"If you look beyond the noise, you don't hear anyone saying there shouldn't be any immigration at all," says Jean-Pierre Garson, head of the Organization for Economic Cooperation and Development's international migration unit in Paris. "They can't. Parts of the economy would grind to a halt."

A 2008 report by the British Parliament says 17 percent of the U.K.'s economic growth in 2004 and 2005 was the result of immigration. The Washington-based Center for American Progress, run by John Podesta, who was chief of staff to former President Bill Clinton, said in January that making it easier for undocumented workers to gain residency and attracting guest laborers would add $1.5 trillion to the U.S. economy over the next 10 years.

Whether to do undesirable jobs or pay into weakened pension funds, workers from poor nations are needed by the West, says the OECD's Garson. "Decisions are often made on the basis of the emotions of the day," says Ben Noteboom, chief executive officer of Randstad Holding, the world's second-largest staffing company. "Yet that emotion will fade away because in the end, when we are in a hospital, we need nurses."

For immigrant-receiving countries, the problem may soon be not an excess of immigrants but a deficit of needed immigrants. Even accounting for the likelihood that low-fertility countries may be just as likely to produce a given volume of emigrants as a high-fertility country, the world's demographic transition will eventually lead to a contraction of the world's working-age population. In these circumstances, countries offering less attractive conditions for migrants will lose out to the ones which offer better deals. Future, take care.