Consider the recent history of homelessness in the United States. In the early 1980s, an advocate for the homeless named Mitch Snyder took to saying that there were about 3 million homeless Americans. The public duly sat up and took notice. More than 1 of every 100 people were homeless? That sure seemed high, but . . . well, the expert said it. A heretofore quiescent problem was suddenly catapulted into the national consciousness. Snyder even testified before Congress about the magnitude of the problem. He also reportedly told a college audience that 45 homeless people die each second—which would mean a whopping 1.4 billion dead homeless every year. (The U.S. population at the time was about 225 million.) Assuming that Snyder misspoke or was misquoted and meant to say that one homeless person died every forty-five seconds, that’s still 701,000 dead homeless people every year—roughly one-third of all deaths in the United States. Hmm. Ultimately, when Snyder was pressed on his figure of 3 million homeless, he admitted that it was a fabrication. Journalists had been hounding him for a specific number, he said, and he hadn’t wanted them to walk away empty-handed.

It may be sad but not surprising to learn that experts like Snyder can be self-interested to the point of deceit. But they cannot deceive on their own. Journalists need experts as badly as experts need journalists. Every day there are newspaper pages and television newscasts to be filled, and an expert who can deliver a jarring piece of wisdom is always welcome. Working together, journalists and experts are the architects of much conventional wisdom.

Advertising too is a brilliant tool for creating conventional wisdom. Listerine, for instance, was invented in the nineteenth century as a powerful surgical antiseptic. It was later sold, in distilled form, as a floor cleaner and a cure for gonorrhea. But it wasn’t a runaway success until the 1920s, when it was pitched as a solution for “chronic halitosis”—a then obscure medical term for bad breath. Listerine’s new ads featured forlorn young women and men, eager for marriage but turned off by their mate’s rotten breath. “Can I be happy with him in spite of that?” one maiden asked herself. Until that time, bad breath was not conventionally considered such a catastrophe. But Listerine changed that. As the advertising scholar James B. Twitchell writes, “Listerine did not make mouthwash as much as it made halitosis.” In just seven years, the company’s revenues rose from $115,000 to more than $8 million.

However created, the conventional wisdom can be hard to budge. Paul Krugman, the New York Times columnist and devout critic of George W. Bush, bemoaned this fact as the President’s reelection campaign got under way in early 2004: “The approved story line about Mr. Bush is that he’s a bluff, honest, plainspoken guy, and anecdotes that fit that story get reported. But if the conventional wisdom were instead that he’s a phony, a silver-spoon baby who pretends to be a cowboy, journalists would have plenty of material to work with.”

In the months leading up to U.S. invasion of Iraq in 2003, dueling experts floated diametrically opposite forecasts about Iraq’s weapons of mass destruction. But more often, as with Mitch Snyder’s homeless “statistics,” one side wins the war of conventional wisdom. Women’s rights advocates, for instance, have hyped the incidence of sexual assault, claiming that one in three American women will in their lifetime be a victim of rape or attempted rape. (The actual figure is more like one in eight—but the advocates know it would take a callous person to publicly dispute their claims.) Advocates working for the cures of various tragic diseases regularly do the same. Why not? A little creative lying can draw attention, indignation, and—perhaps most important—the money and political capital to address the actual problem.

Of course an expert, whether a women’s health advocate or a political advisor or an advertising executive, tends to have different incentives than the rest of us. And an expert’s incentives may shift 180 degrees, depending on the situation.

Consider the police. A recent audit discovered that the police in Atlanta were radically underreporting crime since the early 1990s. The practice apparently began when Atlanta was working to land the 1996 Olympics. The city needed to shed its violent image, and fast. So each year thousands of crime reports were either downgraded from violent to nonviolent or simply thrown away. (Despite these continuing efforts—there were more than 22,000 missing police reports in 2002 alone—Atlanta regularly ranks among the most violent American cities.)

Police in other cities, meanwhile, were spinning a different story during the 1990s. The sudden, violent appearance of crack cocaine had police departments across the country scrapping for resources. They made it known that it wasn’t a fair fight: the drug dealers were armed with state-of-the-art weapons and a bottomless supply of cash. This emphasis on illicit cash proved to be a winning effort, for nothing infuriated the law-abiding populace more than the image of the millionaire crack dealer. The media eagerly glommed on to this story, portraying crack dealing as one of the most profitable jobs in America.

But if you were to have spent a little time around the housing projects where crack was so often sold, you might have noticed something strange: not only did most of the crack dealers still live in the projects, but most of them still lived at home with their moms. And then you may have scratched your head and said, “Why is that?”

The answer lies in finding the right data, and the secret to finding the right data usually means finding the right person—more easily said than done. Drug dealers are rarely trained in economics, and economists rarely hang out with crack dealers. So the answer to this question begins with finding someone who did live among the drug dealers and managed to walk away with the secrets of their trade.

Sudhir Venkatesh—his boyhood friends called him Sid, but he has since reverted to Sudhir—was born in India, raised in the suburbs of upstate New York and southern California, and graduated from the University of California at San Diego with a degree in mathematics. In 1989 he began to pursue his PhD in sociology at the University of Chicago. He was interested in understanding how young people form their identities; to that end, he had just spent three months following the Grateful Dead around the country. What he was not interested in was the grueling fieldwork that typifies sociology.

But his graduate advisor, the eminent poverty scholar William Julius Wilson, promptly sent Venkatesh into the field. His assignment: to visit Chicago’s poorest black neighborhoods with a clipboard and a seventy-question, multiple-choice survey. This was the first question on the survey:

How do you feel about being black and poor?

1. Very bad

2. Bad

3. Neither bad nor good

4. Somewhat good

5. Very good

One day Venkatesh walked twenty blocks from the university to a housing project on the shore of Lake Michigan to administer his survey. The project comprised three sixteen-story buildings made of yellow-gray brick. Venkatesh soon discovered that the names and addresses he had been given were badly outdated. These buildings were condemned, practically abandoned. Some families lived on the lower floors, pirating water and electricity, but the elevators didn’t work. Neither did the lights in the stairwell. It was late afternoon in early winter, nearly dark outside.

Venkatesh, who is a thoughtful, handsome, and well built but not aberrationally brave person, had made his way up to the sixth floor, trying to find someone willing to take his survey. Suddenly, on the stairwell landing, he startled a group of teenagers shooting dice. They turned out to be a gang of junior-level crack dealers who operated out of the building, and they were not happy to see him.