Wednesday, June 21, 2017

Attempted Noah Smackdown, illegal immigration edition


In February, I wrote a Bloomberg View post called "The Myth of the Immigration Crisis" that got a fair bit of attention. In it, I wrote:

Illegal immigration to the U.S. ended a decade ago and, according to the Pew Research Center, has been zero or negative since its peak in 2007: 


About a million undocumented immigrants left the country in the Great Recession. But even after the end of the recession, illegal immigration didn’t resume.
Now, my Twitter buddy Lyman Stone of the USDA has written a post alleging that my post is "bad" and "false". Well, my mom always told me "Son, don't **** with the USDA," and that advice has served me well for many years. However, given the importance of this issue, I may have to ignore my mother's wise words, and rebut Lyman's post. Which won't be that hard to do, because Lyman, being the perspicacious fellow he is, in fact agrees with me on almost every substantive point.


In which Lyman agrees with me on essentially everything important

I'm just going to shamelessly cherry-pick the parts where Lyman agrees with me and then goes on to cite more evidence in support of my thesis:
[Noah's evidence shows] that the illegal immigrant population has fallen since its peak. I 100% agree there. He’s totally correct. The stock of unauthorized residents in the US is almost certainly well below historic highs... 
Pew gets their estimate [of the number of unauthorized immigrants] by starting from American Community Survey 1-year estimates of the foreign-born population, then subtracting naturalized citizens. Then they use non-ACS data to estimate how many non-citizens are lawful permanent residents (LPRs) or legal temporary residents (LTRs). The residual must be unauthorized residents. 
This is the best method we have available and Pew does very good work. I have no criticism of Pew’s estimates insofar as they go.. 
Now, again, we can say with substantial confidence that the illegal immigrant population was declined since 2007... 
Let me be clear. I think Noah is [quite a handsome dude, and is also] correct that net migration of illegal immigrants has been negative in some periods since 2007. And I am very confident that he is correct that the illegal immigrant population is falling... 
What frustrates me is that Noah’s basic point, that illegal immigration is a vastly smaller problem now than 10 or 15 years ago, is totally correct. There’s tons of data to support it...He could have just shown the trend in border apprehensions, or shown the illegal immigrant share of the population, or other kinds of data. If he really wanted to be clever, he could have just lined up border apprehensions with deportations by fiscal year to see what direct migration trends might look like...
OK, I might have taken a few liberties there with the brackets, but the point is, Lyman agrees with me that according to the best estimates we have available, the population of unauthorized immigrants in the U.S. has fallen from its peak. Given that he agrees with both my thesis and the substance of my point, it strikes me as a bit odd that he characterizes my post as "false" and "bad", but as a man who once pasted Paul Krugman's head on a giant cartoon robot, I probably shouldn't criticize bloggers' use of hyperbole.

Lyman is also right that if I expressed the unauthorized population as a percent of the total, the decline would be even more stark. I'm not sure what increased border apprehensions tell us.

So, to reiterate, Lyman agrees with my basic point. The rest of his post consists of A) quibbles about vocabulary and messaging, B) a dubious point about error bars, C) an interesting but ultimately non-game-changing point about mortality, and D) bikini pics of Jim Heckman from 1971.

Well, no, not (D). Lyman's many things, but he's no monster.


Like, dude, what does "illegal immigration" even mean? 

First, note that following Bloomberg convention, I say "unauthorized immigrants" as the noun and "illegal immigration" as the verb. Because an act can be illegal, but a person can't (though I'm sure Jeff Sessions is working on it). So git off my back, y'all SJWs.

Anyway, when we talk about "the amount of illegal immigration", what does that mean? It could mean a couple things:

1. Gross illegal inflows: The number of people who enter the U.S. illegally or overstay their visas over a given period of time

2. Net migration of unauthorized immigrants: The number of people who enter the U.S. illegally or overstay their visas, minus the number of unauthorized residents who exit the country, over a given period of time

What the Pew numbers report, and what I reported, was neither of these. I reported the net change in the unauthorized resident population, That is similar to #2 above, but also includes the effect of mortality (as I'll talk about in a bit).

Anyway, which number do people think of when they hear "illegal immigration"? I'm sure some people do think of the first one. If you're a law-and-order type who is really upset about our porous border, then I'm sure you care about gross flows across that border. Lyman thinks that gross illegal inflows = the One True Definition of the term "illegal immigration":
The point is, everyone who works in this field, all the actual experts, including the folks at Pew whom Noah cites, use “illegal immigration” to refer to inflows which do not have legal authorization. That’s what the term means. It’s not just me. Here’s dictionary.com:


It means inflows. Exclusively.
Well, call me a lawyer, but it seems to me that if you're going to cite dictionary.com to tell you what "illegal immigration" means, you should at least use the dictionary.com page for "illegal immigration" (which BTW doesn't exist).

But that's not the point. The point is come on, brah, my Bloomberg post wasn't fooling anybody. First of all, I define exactly what I mean by "illegal immigration", because the graph is labeled "Annual change in unauthorized immigrant population". It's right there in the graph! I defined my terms! Neener!!

Second of all, that graph has negative numbers on it. How big of a critical theorist dum-dum do you have to be to think a negative number represents gross inflows? Gross inflows can't go negative! They are bounded below by zero! They are defined on the set Z+! Is there someone out there looking at my chart and mistakenly believing that half a million antimatter people snuck across the border in 2008??

God, I hope not. Please let there not be such a reader. But if there is, I'm not sure what it would take on my part to avoid misleading him.


OK, down to brass tacks. What number should we care about here?

Like I said, if you're the type of person who lies awake at night fuming that someone managed to sneak past the almighty Border Patrol unnoticed, then you care a lot about gross illegal inflows. I don't, really. Oh, I think there are a few reasons to care - linguistic assimilation, for example. If the unauthorized population keeps getting switched out, it'll slow the rate at which that population becomes proficient in English, the language of dubbed anime American business and culture. In fact, that's probably one reason unauthorized immigrants tend to assimilate more slowly.

But overall, what I mostly care about - and what I think everyone else should mostly care about - is the stock of unauthorized immigrants living in the country at any given time. First of all, this is what should matter for labor markets. The data has convinced me that the labor market impact of low-skilled immigration is small, but I'm not 100% certain of that, and even a small negative impact on America's most vulnerable workers is bad. But it's the stock, not the gross flow, of unauthorized immigrants that should determine the severity of labor competition faced by low-wage American workers.

Also, the stock is what matters for the welfare state. Low-skilled immigrants probably take as much or more in govt benefits as they pay into the system in taxes, so unauthorized immigrants put pressure on the sustainability of the welfare state. But again, it's the stock, not the gross flow, that matters for welfare payments.

So if what I care about is the stock, why do I talk about changes in the stock? Why do I act like there's no problem just because the stock is hovering at a constant number?

It's all about urgency. If the total number of unauthorized immigrants isn't increasing, there's no reason to panic. There's no reason to start calling for a big shift in our immigration policy. The Obama approach of increased border security and increased criminal deportations is doing a great job of keeping the U.S. from being swamped by illegal immigration, even if it didn't do a great job of winning anti-illegal-immigration voters over to the Democrats.

So I feel like by using the term "illegal immigration" to mean "the change in the total number of unauthorized residents", I was getting at the quantity that really matters.


Did I ignore margins of error?

Yeah. I reported point estimates without talking about margins of error. Let he who is without sin cast the first Stone.

SEE? It was a pun! Lyman's last name is Stone! Get it?? BUAHAHAHA

...OK, anyway. Let's talk about margins of error. Lyman produces a graph of year-on-year changes in unauthorized immigrant population with some error bars he cooked up:


Wow, what looks like zero could actually be an increase of half a million unauthorized immigrants per year, right??

Wrong. The errors don't add up over time. If Pew were measuring border crossings and using that to infer the total unauthorized population, then yeah, the errors in their estimates would cumulate. But what they're doing is re-measuring the unauthorized population over and over each year. Which means that if we want to measure the change in total unauthorized population between Time A and Time B, we don't care about any of the measurement errors in between A and B.

(Random note: Blogger's spell-checker doesn't recognize "cumulate". What sort of fallen world do we live in?)

OK, anyway. I don't know how Lyman produced the graph you see above, since he doesn't include his methodology. It sort of looks like he just added up Pew's standard errors on the yearly population estimates for each pair of years, and then added maximum potential rounding error to each year. But I am an honorable man, and Lyman is an honorable man, and I would never accuse him of making such an undergrad-level math mistake. 

In any case, let's talk about how you calculate the error bars of a difference. 

So, let A be the total number of unauthorized immigrants in 2007, and B be the number in 2014. What we're interested in is the quantity B - A. We have unbiased estimates of B and A, and some random measurement errors e_B and e_A:

Bhat = B + e_B

Ahat = A + e_A

Suppose we want the variance of the difference between our two estimates: Var(Bhat - Ahat) = Var(e_B - e_A) = Var(e_B) + Var(e_A) -2Cov(e_B,e_A)

So the more correlated our measurement errors are between 2014 and 2007, the smaller the error bars will be on the difference of the two estimates. This is a fancy way of saying that if we miscount by the same number of people each year, we get the change in the total number of people exactly right, even if the amount we miscount by is huge. 

I was going to try to write down an expression for serially correlated errors here, with an autocorrelation coefficient of f, so I could use Cov(fe,fe), but I was too lazy.

So the more serially correlated the errors in the ACS and CPS estimates (which are used to derive Pew's estimates) are, the smaller the error bars should be on the difference between the estimates for two years. And I do suspect there is some serial correlation there. Suppose there's some group of unauthorized immigrants that these surveys reliably miss every year. Even if these groups are large - say, 1 or 2 million people - the fact that they aren't measured adds only a little bit to our uncertainty about the change in the total unauthorized population. (That little bit comes from the change in that unobserved subpopulation itself.)

So that's one potential problem with what Lyman is doing here. A second is that he discusses rounding errors. Pew's numbers are rounded to the nearest 100,000, meaning that they can be off by 50,000 in a given year. But those rounding errors obviously don't add up over time! When calculating the change in the unauthorized population over N years, you only have two rounding errors, not N rounding errors. 

The third thing Lyman overlooks is that the intervening years between 2007 and 2014 actually do contain some information. They show remarkable stability


If the measurement error of the yearly first differences were really on the order of 400,000 per year, as Lyman's graph shows, we'd expect to see the numbers jump around a lot more than they do. In fact, after 2008, we never see changes that big. This means Lyman may have made a mistake in how he calculates his error bars, but it also means that Pew may have overestimated its own error bars for the yearly population numbers. (Unless ACS and CPS are smoothing these numbers year to year in some way I am unaware of, which would be a bit naughty!)

Anyway, it's possible that measurement error concealed a moderate amount of (net) illegal immigration between 2009 and 2014. But given the likelihood that the ACS and CPS miss a lot of the same people each year, the number is unlikely to be big. And there's still basically no doubt that (net) illegal immigration was negative between 2007 and the present.


Outmigration to Heaven

As Lyman points out, there are multiple reasons the unauthorized population can decline. One is that people leave the country. Another is that people die. In my Bloomberg View post, I ignored mortality.

The reason I ignored it was that I didn't think of it (an excellent reason, if I do say so myself). But thinking about it later, I confirmed that it isn't that big of a deal, quantitatively. 

The crude death rate for unauthorized immigrants is about 3.9 per 1000, according to this random paper that I got by googling, i.e. The Most Reliable Source Ever. That's close to Lyman's own guess of about half the crude death rate of the U.S. as a whole. Using Pew's point estimates for the total unauthorized population each year, and again ignoring error bars, that's about 357,000 unauthorized immigrant deaths between 2007 and 2014, and about 264,000 between 2009 and 2014.

Let's compare this to the difference in Pew's totals for those years (i.e. what I called "illegal immigration"). The difference between 2007 and 2014 goes from -1.1 million to around -743,000 - still a very substantial decrease. The difference between 2009 and 2014 goes from -200,000 to around +64,000, turning a small decrease into a very small increase.

I still feel justified in saying that (net) illegal immigration halted between 2009 and 2014. As Lyman writes:
Mortality, like adjusting for ACS population estimation errors, has only a small impact.
The impact on Lyman's and my productivity is more substantial.


Summing up

So, ladies and gents and zombie thralls of the USDA Advanced Weapons Program, besides a general agreement with my thesis and main point, what we have here are:

1. A vocabulary complaint

2. An insistence that I'm focusing on the wrong number, which may or may not also be a vocabulary complaint

3. The very real fact that I didn't mention error bars (Bad social science columnist! Bad!)

4. Some dubious and mysterious calculations of error bars

5. That time I almost made a Cov(fe,fe) joke

6. A real, useful point about mortality, which I forgot because I'm a critical theorist dum-dum, but which isn't hugely important in the quantitative sense


I don't feel that I come out of this one looking too bad. 

*turns around and sees horde of zombie USDA attack cows converging*

Gulp.

Sunday, June 11, 2017

Is economics a science?


While I was in Norway to give a talk about macroeconomics, an interdisciplinary group at the University of Oslo also invited me to give a talk about whether economics is a science or not. That's an impossible question, of course, since there's no official definition of what "a science" is. But I did have some thoughts on the matter. Here are the slides from the talk:
   

These slides don't speak for themselves quite as much as the macro slides did, and the topic is much broader and more vague, so I'll turn it into a full post. This post mostly just explains the slides.


What the heck is a "science"?

No one knows. Because no one has ever really been able to make one dominant definition of science stick. Some people define it as a method (e.g. Popper), some as a sociological phenomenon (Kuhn, Lakatos), and others don't even see much need for a definition (Feyerabend). And there are plenty of other opinions too.

So the argument over whether economics is "a science" will never be resolved.

It certainly isn't a lab science; though econ experiments are interesting and can be helpful, they lack ecological validity - i.e., what we really want to know is how the big, messy, complex world works in practice. So lab experiments by themselves aren't going to get the job done, or even come close.

Therefore, if econ is to be a "science", it has to be a largely empirical "science". And since empirical research and lab research are fundamentally different ways of understanding the world, that means some people will always say econ isn't a science. But if you accept that empirics can be "scientific", then econ has a chance.

Anyway, I do think there are three trends in economics that most people will agree are making it more scientific:

1. Theories with strong predictive power

2. Less theory, more empirical work

3. The "credibility revolution" in empirical economics


Theories that work

Lots of people claim that social science can't be a "science", because human beings don't obey precise mathematical laws. That just seems silly to me. First of all, plenty of natural sciences don't involve precise mathematical laws - what's the mathematical law describing how food passes through the digestive tract? Second of all, there are plenty of social science theories that are written in quantitative form - equations, numbers, and all that - and that have consistent ability to predict human behavior.

In the slides I give four examples from economics. These are:

1. Auction theory

2. Matching theory

3. Discrete choice models

4. Gravity trade models

This is an eclectic mix of theories. One is explicitly neoclassical (discrete choice), while another relies on individual rationality (auction theory). One is basically an algorithm for central planning (matching), which makes it a close cousin to lots of models in operations research (which, by the way, also frequently are able to predict human behavior with quantitative precision). And one, the gravity trade model, is a "big" theory that successfully predicts patterns of international trade involving billions of individuals.

So the idea that social science can't predict human behavior is pretty conclusively disproven, just by these four examples. There are, of course, many more such examples, most of which are probably not from econ.

But the fact that econ is making progress on this front is encouraging. Slowly, the discipline is building up a stable of models with good, reliable predictive power.

In fact, although it's a bit prosaic and boring, even the good old Econ 101 supply-and-demand model probably works great for some things. It doesn't work for labor markets, but I bet it can fairly accurately predict the effect of a Florida hurricane on orange prices.

Anyway, theories that work, and which have engineering applications beyond the halls of academia, are generally considered to be a hallmark of "real science," and people ought to know that econ is getting more and more of these theories.


The empirical revolution and the credibility revolution

In recent years, economics has become much more empirical - theory papers used to represent almost two thirds of what got published in top journals, and as of 2011 they had fallen to just over one quarter. The stereotype that economists are "mathematical philosophers" who just sit around and make theories all day is less and less true.

Meanwhile, the "credibility revolution" - i.e., the rise of quasi-experimental methods - is rapidly increasing the direct real-world applicability of empirical economics. Instead of having to use dubious structural theories as an intermediary, economics papers are cutting right to the chase - finding believable estimates of the effects of policies like minimum wage and immigration.

This is having big real-world impacts. Minimum wage studies since the 1990s have found few short-term disemployment effects, which probably helped inform the decisions of a number of cities to increase their minimum wages to $15 in recent years. So far, studies of these new measures have agreed with the earlier results - there hasn't been much disemployment.

So the rise of quasi-experiments is important and good. However, it's important to recognize the limitations of this approach. Quasi-experiments only give us local understanding of the world, not the kind of global understanding that we'd need for really big bold policy moves. In the long run, being able to deeply understand the economy will require working structural models.

The second problem is that quasi-experiments usually must be found by luck rather than purposefully implemented, and even the ones that are purposefully implemented (lotteries, RCTs) are limited in the set of things they can study. This leads to the so-called "lamppost problem," in which easy-to-study things get studied and hard-to-study things get discounted. For example, studies pretty conclusively show that the minimum wage doesn't destroy many jobs in the short run, but the idea that minimum wage constrains job growth over the long run is much harder to study using quasi-experiments. Harder to study, but still important for policy.

So although quasi-experimental results are great and the shift to empiricism is welcome, it's important to keep working on theory as well.


Ways econ could stand to be more scientific

Though it's made progress in the aforementioned areas, econ still has a number of ways it could be more "science-y".

First and foremost, econ needs to get more comfortable with the idea that data can actually kill theories. This is pretty widely regarded as a hallmark of true science - theories can't just be pure assumptions or axioms, they have to be disciplined by data. And adding bells and whistles to patch up theories only gets you so far - at some point, you have to be willing to say "Well, that theory is just wrong," and try something else.

Currently, economists in general are extremely reluctant to toss out any theories at all. Even simple, elementary theories with plausible replacements get excused and excused when they contradict the evidence. A good example is the "Econ 101" theory of labor markets - supply-and-demand might work great for the market for oranges, but it fails pretty catastrophically as a description for the aggregated labor market. Yet this model is still in extremely wide use, both formally and informally.

There are a number of other, less glaring ways that econ continues to over-privilege theory. Paul Pfleiderer notes the prevalence of "chameleon" models that are sold as unrealistic thought experiments but then used by policymakers to support their desired conclusions. Ricardo Reis laments the fact that young economists are forced to insert pointless theory sections into their empirical papers. Econ Nobel prizes are given for developing new methodologies, even if those methodologies haven't yet yielded much in the way of predictive success. And as I noted in my macro presentation, many models continue to include standard elements that are flatly contradicted by the data.

So in order to become yet more scientific, econ needs to stop putting theory on a pedestal. Instead of separating the worlds of theory and empirics, economists should insist that the two follow the same back-and-forth relation that they do in the natural sciences. Theories need to be allowed to fail when measured against data, and data needs to be used to construct new, better theories.


In conclusion

Here's the final slide from the presentation, which I think sums things up quite nicely:

Saturday, June 10, 2017

Summing up my thoughts on macroeconomics


I'd like to close the chapter of my life that involves complaining about macroeconomics. I've been out of that world long enough that it's becoming a distant memory. And much more qualified critics are on the job. Furthermore, macroeconomists I talk to - especially young macroeconomists - mostly seem to have heard and internalized all of the critiques. That doesn't mean I want to stop following developments in the macro field, but that my days as a certified "macro-basher" have come to an end.

So when the Norwegian Finance Ministry, Norges Bank and Statistics Norway asked me to give a talk about "What Has Happened in Macroeconomics (and what still needs to be done)", I viewed it as an opportunity to sum up. Here are the slides from that talk.



Enjoy.

Friday, June 02, 2017

The Shouting Class


In response to the tragic Portland stabbing, I wrote a Twitter thread praising the two men who died defending Muslim women on a train. I pointed out that one of the heroes was a Republican army vet, while the other was a liberal hippie type. I lamented that our current national political discourse so often sets decent guys like this against each other, and wished that liberals and conservatives could put aside their mutual suspicions and unite at the political level to defend the country against white supremacism, fascism, and the general madness brought on by the age of Trump. This thread was very well-received, getting about 2000 retweets, 3300 likes, and numerous mostly favorable quote-tweets.

Nevertheless, there were some who were dissatisfied with the thread. A few dozen people on the left wrote to complain that I was engaging in "both-sides-ism", i.e. putting too little of the blame for the country's woes on the average Republican voter. Some accused me of being an apologist for racism and fascism. A smaller number of responders were alt-right types wrote that the stabber had been a leftist, not a rightist. Some of these responses reached 100 likes.

Seeing this, a thought suddenly occurred to me: the ratio of people who expressed support for the thread to those who expressed annoyance with it was over 20 to 1. But if you scrolled down the thread, the ratio of words devoted to support vs. words devoted to annoyance would be more like 1 to 100. Or pixels, or square centimeters of screen space, or whatever. In other words, the "annoyed" group, though far far smaller than the "supportive" group, grabbed a vastly larger amount of attention.

And this in turn got me thinking: All of social media is like that. Not for the first time, I started to worry that social media technology is hurting our popular discourse and our national politics. But for the first time, I could connect my worry to an identifiable and plausible phenomenon with well-known causes.

I call it the Shouting Class.


Who is the Shouting Class?

Everyone has problems with something in society. And everyone sometimes complains about those problems, which Albert Hirschman called "voice". But for many people, voice is contingent - as soon as the problems are satisfactorily resolved they stop complaining and go back to living their daily lives. But a subset of people will never stop complaining. When a problem becomes less severe, they switch to a different problem. And they will always find some problem that they feel requires their vocal complaint. That subset - the people who will never stop complaining and giving negative feedback - are the Shouting Class. (Of course, this isn't really a binary distinction; there are shades of gray, as always.)

There are several reasons people may be part of the Shouting Class:

Reason 1: Idealism. Some people feel an emotional need to feel like they're improving the world. Since the world is never perfect, and fighting for a better world is intrinsic to these people's motivation (and probably their identity), they will always continue to speak up. Notice that there are many, many idealists who are not part of the Shouting Class - some express their idealism by building homes for the poor, or volunteering at an animal shelter, or working as a civil rights lawyer, or being a politician. Shouting Class idealists are only those idealists who see shouting as a key way to bring about positive change.

Reason 2: Personal Unhappiness. Research shows that negative moods make people much more likely to engage in online trolling. We also have good reason to believe that some people are just generally unhappy people - though life events make them relatively happier or sadder, their baseline is a negative emotional state that changes only very slowly. In other words, some people join the Shouting Class because they are giving vent to the negative emotion that they are constantly experiencing for reasons mostly unrelated to the problems they're complaining about.

Reason 3: Sadism. Research shows that many trolls are sadists, who delight in making other people feel uncomfortable. Since recipients of complaints and negative feedback often feel uncomfortable, joining the Shouting Class can be a way of indulging sadism.

Reason 4: Argumentativeness. Arguing is an intellectual activity, and many people enjoy intellectual activity. Some people enjoy argument specifically, while others just use it as a break from other kinds of intellectual activities (writing code, etc.). A subset of these people are "mansplainers" who just want to show off how clever they are or listen to themselves talk.

This is probably not an exhaustive list; I'm sure you can think of others. But it's pretty clear, from research and from personal experience, that there are a few people in society who fall under one or more of these categories.


Shouting Class case study: Me

I'm obviously part of the Shouting Class. I write a blog in which I often complain about stuff, and I have a Twitter account where I often complain about stuff. So some of my characterization of the Shouting Class is just introspection, coupled with the realization that there are other people out there who are like me in various ways.

So it's no coincidence that all of the above motivations for Shouting have applied to me at one time or another. I'm a fairly idealistic person. Like everyone, I was pretty unhappy in grad school, which I think had a lot to do with why I started blogging and tweeting. As a teenager, I delighted in flustering people in anonymous internet forums. And many people tell me that I am an argumentative guy. :-)

Although I feel a lot less of a desire to complain, criticize, and confront people online than I used to - I am not as much of a Shouter as before - I doubt this desire will ever completely go away. But when I look at my friends and family, very few are like me in this regard. They occasionally vent about their problems to people they know, but going on social media and giving people a hard time about things just isn't something they do.


Social media reduces the costs of joining the Shouting Class

Above I listed some of the possible reasons to join the Shouting Class - i.e., the benefits. But there are also costs. Time and effort are one cost. Reputation risk is another - if everyone knows you as a complainer, you may have fewer friends, build fewer useful business connections, or find yourself signing divorce papers. Also, trolling people in real life can lead to getting punched in the face.

Social media changes all this. First, there is no risk of getting punched in the face (though you may get doxxed). Posting on social media takes almost no time or effort. And with pseudonymity, there are no reputational consequences.

Twitter is obviously much more extreme than Facebook in these regards. It's a lot easier to be pseudonymous. Posts can get shared much more quickly. And you can talk to anyone you like, unless they block you. More importantly, you can talk to the followers of anyone you like - if I'm the first to reply to a Donald Trump tweet, most of the people who click on that tweet will see my reply as well. That's massive exposure.

So Twitter, especially, gives instant safe mass exposure to anyone who wants to complain about anything. In practice, this gives an enormous bullhorn to the Shouting Class, because they are defined as the people who want to use the bullhorn.

Consider life before social media. If you wanted to complain about something, you could do it in person, but you'd suffer reputational and other risks. You could write a letter to the editor, but it was subject to editorial filtering, and you could only get letters published occasionally. Same with calling in to radio shows. You could start your own media outlet and pass it around, but dominance of large newspapers, radio shows, and TV stations limited the circulation you could achieve. Even in the early age of the internet, shouting was a lot harder than it is now. You could make a website, but because of the lack of social sharing, it would be relatively hard to gain a large audience. Forums were highly fragmented and also lacked the sharing option.

In other words, shouting just wasn't nearly as easy in 1987 or even 2007 as it is in 2017. Social media, especially Twitter, has changed the game entirely.


The Shouting Class looks larger than it really is

It's important to realize that the Shouting Class isn't really that large. Most of the people out there in the world are not the kind of people you see commenting on your tweets or posting in your Facebook politics group.

In August of 2016, NPR disabled comments on their website, after finding the following:
I did find the numbers quite startling. In July, NPR.org recorded nearly 33 million unique users, and 491,000 comments. But those comments came from just 19,400 commenters, Montgomery said. That's 0.06 percent of users who are commenting, a number that has stayed steady through 2016. 
When NPR analyzed the number of people who left at least one comment in both June and July, the numbers showed an even more interesting pattern: Just 4,300 users posted about 145 comments apiece, or 67 percent of all NPR.org comments for the two months. More than half of all comments in May, June and July combined came from a mere 2,600 users. The conclusion: NPR's commenting system — which gets more expensive the more comments that are posted, and in some months has cost NPR twice what was budgeted — is serving a very, very small slice of its overall audience.
This is why people say "Don't read the comments" - they recognize that blog comments are the domain of the Shouting Class. But social media, and especially Twitter, is like one giant comments section.

Twitter, especially, acts as an incredible force multiplier for the Shouting Class. A study by the Anti-Defamation League found that two out of three anti-semitic tweets sent in 2015 were sent by just 1600 accounts. That's an insanely powerful bullhorn for an incredibly small number of people. The advent of bots, of course, just makes the bullhorn even bigger.

My own Portland thread shows that even on Twitter, whose users are probably far more likely than the average American to belong to the Shouting Class, the ratio of quiet approvers to vocal complainers was something like 20 to 1. That's not as lopsided as the ratios for NPR comments, but it's striking.

In other words, the Shouting Class is a tiny minority of society that dominates much of our political discourse, thanks in part to the bullhorn created by the technology of social media.


The potential benefits of the Shouting Class

It would be unfair to paint the Shouting Class as a purely negative phenomenon. Many of the Shouters are not Twitter Nazis or compulsive mansplainers or pissed-off PhD students. Many are simply idealists who are sincerely trying to change the world for the better.

And sometimes that works. "Voice", to use Hirschman's terminology, is sometimes effective. For example, the marriage equality movement was quite effective in changing people's minds about gay marriage. That movement didn't rely very much on direct action, threats of violence, civil disobedience, foreign political pressure, etc. Instead, it was basically just a bunch of people speaking up. And in my opinion, they changed the world for the better in a significant way.

Now, it's important to realize that most of the people in the marriage equality movement were not, themselves, part of the Shouting Class. They were speaking up because of a specific problem, and once that problem was solved, their use of "voice" diminished. In other words, they could be satisfied.

But the members of the Shouting Class who joined that movement brought a lot of important resources to the table. They brought experience (which of course they had plenty of), energy, solidarity, time and money. Without them, it might have been much harder to win the marriage equality fight.

I'm sure if you look throughout history you'll see plenty of movements like this, where the Shouting Class served as the vanguard and the shock troops for a larger group of temporary activists. Without the idealistic Shouters always looking for new problems to be upset about, social progress might grind to a halt, and deep injustices or inefficiencies remain undiscovered for decades or even centuries.

Or it might not. Of course it's hard to know, since there's always a Shouting Class, so we don't really know what the world would be like without them. After social media has been around for a while, it'll be possible for political scientists to find natural experiments to tell whether its amplifying effect sped up social progress or not. But until then, I'd say we have to at least acknowledge the distinct possibility that a subset of the Shouting Class often does good for the world.


The Shouting Class and excess negativity

But now I'd like to talk about some of the costs of the Shouting Class and the bullhorn social media has given them. The most obvious cost is just negativity. People like it when other people agree with them and say nice things to them, and they dislike it when people disagree with them and say mean things to them. That's so obvious that I'm not even going to bother looking up research to confirm it!

So now consider how the Shouting Class creates an asymmetry here. Suppose I say something on social media that 100 people agree with and 50 people disagree with. If everyone gave me explicit feedback, I'd get 100 zaps of positive emotion and 50 zaps of negative emotion. And assuming (for the sake of simplicity) that those zaps are of the same intensity, I'll come away feeling happy on balance.

But the 100 agree-ers are much less likely to be part of the Shouting Class than the 50 disagree-ers, because the Shouting Class goes around disagreeing with stuff. So the agree-ers will simply click the "like" or "favorite" button, while the disagree-ers will write an explicit response to explain why they disagree. The way social media is set up means that the number of likes or favorites is just a number, which I can click on to show a face or profile. But all the disagreeing responses will be full, written-out things. The negative emotional zap I get from each of those certainly outweighs the positive zap I get from my favorite count going one higher.

So I end up walking away feeling bad, because I got 50 very powerful negative zaps from the Shouting Class, and 100 very weak zaps from the majority who agreed with me.

Oh, but it gets much much worse. Instead of thinking of just my one post, let's think about 1000 people, each with their own post. It's here that we see the massive emotional destructive power of the Shouting Class.

The Shouting Class is always going around looking for something to shout at. This could mean they're likely to either follow more accounts, or read more posts, or both. If that's the case, then their negative emotional impact is multiplied even further. Imagine each of the 1000 people writes one post, each of which gets a like or favorite from 100 people, each of whom only reads that one post. But imagine there are 50 Shouters who go around reading all 1000 posts, and leaving a disparaging comment with each one.

In this extreme example, the positive emotional feedback of 100,000 positive people is vastly outweighed by the negative emotional feedback of just 50 people!

Now, that's an extreme example. But it shows how the greater energy, zeal, and time commitment of the Shouting Class, combined with the bullhorn of social media, tips the balance of social media's emotional effect dramatically toward the negative.

Much research has been done on the negative impact of Facebook on mood and mental health. Most of the explanations offered have involved excessive forced social comparison and isolation from real-world interaction. But I believe the impact of excessively negative discussions could also be a reason for the effect, and I suspect that it's much worse for Twitter.


The Shouting Class and excessive social censure

If your social group is small - a handful of friends, a social club, etc. - there's a good chance that everyone in it will like you. And there's a good chance that if you only mildly annoy someone, the chance that they will complain openly to the group is low, because of the costs involved. They might take you aside and say something privately, or say nothing, but they'll only take it to the whole group if you've very egregiously offended the.

But as your social group increases in size, the number of people who might be annoyed, offended, or upset by any given thing you do goes up. With a group of 5, there's a high probability that any given thing you do or say will be inoffensive to all involved. With a group of 500, the probability of you being offensive to someone is far larger, since A) 500 is a lot more than 5, and B) the 500 is going to be a less carefully selected set than the 5.

In particular, with 500 people rather than 5, your social group is much larger to include at least one member of the Shouting Class, who by his or her nature goes around looking for things to complain about, argue with, or be offended by.

Social media expands social groups enormously. That has, pretty predictably, resulted in the development of what young people refer to as "callout culture." Conor Friedersdorf has some good interviews with college students, where they discuss the stress and social isolation of knowing that you're always vulnerable to being "called out" by a member of the Shouting Class. Some excerpts from his article:
Today, so many people are declaring so many things problematic on college campuses that the next controversy is almost impossible to predict; it is increasingly common to have done something without any fear of giving offense (say, urging a sushi night in the dining hall) only to subsequently read that the thing you’re on record having done is the object of a huge controversy elsewhere. Does the faraway story portend a future where you’ll be the one in the hot seat? 
No wonder so many students are stressed out by this. And the risk-averse have it especially hard. “I probably hold back 90 percent of the things that I want to say due to fear of being called out,” another student wrote. “People won’t call you out because your opinion is wrong. People will call you out for literally anything. On Twitter today I came across someone making fun a girl who made a video talking about how much she loved God and how she was praying for everyone. There were hundreds of comments, rude comments, below the video. It was to the point that they weren’t even making fun of what she was standing for. They were picking apart everything. Her eyebrows, the way her mouth moves, her voice, the way her hair was parted. Ridiculous. I am not the kind of person to be able to brush off insults like that. Hence why I avoid any situation that could put me in that position. And that’s sad.”
This seems like much more of a Facebook problem than a Twitter one, given that Facebook represents offline social ties far more. It's never happened to me, but I went to college before the Facebook age (yes, I'm that old). But I know a lot of people seven or ten or fifteen years younger than me who all loudly bemoan "callout culture" in very similar terms to what Friedersdorf's article describes. Even though some of them engage in it themselves.

I should note that something like this can happen on Twitter too. On Twitter, people tend to form groups not by real-life friendships, but by political affiliation. I'm a generally left-leaning kind of guy, so my followers tend to be left-leaning folks as well. So when I write something that seems vaguely nice, conciliatory, or even non-condemnatory about any Republicans, it's highly likely I'll be "called out" by left-leaning people elsewhere in the Twitterverse. On Twitter, as in the Facebook-real life nexus, many people respond to this ever-present threat by staying silent; others, by trying to hunt down and shame anyone who calls them out. Fortunately for me, I don't much mind, but I think most people mind more than I do.

Anyway, for centuries, humans have tended to have small, strongly tied inner social circles and larger, more weakly tied outer circles. We tend to respond strongly to any vocal criticism within the inner circle. But social media has thrown this concentric pattern of social circles into disarray, by making the inner circle just as "strong" as the outer one in some ways. So criticism from someone in the outer circle now often carries the social weight and destructive power that only the inner circle should really have. The Shouting Class have thus gained inordinate power over who gets liked and who gets ostracized, and much of the result seems negative so far. I'd like to see a lot more formal research on this, however, before we draw firm conclusions.


The Shouting Class and social discord

If you don't know what the Availability Heuristic is, you should! Basically, it means that when you see some examples of a thing, it makes you think that thing is common. That's a fallacy, of course; it's a case of selection bias, and is related to base rate neglect. It's easy to come up with hand-wavey evolutionary explanations for the Availability Heuristic, most of which boil down to the old adage "where there's smoke, there's fire." But research has found this bias again and again. Here are some examples of how it applies to media exposure:
After seeing news stories about child abductions, people may judge that the likelihood of this event is greater. Media coverage can help fuel a person's example bias with widespread and extensive coverage of unusual events, such as homicide or airline accidents, and less coverage of more routine, less sensational events, such as common diseases or car accidents. For example, when asked to rate the probability of a variety of causes of death, people tend to rate "newsworthy" events as more likely because they can more readily recall an example from memory. Moreover, unusual and vivid events like homicides, shark attacks, or lightning are more often reported in mass media than common and un-sensational causes of death like common diseases.
Now apply this heuristic to social media and the Shouting Class. If you see a bunch of people arguing and shouting about stuff, and you don't see the people who aren't arguing and shouting about stuff, you're probably going to think that society is a lot more discordant and divided than it really is.

To this, add the fact that the Shouting Class shouts at each other. In fact, because the Shouting Class' appetite for shouting is infinite, they will shout at each other infinitely if they encounter each other.

Before social media, it was difficult for members of the Shouting Class to find each other. I remember walking by a protest in October of 2001 and seeing pro-war and anti-war protesters shouting in each other's faces. I was stunned at the intensity of the discord. Today, I wouldn't even bat an eye. Social media, especially Twitter, immediately puts shouters from all over the world in contact with each other. The shouting is continuous.

This fuels the perception that society is irrevocably and deeply divided. Americans these days are saying that their society is more divided than ever before, and they have little optimism that the divide will be resolved any time soon.

Of course, part of that is just because of American politics - partisanship, partisan geographical sorting, and other trends have certainly contributed strongly to this feeling of division, along with things like changing racial demographics and the fallout from the Great Recession, 9/11, and Iraq.

But I bet that social media is exacerbating the effect. There was plenty of partisanship, racism, anti-immigrant xenophobia, economic pain, fear of terrorism, geographical sorting, etc. in 2012 and 2008 and even before, but we never saw anything like the bitterness and anguish and hate of the 2016 election. And that bitterness, anguish and hate has not dissipated during Trump's presidency, and it would not have dissipated during Clinton's if she had won. There is something structural at work here, and I believe social media is a part of it. Facebook's "fake news" problem gets by far the most attention, but I wouldn't underrate the toxic impact of Twitter, or the infinite, eternal battles of the Shouting Class on both platforms. More than one writer referred to 2016 as the "Twitter Election," and I think there's more than a grain of truth to that description.

The problem is not just that the Shouting Class creates the illusion that the country is more deeply divided than it is, but that this illusion can then become fact. Research shows that trolling is contagious; when people see other people complaining and fighting, they get the urge to join. If society is irrevocably divided, even people who aren't members of the Shouting Class feel the need to fight for their side, or risk being overwhelmed by the enemy.

But 2016 won't be the last Twitter Election. Remember, because the Shouting Class' appetite for shouting is insatiable, nothing that happens at the wider political level will make them stop shouting. So there will always be a baseline level of discord and anger visible to everyone on social media, and there will always be the danger of that discord spreading like a contagious disease.


The Shouting Class and the exhaustion of sympathy

Imagine that you are walking through a city and a hungry homeless person comes up to you and asks for money to buy food. You might give him money and walk away feeling good, imagining how now he won't go hungry. But now imagine that you walk past a row of 100 hungry homeless people, and you don't have nearly enough money to give all of them. Even if you now feed 5 hungry people instead of just 1, you'll probably walk away feeling bad, knowing that there are more hungry people out there than you can help.

That's not a perfect analogy for the way the Shouting Class overloads our empathy. Hunger and homelessness could be completely solved if we had the social will to pony up the money. But the anguish of the Shouting Class is a hole that will never be filled.

Most of the Shouting Class, with the exception of those who are just there to mansplain, blow off steam, or have a good argument, express deep dissatisfaction with society. But unlike those who join movements temporarily in order to get concrete results, the Shouting Class' disaffection is not something that can be fixed.

Idealistic Shouters will continue to find something to crusade for, but their ideals will differ, so they can't all be appeased at once. Moving in the direction of a liberal idealist's vision will not diminish his hunger for further social change, even as it drives conservative idealists to rage and despair.

Unhappy Shouters will continue to have problems with society because they are displacing negative emotion that really doesn't come from society at all. An angry, unhappy person can't be made happy by living in even the most utopian of feasible societies. What they need is not utopia but a hug, and possibly some therapy.

Sadistic Shouters don't actually want social change, but merely to give people a hard time, troll them, or "trigger" them. Sometimes they do this by pretending they have deep problems with society.

So no matter how much we change society, or in what direction, there will be a baseline level of publicly expressed social disaffection that we can't get below. With social media giving the Shouting Class a bullhorn, that baseline level is far higher than

For those of us who feel sympathy for victims of social injustice, this can be maddening. Most of us are basically good folks - when we see someone who appears to be victimized by society, we want to help them. We want to improve society, to make it better. We don't want to be part of a system that hurts even one single person.

So when nothing we accomplish can diminish the seeming anguish of the Shouters, what are we to do? We can exert ourselves ever more mightily to change society, but when we look at the past, and see that all our past successes have failed to diminish the shouting, we are liable to fall into despair and a feeling of powerlessness.

Which brings me to the final potential problem with the Shouting Class...What if it forces us to become callous?


The cost of immunity to the Shouting Class

Ultimately, all of these problems - negativity, social censure, social discord, and empathy exhaustion - can be dealt with by developing an immunity to shouting. In other words, by becoming callous toward people's expression of approbation, dissatisfaction, anguish, irritation, etc. In fact, people on social media say this all the time: "Grow a thicker skin!"

But what are the costs of having a thick-skinned society? One potential cost is that elites and social institutions become unresponsive to legitimate calls for social change.

I saw this in action in college, with the protest community. At one point I joined some protests demanding higher wages for Stanford custodial workers. We marched, we yelled outside the president's office til he came out and talked to us. They raised the workers' wages.

But after that victory, I was astonished to see the protest leaders sit down and immediately start planning their next campaign. I asked them why they didn't intend to reward the establishment for giving ground on the wage issue. They just sort of gave me disparaging looks, except for one guy who told me "Frankly, I go to protests to hook up with hot anarchist chicks."

The Shouting Class is mostly not looking to hook up. But like those protesters, social media's Shouting Class has adopted criticism, anger, and disaffection as ways of life - not the means to an end, but the end in and of itself. Eventually, just as Stanford's administrators mostly stopped paying heed to the protest community, the powers that be will learn to accept and ignore a very high baseline level of popular disaffection. And that will raise the bar for movements whose needs are more urgent and realistic and capable of being satisfied - it'll be hard to tell real grievance from grievance-as-a-lifestyle.

(I sort of think suspect something like this happened with the Tea Party and Obama. The theory is that when Democrats saw that nothing could possibly appease the Republicans, they stopped paying attention to them at all, which I think drove GOP voters to amp up the extremism to levels previously unheard of, in the form of Trump.)

I worry that the costs of evolved callousness could go way beyond political protest movements, though. When people shout "We are in pain!" after seeing chalk slogans for a political candidate they don't like, it teaches lots of people to just ignore anyone who says they're in pain. Even if you're ripping them away from their children and shoving them into detention centers. Even if you're watching them get beaten up on the street. Even if you're hacking them up with a machete or machine-gunning them into a mass grave.

Not a pretty picture, is it?

The Shouting Class, with the bullhorn of social media, is forcing us all to grow thicker skins. But I don't want to live in a society of thick-skinned people.


Social media as a prison and the Shouting Class as the prison guards

At this point, I think I've explained enough potential downsides of the Shouting Class where it's time to start talking about potential solutions. So far, I've portrayed the problem as a negative externality caused by the advent of social media technology. So why not just quit social media?

The answer is: Strong network effects. The very thing that makes Facebook so much profit, and would make Twitter so much profit if they could figure out how to put advertisements in a Twitter feed, is also the thing that makes these networks impossible to leave. You're on Facebook because your friends are on Facebook, and they're on Facebook because you're on Facebook. You're on Twitter because your colleagues, readers, customers, voters, etc. are on Twitter, and they're on it because you're on it. In other words, everyone is locked in because of the network effect.

To the extent that that network effect creates social value (and it does), it's a positive network externality. But to the extent that everyone being on social media creates net negative value due to the depredations of the Shouting Class, then that network effect is a negative externality, and it would be better not to have social media at all.

If, as studies suggest, social media has a deleterious overall effect on people's well-being, but they can't afford to stop using it, that means that social media is a prison. Rorschach once said "I'm not locked in here with you, you're locked in here with me!" On Twitter, and often on Facebook, all the non-shouters are locked in with the Shouting Class. It's a Stanford Prison Experiment, and the Shouting Class are the prison guards. Except the experiment never ends. Maybe Satre's No Exit is the better analogy.

OK, OK, that's probably way too dystopian. But if the very worst-case scenario comes true, and social media causes civil wars and a breakdown of national institutions due to the discord created by giving a bullhorn to the Shouting Class, well...maybe it wasn't too dystopian.


Why blame the technology? The Shouting Class and Revolutionary France

It's important to note a couple of things here. First, social media may have given the Shouting Class a bullhorn, but it didn't create the Shouting Class; they have always been with us. Second, the Shouting Class has caused enormous disasters before, even without social media. One fairly clear-cut case is the French Revolution.

The French Revolution started well, with an outpouring of republican sentiment, democratic reform, and positive institutional change. And it ended well, with the creation of the French Third Republic. Unfortunately, those events were 81 years apart. And during those 81 years, France was almost continuously riven by atrocities, social upheaval, large-scale war, civil war, and a political climate of paranoia, denunciation, and mutual distrust. During that upheaval, France was permanently displaced as Europe's greatest power and conquered multiple times. In fact, even the Third Republic didn't really calm France down, as the Dreyfus Affair of 1894 proved; it took the epic disaster and national unifying struggle of World War 1 to do that.

I highly recommend reading a history of the French Revolution, or listening to one on audio. Youl will learn all about the Sans-culottes, the Muscadin, the Enrages, and other groups of angry French people who battled each other over the course of those turbulent years. You will undoubtedly see some parallels between those tribes and the factions trolling each other on Twitter in America today.

And you will also learn about the explosion of newspapers and other publications that sprung up in the long revolutionary century. Many of these tried to be voices of reason, but many others were radical and incendiary. There were radical journalists who urged yet more revolution, and reactionary journalists who prompted people to assassinate the former. It was informational anarchy.

The journalists, activists, firebrands, and organizers of France's century of revolution were the Shouting Class of that day, and newspapers were their Twitter. We think of the age of newspapers as one of cozy oligopoly, with men in hats reading the Paper of Record over their breakfast. In fact, that was only the end of the newspaper age; its beginning was chaos. And France, a society primed for social division and discord, took the brunt of that chaos.


The Shouting Class and media industry concentration

That history suggests that even without all the fancy sharing and matching and instant-response technology of Twitter and Facebook, media fragmentation can give a bullhorn to the Shouting Class. It hints that there are cycles of media fragmentation and concentration, where new information technologies create an explosion of independent producers that gradually (or suddenly) gets crushed back into oligopoly.

Decentralization, of course, was the original idea. The people who built social media were primarily techno-libertarian types who believed that information wants to be free. They sought out to use the new technology of the internet to smash the established oligopolies - broadcast news, cable news, major newspapers. Evan Williams, one of Twitter's founders, created Blogger - the platform on which I'm now typing this lengthy rant - to create "push-button publishing for the people." Social media titans like Mark Zuckerberg and Jack Dorsey have all publicly committed to the ideal of "free speech" - but unlike Thomas Jefferson, they mean much more than simply having the government get its hands off the press. They saw social media as a way of democratizing information - of allowing everyone to be a part-time journalist and op-ed writer.

Of course, the self-interested part of this vision was that the slaughtering of ABC News and the New York Times would put billions of dollars into the pockets of whoever owned the social media platforms. But there was also a real idealism there. Many people were jubilant over social media's role in the Arab Spring. The little people, with their tweets and blog posts and cell phone cameras, were overthrowing vile dictators!

But fast-forward to 2017, and it doesn't really feel like the world's dictators have been overthrown, does it? All the shouting that the Shouting Class is doing on social media doesn't seem to have made America - or almost anywhere, really - a more free place.

Maybe it just takes time. As Zhou Enlai apocryphally said of the French Revolution, it's too early to say. Nobody really knows how technology affects society until well after the fact.

But I think we're now learning that the simple, decentralized information utopia envisioned by social media's creators and early boosters is not quite what we get. When Big Media's power was shattered, that power passed to a wider group of people, but not all The People. It passed to the Shouting Class.

So what do we do if we decide that rule by the Shouting Class is suboptimal? The old titans of old media aren't coming back - we will see no return to Edward R. Murrow, and the big newspapers will probably continue to drift toward their future as niche publications.

But as I see it, some kind of concentration is needed. Informational anarchy is always ruled by the Shouting Class, so the only way to curb the Shouters' power is to end the anarchy. Maybe social media platforms themselves will become the new quality filters. Maybe algorithmic blocking will use robots to shut down the Shouters. Maybe people will just stop using Twitter, and stop joining political argument groups on Facebook. Maybe everyone will make their profiles more private, and learn to unfollow people who engage in callout culture.

Or maybe everyone will just grow a really thick skin and we'll all just sort of learn to ignore each other.

Or maybe I'm wrong, and the Shouting Class really isn't that big a problem, and social media doesn't really make people that unhappy, and American or world politics are just going through an inevitable turbulent phase, and callout culture is vastly exaggerated, and I just spend way too much time on Twitter.

Or maybe the country will fall apart and our new totalitarian leaders will nationalize social media like they effectively have in China. I must say, I hope that doesn't happen.

Saturday, May 20, 2017

The NIMBY challenge


The other day I wrote a Bloomberg View post about how California is waking up to the problem of NIMBYism - development restrictions that limit economic activity and make cities less affordable. Ground Zero for this struggle is the Bay Area, San Francisco in particular. The pro-development activists known as the YIMBYs have been at the forefront of the fight. Economists have also been weighing in. Enrico Moretti & Chang-Tai Hsieh and Andrii Parkomenko have both come out with new theory papers showing negative impacts of housing development restrictions. Ed Glaeser and Joe Gyourko have a paper reaching similar conclusions after looking at data, theory, and institutional and legal details. And Richard Florida has a whole new book about the problem.

But the YIMBYs have faced a great deal of intellectual pushback from certain folks in the Bay Area. Even as I was writing my post, physicist Phil Price was writing an impassioned attack on YIMBYism over at Andrew Gelman's blog. He followed it up with a second post three days later, after getting a great deal of pushback in the comments. The commenters have made most of the points I would make in rebuttal to Price, but I think his posts are worth a close look, because they reveal a lot about the way NIMBYs think about the housing market. In order to understand and meet the YIMBY challenge, pro-housing activists should familiarize themselves with the arguments Price makes.

The first thing to note is that NIMBYs think that a house's price is defined when it's built - almost as if the price is built into the walls. Price writes:
[N]ew high-rise apartments are going in that have hundreds of apartments each, typically with a rent of $4000 – $8000 per month. If you let a developer build “market rate” apartments, that’s what they’ll build.
These numbers are a bit exaggerated, but that's not the point. What Price seems to ignore is the impact of construction on all the non-new units. Here's an example. I live in SF, in a market-rate apartment (though not one quite *that* pricey). But when my apartment was built, it didn't have the high rent it now has. It's a small, older apartment, once occupied by working-class families. The rent changed over time, turning an affordable home into a luxury home for a member of the upper middle class. In fact, when I moved into this apartment, I increased demand in this neighborhood, putting increased pressure on any working-class people who still happen to live here. What if, when I moved to SF, instead of moving into this apartment, I had moved into a nice fancy new "market-rate" unit in one of those towers that Price decries? I would not have increased demand in this neighborhood, and would not have put upward pressure on the rents of the families living nearby. 

Later, Price repeats the fixed-price idea when he writes:
Sorry, no. If the ‘market rate’ for newly developed apartments is substantially higher than the median rent of existing apartments, then building more market-rate apartments will make median rents go up, not down.
That sounds like simple math. And if the price of an apartment was somehow built into its walls and floors, it would just be simple math. In fact, though, it's wrong. Here's why. Suppose there are 2000 people in a city, living in 2000 apartments. One quarter of the people are rich, and rent apartments for $4000 apiece. Three quarters are poor, and rent apartments for $1000 apiece. The median rent is therefore $1000. Now build 400 fancy new luxury apartments that rent for $5000 each. And suppose no new people move to the city. All 500 rich people move into the fancy new $5000 places, leaving their old $4000 places vacant. The previously-$4000 apartments fall in price to $2000, and 500 poor people move into them, leaving 500 of their apartments vacant. These are used as second apartments, storage, or whatever. The rent of the 1500 apartments that used to all cost $1000 falls to $900 because of this drop in demand for low-end apartments. The median rent of the city's 2400 apartments is now $900, down from $1000 before.

So the "simple math" is not necessarily correct.

NIMBYs do seem to recognize this on some level. So they intuitively turn to a phenomenon called "induced demand" (though they may not realize it's called that). The theory is that if you build more housing in SF, you encourage people to move into SF, preventing prices from going down, or even pushing them up. Price espouses a version of this theory when he writes:
Tens of thousands of high-income people who would like to live in San Francisco are living in Oakland and Fremont and Berkeley and Orinda because of lower rents in those places. As market rate housing is built in San Francisco, those people move into it...There is a cascade: some people move from Berkeley and Oakland to San Francisco, which allows replacements to move from Richmond and El Cerrito into Berkeley and Oakland, and so on. Ultimately, rents in San Francisco go up, and rents in some outlying communities go down. Yes, the increased supply of housing lead to decreased housing prices on average but they’ve gone up, not down, in San Francisco itself.
It's perfectly possible in theory that this happens. In fact, this is even possible in the simplest, Econ 101 type supply-and-demand theory - it's just the case where supply is infinitely price-elastic.

Is this realistic, though? Price cites Manhattan as a counterexample - a very dense place where rents are still high. I'm not sure this counterexample applies - I see a lot more poor Black people living in Manhattan than in SF, for example. But anyway, a counter-counterexample is Tokyo, where construction seems to have been successful in keeping rents low.

The question is what would happen to SF. As I wrote in a Bloomberg View post last December, there's OK evidence that more housing would ease the city's affordability crisis:
In 1987, economists Lawrence Katz and Kenneth Rosen looked at San Francisco communities that put development restrictions in place. They found that housing prices were higher in these places than in communities that let developers build... 
[Recently, blogger Eric] Fischer collected more than 30 years of data on San Francisco rents. He modeled them as a function of supply -- based on the number of available housing units -- and demand, measured by total employment and average wages. His model fit the historical curve quite nicely. 
Recent experience fits right in with this prediction. In response to the housing crisis, San Francisco recently allowed a small increase in market-rate housing. Lo and behold, rents in the city dropped slightly
Admittedly, this data is not decisive. More SF construction might have pushed rents down a bit this year, but a big construction boom might suddenly induce a flood of rich people to decide to move to the city. It's not possible to know.

But if NIMBY theorists like Price really believe that induced demand determines SF rents, they should do the following thought experiment: Imagine destroying a bunch of luxury apartments in SF. Just find the most expensive apartment buildings you can and demolish them. 

What would happen to rents in SF if you did this? Would rents fall? Would rich people decide that SF hates them, and head for Seattle or the East Bay or Austin? Maybe. But maybe they would stay in SF, and go bid high prices for apartments currently occupied by the beleaguered working class. The landlords of those apartments, smelling profit, would find a way around anti-eviction laws, kick out the working-class people, and rent to the recently displaced rich. Those newly-displaced working-class people, having nowhere to live in SF, would move out of the city themselves, incurring all the costs and disruptions and stress of doing so. 

If you think that demolishing luxury apartments would have this latter result, then you should also think that building more luxury apartments would do the opposite. Price should think long and hard about what would happen if SF started demolishing luxury apartments. 

In any case, I think Price's posts have the following lessons for YIMBYs:

1. Econ 101 supply-and-demand theory is helpful in discussing these issues, but don't rely on it exclusively. Instead, use a mix of data, simple theory, thought experiments, and references to more complex theories.

2. Always remind people that the price of an apartment is not fixed, and doesn't come built into its walls and floors.

3. Remind NIMBYs to think about the effect of new housing on whole regions, states, and the country itself, instead of just on one city or one neighborhood. If NIMBYs say they only care about one city or neighborhood, ask them why.

4. Ask NIMBYs what they think would be the result of destroying rich people's current residences.

5. Acknowledge that induced demand is a real thing, and think seriously about how new housing supply within a city changes the location decisions of people not currently living in that city.

6. NIMBYs care about the character of a city, so it's good to be able to paint a positive, enticing picture of what a city would look and feel like with more development.

I believe the YIMBY viewpoint has the weight of evidence and theory on its side. But the NIMBY challenge is not one of simple ignorance. Nor is it purely driven by the selfishness of incumbent homeowners trying to feather their own nests, or by white people trying to exclude poor minorities from their communities while still appearing liberal (two allegations I often hear). NIMBYism is a flawed but serious package of ideas, deserving of serious argument.