Megan McArdle’s Hack Post on Elizabeth Warren’s Scholarship

So Megan McArdle wrote a long post attacking Elizabeth Warren as a scholar. What’s surprising is how little “there-there” there is to her critique. I would love to see nomination hearings based around how expansive of a definition to use for medical bankruptcies and watching Warren rip the face off of Senators when it comes to empirical methods. I doubt it is going to come to this, but I’ll go ahead and respond. (I’ve been waiting for part two to respond, which I assume may not show up.)

Because that isn’t what this is about. It’s about giving the impression that Warren is a weak scholar. Given that Warren is considered “the leading authority in the country on bankruptcy law,” being called a hack by McArdle, of all people, is something. Especially when we get a gem of a major screwup like this right out the door in the post:

Megan McArdle, blog post: 2. The response rate on their survey was only 20%. Given the deep shame surrounding bankruptcy, you have to worry that they got an unrepresentative sample. And how is that sample most likely to be unrepresentative? Well, one pretty likely way is that people who went bankrupt through no fault of their own–folks who got whacked by large and unpayable medical bills or a business closure–were more likely to respond than the people with drug or alcohol problems, profound depression that left them unable to work, compulsive gambling issues, and so forth….

Katie Porter, comments: Also, I would like to correct the misstatement, I believe of a commentator, that Ms. McCardle reproduces in her article above, that the response rate to the survey was 20%. The response rate was right at 50%, or just under that, depending on the exact metric for response rate used. More detail on the response rate for the written survey, as well as on the bias checks that were performed for sample selection bias, is also available in the above articles.

Megan McArdle, comments: They had a 50% response rate on the questionnaires, but by the time you got to the interviews, they were down to 20%. It’s in the article.

I have no idea what to make of this. Megan opens her critique by saying that there’s a massive bias in the data sample implied by the low response rate of 20%. A commenter politely responds that the response rate is 50%. She is very polite as the 50% is on the front page of the 2009 study. Megan then says she meant the interview rate.

Nobody is perfect, especially on the blogs. I’ve messed up data on this blog before, I’ve confused terms that I knew but didn’t catch in a proofread, and I’ve used data and terms that I thought meant one thing that turned out to mean another thing. Anytime someone points this out I correct it, or pause and double-check what I thought, or quietly ditch using that information. Humility is usually the best antidote to being a hack.

But notice how Megan just keeps on going. This is one of the major planks of her argument, that the sample is corrupted, and when someone points out that what she stated was factually incorrect she just changing the terms and keeps on going as if she what she wrote wasn’t wrong. How is a reader supposed to read this? Did she mean to say interview rate in the beginning? Does she think that a 50% response rate is too low? Useless without a 50% interview rate? Did she know at the time of writing the difference between the two terms? Does she want to reconsider her argument?

(It’s pretty similar to the classic McArdle instance of “It wasn’t a statistic–it was a hypothetical” when it came to US profits of pharma.)

Which is a shame. Like the hypothetical case there’s no pause, no reflection, so as a reader I just want to assume bad faith and move along.

But I won’t. Let’s continue.

Causes, Contributes

“4. Their methodology is quite explicitly designed to capture every case where medical bills, or medical loss of income, coexist with some other causal factor–but the medical issues are then always designated as causal in their discussion…If you’re a plumber who has a stroke, you may well end up in bankruptcy simply because you lose income while you can’t work (the medical bills may or may not play a large causal role).”

Another problem Megan has with Warren’s research is that Megan believes it says medical debt is the cause of bankruptcies instead of something that contributes to bankruptcies. Instead of simply being a contributor among many multi-causal problems Megan states that Warren believes that medical debts are the sole cause (“always designated as causal”) instead of a contributor among a multi-causal set of items.

Is that true? Let’s look at the title of the paper that kicks off this line of research: “Illness And Injury As Contributors To Bankruptcy.” (my bold, italics, and underline.) It’s in the #@$%@# title that it’s a contributor and not the sole cause!

From the abstract of the 2007 paper Megan hates: “Our 2001 study in 5 states found that medical problems contributed to at least 46.2% of all bankruptcies…CONCLUSIONS: Illness and medical bills contribute to a large and increasing share of US bankruptcies.” (my bold, italics, and underline.)

This may look like a little nitpick but it is important: bankruptcies are multi-causal, and as far as I can tell Warren’s research has always emphasized this. Certainly the titles and conclusions of her paper place emphasis on this. Megan is trying to imply a con job, that Warren is an ideologue who manipulates her results and her conclusions to be stronger than they deserve. That’s not true.

Data Data Everywhere

There’s a lot of this: “The authors have an odd tendency to ignore what the respondents themselves say. 32% of those surveyed about their 2007 bankruptcies–not 62%–reported that ‘medical problem of self or spouse was reason for bankruptcy.'”

Notice what is going on here. Warren and her co-authors realize that there are a lot of ways to interpret the data and, ethically, put the data out there so others can disagree and make counter-arguments. All the data results are there. Megan does make these counter-arguments but gives off the impression that something is being hidden, or a sneaky move is being made.

Which gets to the bigger complaint Megan has about the paper: “As I discussed at the time, early 2007 is a terrible, horrible, no good, very bad time to do any sort of study on bankruptcy… Bringing me to my next point: the paper thoroughly obscures the point that by their own calculations, the number of medical bankruptcies fell quite dramatically between 2001 and 2007.”

I still don’t get this complaint. There was an absolute overhaul in the way bankruptcy is carried out in 2005. Comparing the absolute numbers before and after wouldn’t be an apple-to-apples comparison. You can argue that no valid research could possibly done and that any empirical statistics should never be carried out on post-2005 data, which is what I think Megan argues, or that you acknowledge the data limitations, do your best to compensate and provide additional information, which Warren and the rest do under the section “Changes in the Law”, and carry on. I’m in the second camp.

I’m in Delaware.

“A pretty convincing paper argues that the single best predictor of bankruptcy is simply how much debt you’ve accumulated–not income, job loss, divorce, or what have you.”

This paper Megan finds convincing is a study of the population of the state of Delaware. Delaware. Nevermind that it puts together Delaware bankruptcy records with data from the Survey of Consumer Finance in a way designed to exacerbate data differences and corrupt instruments, which is a major problem. Are we comfortable thinking that Delaware can substitute in for the nation, especially when we have a representative nationwide sample with a 50% response rate like the one Warren uses?

And as Thomas Levenson points out:

But as McArdle completely failed to acknowledge, Zhu does so while using somewhat more stringent standard for counting medical expenses as a factor in bankruptcy than other scholars employed — as he explicitly acknowledges. He concedes the continuing significance of medically -induced bankruptcy. He acknowledges what he believes to be a weak underweighting of that factor (because some people pay for medical expenses on credit cards). And he notes that a number of other studies, not limited to those co-authored by Warren, come to different conclusions.

In other words: McArdle correctly describes one conclusion of this paper in a way that yields for its readers a false conclusion about what the paper itself actually says.

Megan herself says that she thinks health care costs contribute to a real number of bankruptcies. The question is how to best go about defining what constitutes a health care cost related bankruptcy. If you read McArdle’s post you would think that they use similar data and use similar methods but find different results. Instead, the study she likes uses Delaware and a consciously described more exclusive definition.

How you rate bankruptcy rates from medical conditions is going to be a function of how strictly you define a medical bankruptcy. Zhu has a higher-filter than Warren’s, so these results aren’t surprising. I could see using a stronger filter than Warren’s. The question is whether or not Warren’s filter is a good one. If you read Warren’s 2009 paper you see that her results are robust to alternative specifications: “Adopting an even more stringent threshold for medical debts (eg, eliminating those with medical debts below 10% of family income) would reduce our estimate by <1%.”

That Zhu paper is part of a dialogue that Warren started about the role of medical debt in bankruptcy. It’s weird for Megan to call Warren a hack but then applaud Zhu’s paper: it’s all part of the same continuum of research. If she likes Zhu’s paper she should thank Warren for starting the discussion.

Data Undercounting

Before moving on I want to point out that the coolest new research is that the 2007 data Warren uses is probably undercounting medical debt significantly. From Jacoby and Holman’s Managing Medical Bills on the Brink of Bankruptcy:

This paper presents original empirical evidence on financial interactions between medical providers and their patients who go bankrupt. We use a nationally representative sample of people who filed for bankruptcy in 2007 to compare two popular but hotly contested methods of measuring medical burden. By applying both methods to the same filers, we find that nearly four out of five respondents had some financial obligation for medical care not covered by insurance in the two years prior to filing as measured by the survey method. The court record method paints a different picture, with only half of the cases containing identifiable medical debt, and of substantially more modest amounts. We test several theories to explain the discrepancy and find we can account for it to a significant extent by filers’ methods of managing medical bills that make it difficult or impossible for the court record method to detect them. For example, we find the highest rates of mortgage and credit card use for medical bills among respondents with the largest discrepancies between the survey method and the court record method. Respondents who report medical bills as a reason they filed for bankruptcy mortgaged their homes for medical bills at nearly four times the frequency of other filers, and were about a third more likely to use credit cards for medical bills. We also find disparities by age, sex, race, and housing tenure that skew the court record measure. Our findings suggest that the advice offered by experts in “medical practice management” reduces providers’ financial exposure from patient liabilities. One implication of this “success” is that the court record method of measuring medical bills should not be used on a standalone basis to measure the impact of medical bills on financially distressed families. Also, the court record method should not be used to refute survey estimates of medical burden in debates over health care or bankruptcy reform.

It look like the 2007 data that Warren uses for the 2009 paper undercounts medical debt. The creditors from bankruptcy aren’t always clear on what is and is not medical debt so these authors hit the data to re-examine it. So the results are probably stronger. Cool.

Two-Income Trap

If you made it this far, I feel terrible for you. I feel like Virgil leading you through a Glibertarian Inferno.

As far as I can tell, Megan thinks the book Two-Income Trap is a good book. She has three critiques:

1) “The first is that Warren simply fails to grapple with what her thesis suggests about the net benefits of the two-earner family.” I’m not sure I understand what this means. I think, and there’s certainly nothing in the book to discount this, that Warren believes women having careers is a major net benefit both to women themselves and society as a whole. (Warren is a pretty accomplished person, after all.)

What she points out is that as a rising costs for certain goods (health care, education, and housing in particular) there isn’t a safety net if one income is lost via illness, divorce, or death via the family’s work structure. As someone familiar with risk management it makes perfect sense.

2) “Warren kind of waves her hands and mumbles about social programs and more supportive work environments. There is no possible solution outside of a more left-wing government.”

I completely grant that this books identifies a major problem for the economic security of the middle class but doesn’t completely solve it. True. Is that a serious knock? It is also true to me that the obvious solution is something like single-payer health care and free college. But one could also imagine a solution of a Hong Kong style health care, or more competition/consolidation of health care across state lines. I don’t think those are good solutions, but maybe you do. If the right wants to become relevant again, addressing these insecurities is incredibly important.

Side note: Is Warren a crazy left-winger? As Thomas Levenson brings up in a must-read beatdown of that post, Warren’s essential final point in the book is that ”…families need to safeguard themselves”. If you read this story from her money-planning guide “All Your Worth” she advises a family to be clearer in separating discretionary from non-discretionary spending and budgeting accordingly. Is this the sound of the proletariat seizing the means of production? Comrades: rise up and plan your monthly budget in a smarter way!

3) “Warren argues that housing consumption hasn’t actually increased much in the last few decades: by less than a room per house.” Housing price increases have vastly outstripped housing price consumption; housing prices have tripled since 1987. That easily surpasses a 20% increase in housing consumption, and that’s all that’s necessary for Warren’s point to carry. Ed Glaeser’s work actually supports Warren’s point. If housing supply is inelastic, then of course as populations grow the price of housing will be bid up. But again, if this is the biggest attack on the book I think it’s safe to declare it a victory.

I actually came across Warren from this 2005 issue of The Boston Review, where Warren argued against the myth of overspending contra people like Juliet Schor and others who thought Americans were on a luxury fever trip. Warren pointed out that people actually spend less on clothing that they used to as a result of globalization and technology, but that the real squeeze on middle-class families came from increases in health care, education and housing.

This made me prepared for the wave of “consumption inequality isn’t that bad” arguments that washed over the internet last year. Because if you just look at discretionary spending, they are right! People get more cheaper. But if you look at the whole picture of what constitutes middle-class security, it is actually getting much more expensive. Now the big paper quoted by the consumption inequality camp only looked at discretion income. Christian Broda and John Romalis’s “The Welfare Implications of Rising Price Dispersion” is mostly about food. When pieced together, the idea that people are eating cheaper food to keep discretionary spending tight, are eating cat food to pay for health care, is a much different picture than simply looking at how cheap you can get a nice stereo.