9/10/15: Is Economics Research Replicable?… err… ”Usually Not”

An interesting, albeit limited by the size of the sample, paper on replicability of research findings in Economics (link here).

The authors took 67 papers published in 13 “well-regarded economics journals” and attempted to replicate the papers’ reported findings. The researchers asked authors of the papers and journals for original data and codes used in preparing the paper (in some top Economics journals, it is a normal practice to require co-disclosure of data and empirical models estimation codes alongside publication of the paper).

“Aside from 6 papers that use confidential data, we obtain data and code replication files for 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files.”

Here is the top line conclusion from the study: “We successfully replicate the key qualitative result of 22 of 67 papers (33%) without contacting the authors. Excluding the 6 papers that use confidential data and the 2 papers that use software we do not possess, we replicate 29 of 59 papers (49%) with assistance from the authors.”

In other words, even the authors of the original papers themselves were not able to put the results to re-test.

“Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable.”

This is hardly new, as noted by the study authors. “Despite our finding that economics research is usually not replicable, our replication success rates are still notably higher than those reported by existing studies of replication in economics. McCullough, McGeary, and Harrison (2006) find a replication success rate for articles published in the JMCB of 14 of 186 papers (8%), conditioned on the replicators’ access to appropriate software, the original article’s use of non-proprietary data, and without assistance from the original article’s authors. Adding a requirement that the JMCB archive contain data and code replication files the paper increases their success rate to 14 of 62 papers (23%). Our comparable success rates are 22 of 59 papers (37%), conditioned on our having appropriate software and non-proprietary data, and 22 of 38 papers (58%) when we impose the additional requirement of having data and code files. Dewald, Thursby, and Anderson (1986) successfully replicate 7 of 54 papers (13%) from the JMCB, conditioned on the replicators having data and code files, the original article’s use of non-confidential data, help from the original article’s authors, and appropriate software. Our comparable figure is 29 of 38 papers (76%).”

A handy summary of results:

So in basic terms, economists are not only pretty darn useless in achieving forecasting accuracy (which we know and don’t really care about for the reasons too hefty to explain here), but we are pretty darn useless at achieving replicable results of our own empirical studies using the same data. Hmmm…

Economics 6/11/10: Is Modern Academia Failing?

A very interesting paper titled "Withering Academia?" by Bruno S. Frey was published by CESIFO Working Paper 3209, October 2010 (download here).

From the abstract: "Strong forces lead to a withering of academia as it exists today. The major causal forces are:

• the rankings mania,
  
• increased division of labor in research,
  
• intense publication pressure,
• academic fraud,
  
• dilution of the concept of “university,” and
  
• inadequate organizational forms for modern research.

Academia, in a broader sense understood as “the locus of seeking truth and learning through methodological inquiry,” will subsist in different forms. The conclusion is therefore pessimistic with respect to the academic system as it presently exists but not to scholarly endeavour as such. However, the transformation predicted is expected to be fundamental."

This some pretty strong stuff.

"Today, in many disciplines, the importance of a scientific idea and the position of a scholar are defined by rankings. What matters nowadays is the recognition produced by a general rankings system, normally based only on the quantity of scientific output, irrespective of quality. If quality is considered, this is done by counting the number of citations. Rankings provide simple measures of relative position in science... Dependence on rankings has been substituted for consideration of content."

"The scientific production process has increasingly been divided into neatly separated steps. ...The division of labor has led to a more efficient and rapid output of scientific results but favors partial views and discourages comprehensive considerations." Interestingly, Frey refers here to the tendency to co-author papers, not to the more worrying (from my point of view) reduction in researchers' ability to think across disciplines and deeper into broader subjects.

"The incentives to publish are not necessarily the ideal ones to gain valuable new knowledge." The need to publish as much and as well as possible may influence the choice of:

• Subjects studied
• Methods used
• Type of collaboration
• Presentation of the results
• "Extent scholars are ready to engage in “academic prostitution,” that is, to revise their paper according to the “suggestions” of the referees even if they know that they are questionable or even plainly wrong (see Frey 2003)."
  

"The stronger the publication pressure, the stronger are these deviations from how scholars
are ideally assumed to behave (Anderson et al. 2007). Overall, such practices undermine
the claim of academia to pursue true knowledge."

We've recently seen a massive scale exposure of these outcomes of research pressures in the case of environmental science publications. But Frey's arguments are much stronger than that - they are systemic in nature.

"It can be predicted that academic misconduct and fraud have increased over recent
decades. The major reason is not that scholars are less moral then they used to be. Instead, the incentives to cheat have greatly increased due to higher stress in academia." Frey offers an excellent, iconclastic outlook on the drivers and methods used in fradulent 'research' - well worth reading.

Frey also deals with the issue of grade inflation and courses overproliferation that can lead to reduced standards of teaching, research and general public good inquiry. "The high reputation of a university is a public good shared by all professors and students, but it is undermined by having too many students of lower quality."

In my view - this is an excellent paper that is worth reading for anyone concerned with the nature of learning and discovery as well as broader concept of academia in our modern society. It is strongly polemical, provocative and certainly deserves a deeper debate.

But let me add to Frey's concerns - based on personal experieince - modern academia, in pursuit of quantitive (teaching & research) targets has lost much of its real societal relevance. Vast majority of senior faculty members are withdrawn from broader social and scientific debate, residing in their own isolated towers of specialist knowledge. This problem is most acute in social sciences, where the unwritten and often unspoken rules for younger faculty are:

• Don't engage in political, social and economic policy debates outside academic research,
• Don't engage with broader community outside the walls of academia.

As a young academic, I was told on numerous occasions that writing in press is 'below academic standards', that speaking at non-academic conferences 'doesn't earn one any credit within the University walls', that 'peers don't look kindly on those who disagree with their philosophies in public', etc. The victim in all of this will be the entire academia, which is at a risk of ceasing to be “the locus of seeking truth" risks becoming a Faculty of Useless Knowledge, irrelevant to the society.