Do economics journals foster replicable research?

Abstract

Watch the VIDEO here.

Replications are pivotal for the credibility of empirical economics. It is widely recognized in economics that replication studies are a necessary condition for scientific integrity. Alarmingly, several studies indicate that a significant share of empirical economics research cannot be replicated. At the same time, the awareness among researchers, that empirically-based research is often based on shaky grounds, has increased in the last years. It becomes more and more evident that there is a need for more replications in economics to regain trust and credibility in empirical economics research.

Though established scholarly journals have adopted replication policies in recent years, replication activities only slightly increased. Against this background our talk investigates if and how journals in economics foster replicable research. For this purpose, we will address two aspects:

1. Journals’ data policies and their effective enforcement in economics: The first part of our talk presents the findings of a new study, in which we evaluated almost 600 articles published in 37 well-regarded journals with a data availability policy. First, we highlight the share of articles that fall under the data policy. Subsequently, the talk contrasts for how many of these data-based articles replication files were available in journals’ data archives and/or the supplemental information section of the article. Moreover, the exact requirements of journals’ data policies have been contrasted to the replication files available on journals’ web pages (respectively in their data archives).We developed a ‘compliance rate’ for each journal in our study. The higher the compliance rate the more do journals enforce their data policy. In the first part of our talk, we also discuss the question whether voluntary data policies are effective in fostering replicable research. For this purpose, we compare the compliance rate of journals with a voluntary data policy to their mandatory counterparts.
2. Journals as publication outlets for replication studies: Though researchers agree that replication studies are needed to regain trust and credibility in empirical economic research, replication activities only slightly increased in recent years.  One reason for that finding can be that in the current system replicating other people’s result does not progresses researcher’s career. Another reason is the paucity of publication outlets for such replication studies. In this part of our talk we also discuss whether established journals should implement replication sections or whether a journal which is entirely dedicated to replication, would be a better way to foster the publication of replication studies. As a showcase we will briefly introduce the newly founded “International Journal for Re-Views in Empirical Economics” (IREE).

To conclude, we sketch the current and potential future developments in economics when it comes to reproducible research.