A Funder-Driven Registered Report Initiative Compared with Standard Applications: A Study on Research Quality

Abstract

(Watch the RECORDING.)

Background and aim: Registered report (RR) is a research publishing format designed to prevent reporting bias and promote open science by conducting peer review of a research project prior to data collection. The process requires a detailed description of the method and study design, which receives peer review at a journal before data collection. If accepted, an in-principle acceptance is received, which means that the same journal commits to publishing a Stage 2 manuscript, including results, provided the authors follow the Stage 1 protocol. The Foundation Dam, a Norwegian health-research funder promoting open science, works actively to test initiatives encouraging scientific transparency and replicability. The foundation, therefore, implemented a two-year initiative during the 2024-2025 research grant calls with an incentive for applicants to publish their research as an RR. Since RR has only emerged over recent years, gaining more knowledge regarding the characteristics of research projects that choose this format is essential to improve current RR models. Moreover, previous research comparing peer reviews of RRs with standard publications (SA) has revealed that RRs are scored higher in research quality. Still, it is unclear whether this difference is observable at the grant application stage. This study aims to (1) compare the characteristics of projects opting for RR versus SA publication formats, and (2) assess whether grant applications planning to publish as an RR receive higher peer review scores for research quality than those pursuing SA.

Method: The sample consists of all applicants for research grants at Foundation Dam, a health research funder in Norway, from 2023 to 2025 (N=1170). Experts in their fields evaluate all applications. Using a hierarchical multilevel linear regression model, the study will compare Excellence (quality of research methods and innovativeness of research) and Implementation (quality of the research group, project organization, and governance) assessment scores of RR applications with SA. The applications from the 2023 research call (SA n=350) will be compared with those from the 2024 and 2025 research calls (RR n=143, SA n=677) to assess the overall effect of implementing the RR initiative on assessment scores. Intraclass correlation coefficients will be calculated to evaluate how much of the assessment score variance is due to differences in scoring between reviewers. In addition, correlation analyses will be performed to explore the potential association between evaluation scores, the confidence level of reviewers, and the agreement level between reviewers. Descriptive characteristics of research method, study design, and health research classification system categories will be presented by application group (RR or SA) and research call (2023 or 2024/2025).