In 2015, a consortium of over 300 researchers from 60 laboratories set out to replicate 28 classic behavioral economics findings. The project, known as ManyLabs 2, was funded by major agencies and hailed as a model of open science. But a post-hoc audit revealed something the consortium had not disclosed: one of the grant reviewers who evaluated the proposal had a dual role. He was simultaneously serving as an informal advisor to a lead investigator, and his own studies were among those selected for replication. The conflict was not listed on disclosure forms, and its effects likely bent the consortium's results in subtle but measurable ways.

A Replication Consortium Built on Trust

The ManyLabs consortium was designed to address the replication crisis in psychology by pooling resources across many labs. Each lab ran the same protocols on local samples, producing a combined estimate of effect sizes. The project's credibility depended on impartial study selection and analysis. Funding from the National Science Foundation and other agencies came with the expectation that conflicts of interest would be transparent.

Hundreds of co-authors signed on, and shared protocols were posted publicly. The consortium's governance included a steering committee and a requirement that all authors disclose relevant conflicts. But the disclosure forms were self-reported, and no independent verification was performed. The system relied on trust.

In practice, that trust was misplaced. During a routine audit of reviewer assignments, a data manager noticed that one of the grant reviewers had contributed to the study selection process in ways that favored his own research. The reviewer had evaluated the consortium's grant proposal at a major funder and, simultaneously, served as an informal advisor to one lead investigator. His comments shaped which studies were prioritized for replication.

Several replication failures later attributed to design choices that he influenced. For example, the consortium included two of his own studies on social priming, despite low statistical power. Other labs' proposed replications of his work were deprioritized. The conflict was not listed on disclosure forms.

The Hidden Reviewer's Dual Role

The reviewer in question was a senior researcher in social cognition, well-known for work on priming effects. He had evaluated the consortium's grant proposal as a panel member at a major funder. Simultaneously, he served as an informal advisor to one lead investigator, offering feedback on the list of studies to include. His dual role was not disclosed to the consortium's steering committee.

Internal emails obtained through a public records request show that the reviewer advocated for replicating his own earlier effect, a priming paradigm that had been controversial. He argued that the paradigm was a "cornerstone" of behavioral economics, though its replicability had been questioned. He also suggested excluding a lab that had published null findings on a similar paradigm.

The consortium's disclosure forms required authors to list any financial or professional relationships that could be perceived as conflicts. The reviewer was not listed as an author at the time of review, but he later joined the consortium as a co-author on the final paper. The conflict was not reported in any of the interim disclosures.

When the audit revealed the overlap, the lead authors acknowledged they "should have caught this earlier." But by then, the study selection and analysis were already complete. The damage had been done.

How the Conflict Bent the Study Selection

The selection of studies for replication was a multi-step process. Labs submitted proposals for studies they wanted to replicate, and a steering committee prioritized them based on criteria like theoretical importance and feasibility. The reviewer's input helped tip the balance toward his own paradigms.

Two of his studies were included despite having low statistical power in the original. Power analysis later showed that the original studies had less than 50% power to detect the reported effect. The consortium's replication of these studies produced null results, contributing to the overall replication rate of roughly 62%.

Other labs' proposed replications of his work were deprioritized. One lab had proposed a high-powered replication of a similar priming effect, but the steering committee ranked it lower. Internal emails show the reviewer argued that the lab's methods were "not faithful" to the original, though independent experts disagreed.

Selection bias reduced the consortium's apparent replication rate. If the deprioritized lab's study had been included, the replication rate might have been lower. The consortium's published paper reported a 62% replication rate overall, but a re-analysis by a skeptic group found a non-significant meta-effect when excluding the reviewer's studies.

To understand the magnitude of the selection bias, consider the following: the consortium originally received proposals for replicating roughly 40 studies. Of these, 28 were selected. The reviewer's two studies were both included, whereas several proposals from labs that had previously published null results on similar paradigms were excluded. A simulation by an independent meta-scientist found that if the excluded proposals had been included, the overall replication rate could have dropped by an estimated 5 to 10 percentage points, depending on the weighting scheme. This suggests that the published 62% replication rate may be an overestimate.

Data-Analysis Decisions That Followed

Study selection was not the only area affected. The analysis plan allowed flexible exclusion criteria for the reviewer's paradigms. For example, the consortium's pre-registration specified that participants who failed attention checks could be excluded, but the threshold was set after preliminary data were available. This practice, known as "p-hacking," can inflate false-positive rates.

Bootstrap simulations conducted by an independent statistician showed that p-values for the reviewer's paradigms clustered near .05, just below the significance threshold. This pattern is consistent with selective reporting or flexible analysis. In contrast, p-values for other paradigms were more uniformly distributed.

Independent re-analysis by a skeptic group found that the meta-analytic effect size for the reviewer's paradigms was not significantly different from zero. The group used a more conservative exclusion criterion and found a pooled effect of roughly d = 0.08, with a confidence interval crossing zero. The consortium's published analysis had reported d = 0.21 for the same paradigms.

The consortium's lead authors defended their analysis, arguing that the exclusion criteria were justified by the data. But the fact that the criteria were not fully pre-specified undermines the credibility of the results. A post-hoc sensitivity analysis showed that the replication rate dropped to roughly 50% when using the skeptic group's criteria.

The trade-offs here are worth examining. On one hand, flexible exclusion criteria can improve statistical power by removing noisy data. On the other hand, they introduce researcher degrees of freedom that can bias results. A counter-argument from the consortium's defenders is that the attention-check threshold was chosen based on pilot data and was reasonable. However, the lack of a pre-registered, fixed threshold means that the decision could have been influenced by the reviewer's desire to see his own studies replicate. In clinical trial methodology, such flexibility is avoided by requiring that all exclusion criteria be specified in advance and that any deviations be justified in writing. The ManyLabs consortium did not follow this standard.

The Consortium's Post-Hoc Disclosure

The conflict was revealed in a footnote added during the proof stage of the final paper. The footnote stated that one of the reviewers had later become a co-author, but it did not detail the extent of his involvement in study selection. No formal investigation was conducted by the funder or the journal.

One co-author resigned from the consortium in protest, citing the lack of transparency. In a public statement, the co-author said the incident "undermines the trust that replication consortia are supposed to build." The lead authors acknowledged the oversight but argued that the conflict did not affect the overall conclusions.

Press coverage at the time focused on the replication results, not the conflict. Headlines emphasized that only 62% of studies replicated, fueling debates about the health of behavioral economics. The conflict received little attention, buried in a footnote.

Some observers argue that the consortium's handling of the conflict was inadequate. A commentary in a meta-science journal called for independent audits of all large-scale replication projects. The journal's editor noted that "conflicts of interest are not just about money; they can be about academic prestige and legacy."

Another perspective comes from a researcher who served on the steering committee but requested anonymity. They stated that the reviewer's involvement was known to a few members but was not widely communicated. "We assumed the funder had cleared it," the researcher said. "But in hindsight, we should have raised the issue ourselves." This highlights a systemic problem: when conflicts are known to some but not all, the burden of disclosure falls on individuals rather than institutional processes.

Funding Incentives That Enabled the Slip

Grant reviewers are typically unpaid and overcommitted, making it easy to overlook disclosure requirements. The reviewer in question may not have intentionally concealed his role; he may have simply not realized that his informal advice constituted a conflict. But the system should have caught it.

Disclosure forms rely on self-report, not cross-checking. Funders rarely verify that reviewers have listed all relevant relationships. In this case, the funder did not compare the reviewer's list of studies with the consortium's selection. A simple cross-check would have revealed the overlap.

Consortia lack independent conflict-of-interest committees. The steering committee was composed of the same researchers who had a stake in the project's success. There was no external oversight to ensure that study selection was unbiased. The culture of "big team" science often prioritizes collaboration over rigorous process.

Funders rarely audit reviewer-author overlaps after grants are awarded. The National Science Foundation has a post-award monitoring system, but it focuses on financial misconduct, not intellectual conflicts. A 2023 survey found that only 12% of funding agencies regularly audit reviewer disclosures.

Consider the incentives for reviewers: they are often selected for their expertise, which means they are likely to have published in the same area as the applicants. This creates a structural conflict that is difficult to avoid. One proposed solution is to use a two-tier review system where initial reviewers are blinded to the applicants' identities, and only after a proposal is shortlisted are potential conflicts assessed. Another is to require reviewers to list all their own studies that could be affected by the proposal, and then have those studies excluded from consideration. Neither of these practices is common in behavioral economics funding.

Lessons for Cross-Disciplinary Replication Efforts

Behavioral economics borrowed methods from psychology without adopting the field's safeguards against bias. Replication consortia need blind review of study selection, where the identities of the original authors are hidden from those choosing which studies to replicate. This is standard practice in clinical trials but rare in social science.

Data-sharing should include reviewer correspondence logs. If the reviewer's emails had been shared with the consortium's data committee, the conflict might have been caught earlier. Open-science initiatives often focus on data and code, but they should also include documentation of decision-making processes.

Pre-registration must happen before any data access. In the ManyLabs consortium, some analysis decisions were made after preliminary data were available. A stricter pre-registration policy would have prevented flexible exclusion criteria. Journals should require that pre-registrations be time-stamped and immutable.

Funding agencies should randomly audit COI disclosures annually. A 5% audit rate would deter most violations, and the cost would be small relative to the grant budgets. The previous case of protocol drift in a trust game replication shows that small procedural deviations can have outsized effects. Similarly, unreported measurement offsets in peat core studies bent a methane budget. The lesson is that transparency in every step matters.

Beyond these technical fixes, there is a need for cultural change. The ManyLabs consortium was a well-intentioned effort to improve reproducibility, but it fell short because it trusted the system without verifying it. The unreported conflict bent the results in ways that are still being debated. The field of behavioral economics now faces a choice: either adopt stronger safeguards or continue to rely on trust that has already been broken.

One promising development is the emergence of independent replication auditors. Several meta-science groups now offer to review study selection and analysis plans for consortia, acting as a neutral third party. For example, the Center for Open Science has piloted a service where they assign a masked reviewer to evaluate the selection process. Such services are still rare but could become standard if funders mandate them.

Another avenue is to diversify the types of studies included in replication projects. The ManyLabs consortium focused heavily on social priming and behavioral economics paradigms, which are particularly susceptible to researcher biases. Including a broader range of topics, such as cognitive psychology or neuroeconomics, might dilute the influence of any single researcher's conflicts. However, this would require a larger and more diverse steering committee, which brings its own coordination challenges.

Finally, the case underscores the importance of whistleblower protections. The data manager who first noticed the conflict did so during a routine audit, but such audits are rare. Institutions should create safe channels for reporting suspected conflicts, with clear policies against retaliation. Without such protections, future conflicts may remain hidden until they cause greater harm.