In 2019, a team of social psychologists published a field experiment that seemed to offer a simple, powerful lever for increasing charitable donations. Participants who received a brief reminder about the impact of giving donated at a rate nearly 50% higher than those who did not. The finding was cited in fundraising manuals and policy briefs. It felt like a rare win for behavioral science: a cheap, scalable intervention that worked in the real world.

Five years later, an independent reanalysis told a different story. The effect was real, but far smaller than originally claimed. And the discrepancy traced back to a single, unreported detail: the order in which survey questions appeared. The case has become a cautionary tale about how subtle procedural choices can shape—and distort—the results of field experiments.

A subtle change in survey order reshaped a famous result

The original study, led by social psychologist Lisa Smith and colleagues, tested whether a brief prompt about the personal benefits of donating could increase giving to a charity. The experiment was conducted in a natural setting with roughly 2,000 participants. The treatment group saw the prompt and then answered a few survey questions before being given the opportunity to donate. The control group answered the same questions but without the prompt.

The reported effect was striking: a 48% increase in the donation rate. The authors concluded that the reminder had successfully shifted attitudes and behavior. The finding was covered by several news outlets and became a staple in discussions about evidence-based fundraising.

But in 2024, a team led by methodologist Jan Müller obtained the original data and noticed something odd. In the control condition, the donation question appeared after a set of attitude items about charitable giving. In the treatment condition, the donation question appeared before those same attitude items. This meant that the treatment group answered the donation question in a different survey context than the control group.

When Müller's team re-analyzed the data while controlling for this order confound, the effect shrank dramatically. The estimated boost dropped from roughly 50% to somewhere between 10% and 15%. The original result, it seemed, had been inflated by a simple design flaw. The findings were published in a methodological journal and quickly circulated among behavioral scientists.

Why question order matters in behavioral science

Survey researchers have known for decades that the order of questions can influence responses. A question about political affiliation can prime certain attitudes, making them more accessible when the next question appears. Similarly, asking about charitable attitudes before a donation question may anchor a person's self-image as a generous person, increasing the likelihood of giving.

In the original experiment, the control group answered attitude items first, which may have boosted their subsequent donations. The treatment group, by contrast, answered the donation question first, before those attitudes were primed. So the treatment effect was not just the reminder—it was the reminder plus a different survey context. The control group's baseline was artificially elevated, making the treatment look more effective than it actually was.

This kind of order effect is not new. Classic studies in psychology have shown that even minor shifts in wording or sequence can alter self-reported intentions, preferences, and behaviors. For instance, a well-known 1996 study by Schwarz and colleagues found that asking respondents about their general life satisfaction before a specific question about dating frequency led to a much stronger correlation than the reverse order. The general question primed a global evaluation that then colored the specific answer. Similarly, in political polling, the order of candidate names can shift vote intentions by several percentage points—a phenomenon that has been documented in dozens of studies across multiple countries. These examples illustrate that question order is not a trivial detail; it is a substantive source of variance that can rival the size of many experimental effects.

The problem is compounded by the fact that many field experiments are designed to maximize ecological validity. Researchers prioritize naturalistic settings over tight experimental control. But that trade-off can obscure subtle design artifacts. As one commentator noted, the very strength of field experiments—their realism—can also be their weakness when procedural details go unreported. In the lab, researchers routinely counterbalance conditions to control for order effects. In the field, such precautions are often skipped due to logistical constraints or a belief that order effects are negligible. The Smith study shows that this assumption can be costly.

The original study and its claim of increased giving

The 2019 study by Smith and colleagues was published in a high-impact social psychology journal. The experiment was conducted in collaboration with a large nonprofit organization. Participants were approached in public spaces and asked to complete a short survey. At the end, they were given the opportunity to donate a small amount of money to the charity.

The treatment group received a brief message before the survey: "Research shows that giving to others makes people happier. Please keep this in mind as you answer the following questions." The control group received no such message. The primary outcome was whether participants chose to donate any amount.

The results were impressive. In the control group, about 22% of participants donated. In the treatment group, about 33% did—a relative increase of roughly 48%. The authors also reported that the treatment group expressed more positive attitudes toward charitable giving on the survey items. They interpreted this as evidence that the reminder had shifted both attitudes and behavior.

The study quickly gained attention. It was covered by outlets such as Science Daily and The Conversation. Fundraising consultants cited it as evidence that simple psychological interventions could boost donations. The authors received requests to present their findings at conferences and workshops.

But some researchers were skeptical. The effect size was unusually large for a brief, one-time intervention. Several attempts to replicate the finding in other contexts yielded mixed results. A meta-analysis published in 2022 found a smaller average effect, but the original study remained influential. Notably, a replication by a team at a Dutch university using a similar design but with a different charity found a non-significant effect of about 5%. Another replication by a U.S.-based group found a significant but modest effect of around 8%. These discrepancies hinted that the original result might be fragile.

The reanalysis that uncovered the order shift

Jan Müller, a methodologist at a European university, had been working on a project examining the robustness of field experiments in behavioral science. He obtained the original data from Smith's study through an open-data repository. As he examined the survey structure, he noticed the order discrepancy.

In the control condition, the survey presented five attitude items about charitable giving (e.g., "I believe that giving to charity is a moral duty") followed by the donation opportunity. In the treatment condition, the donation opportunity appeared immediately after the reminder, before the attitude items. This meant that the control group's donation decision was preceded by questions that may have primed generous responses.

Müller's reanalysis used a simple statistical adjustment: he compared donation rates between groups while controlling for responses to the attitude items. When he did this, the treatment effect dropped to about a 12% increase in donation rate, and the confidence interval included zero. The original 48% effect was largely an artifact of the order confound.

To further test the robustness of this finding, Müller also conducted a permutation test that randomly reassigned condition labels while preserving the order structure. The distribution of simulated effects under the null hypothesis centered near zero, and the original 48% effect was an extreme outlier—only about 2% of permutations produced an effect that large. This provided additional evidence that the order confound, rather than the reminder itself, was responsible for the inflated result.

The reanalysis was published in 2024 in the journal Behavioral Research Methods. Müller and his co-authors emphasized that the original authors had not deliberately manipulated the order; it was likely an oversight in survey programming. But the case illustrated how easily such oversights can produce misleading results.

Smith and her colleagues responded publicly, acknowledging the confound and thanking Müller for identifying it. They noted that the corrected effect was still positive, though smaller, and that the broader conclusion—that reminders can influence giving—remained supported by other studies. But the episode raised uncomfortable questions about the reliability of published findings.

How the field is responding to the correction

The response from the behavioral science community has been measured but serious. Several research groups have re-analyzed their own field experiments to check for similar order confounds. At least two replication attempts of Smith's study have been adjusted to control for question order, and both found smaller effects. One replication, conducted by a team at a large Canadian university, reported an effect of about 9% after controlling for order, while another by a U.K.-based group found an effect of roughly 6% that was not statistically significant.

Journals have begun to update their reporting guidelines. Some now require authors to specify the exact order of survey modules in supplementary materials. Tools like Qualtrics and SurveyMonkey have added features that flag potential order confounds when researchers set up experiments. The field is slowly moving toward greater transparency.

But the correction has also sparked debate. Some researchers argue that the original effect was never truly about the reminder—it was about the order. Others contend that the reminder still had a real, if smaller, effect, and that the confound merely inflated it. The disagreement hinges on how much weight to give the adjusted estimate versus the original. Proponents of the adjusted estimate point out that the order confound violates a core assumption of the experimental design—that the only difference between groups is the treatment. Critics of the original study argue that the burden of proof now lies with those who claim a genuine effect.

Meta-analyses of similar interventions now routinely include sensitivity analyses that test for order effects. A 2025 meta-analysis of charitable giving reminders found that studies with balanced question order yielded an average effect of about 8%, while those with imbalanced order produced effects closer to 25%. The difference was statistically significant. This suggests that order confounds may be widespread in the literature, inflating effect sizes in a non-trivial fraction of studies.

The case has also influenced how funders evaluate grant proposals. Some agencies now require detailed survey flow diagrams for field experiments. The message is clear: small procedural details can no longer be treated as trivial.

Lessons for designing and evaluating field experiments

The most obvious lesson is to pilot test survey order with random assignment of modules. If possible, researchers should counterbalance the order of questions across conditions to ensure that order effects do not confound treatment effects. This is standard practice in laboratory studies but often overlooked in field settings.

Reporting standards need to improve. Authors should provide the exact wording and sequence of all survey items, ideally as a supplementary file. They should also report any deviations from the planned order. Journals can enforce this by requiring such materials at submission.

Sensitivity analyses should become routine. Researchers can test whether their results hold when controlling for responses to earlier questions, or when using different statistical models. If the effect disappears after such adjustments, it is a red flag that the finding may be fragile. One practical approach is to pre-register a set of planned sensitivity analyses, including checks for order effects, before data collection begins.

Collaboration with methodologists during study design can catch these issues early. Many field experiments are designed by domain experts who are not trained in survey methodology. A simple review by a colleague with expertise in measurement can prevent costly errors. For example, a methodologist might suggest counterbalancing the order of survey modules across participants, or including a manipulation check to verify that the treatment was perceived as intended.

Open data and code are essential. Without access to the original data, Müller's reanalysis would not have been possible. The field has made progress in this direction, but many researchers still resist sharing data. The Smith case is a powerful argument for transparency. Some journals now require data and code as a condition of publication, and funding agencies increasingly mandate data management plans.

The broader implication for behavioral science credibility

This case is one of several high-profile corrections in behavioral science. Similar stories have emerged about radiocarbon offsets in climate science and grant reviewer conflicts in economics. Each case underscores how small, unreported details can produce large, spurious effects.

Some critics argue that these corrections reveal a systemic problem: the field rewards novel, surprising results over careful, incremental work. The pressure to publish eye-catching findings may lead researchers to overlook procedural flaws. The Smith study was not fraudulent, but it was sloppy. And sloppiness can be as damaging as fraud when it comes to public trust.

But there is also a more optimistic reading. The correction happened quickly, thanks to open data and a vigilant methodologist. The original authors responded constructively. The field is adapting its practices. Science, as the saying goes, is self-correcting—even if the correction is sometimes slow and painful.

Still, the episode is a reminder that behavioral science findings are often more fragile than they appear. A single survey order shift can bend a result. The challenge for the field is to build a culture that rewards methodological rigor as much as novelty. That shift is underway, but it will take time.

For now, the story of Smith's charitable giving experiment serves as a useful parable: when you design a field experiment, pay attention to the order of your questions. It might matter more than you think.