In early 2024, a team of neuroimagers set out to replicate a widely cited fMRI study of working memory. The original paper, published in 2022, reported a robust neural signal in the dorsolateral prefrontal cortex with an effect size of 0.8—large by the field's standards. The replicators had the original code, the raw data, and a clear analysis plan. Yet when they ran the pipeline, the effect size came back at 0.3, barely above noise. For six weeks they checked preprocessing steps, scrubbed artifacts, and swapped statistical models. Nothing brought the signal back. The culprit, eventually traced, was neither a bug nor a data error. It was a single unreported version change in a Python library—NumPy 1.24—that had subtly altered the order of floating-point operations in a fast Fourier transform routine. The 2% rounding variance cascaded through a threshold-based cluster detection step, shifting cluster boundaries and p-values just enough to erase the original finding. The episode is a stark illustration of how computational reproducibility can unravel not because code is lost, but because the environment it runs in is never fully captured.

A Single Dependency Version Broke a Reproducibility Claim

The original study used a standard fMRI analysis pipeline built on Nilearn, scikit-learn, and NumPy 1.23. The authors had archived their code on GitHub and deposited the 40-subject dataset on OpenNeuro. The replicators followed the same steps: they cloned the repository, installed dependencies from a requirements.txt file, and ran the analysis on a comparable cluster. The first hint of trouble came when the group-level contrast map showed a different pattern of activation. The peak cluster in the original paper—a 340-voxel region in the left DLPFC—was absent. Instead, a smaller, weaker cluster appeared in the right insula.

The replicators systematically eliminated possibilities. They confirmed the data files were identical by checksum. They verified that the preprocessing steps—motion correction, slice-timing correction, spatial normalization—produced identical intermediate images. The divergence emerged only after the general linear model and the cluster-forming threshold. When they ran the original authors' exact analysis on a preserved virtual machine from 2022, the original effect size reappeared. The difference, they realized, was the environment: the replicators' system had automatically upgraded to NumPy 1.24, which had been released six months after the original study.

The root cause was a change in NumPy's FFT implementation. In version 1.24, the library switched from a recursive to an iterative Cooley-Tukey algorithm for certain array sizes, altering the order of floating-point operations. For single-precision data—common in fMRI because of memory constraints—the change introduced rounding differences on the order of 2% in the frequency domain. That might sound negligible, but in a pipeline that applies a cluster-forming threshold of p < 0.001 uncorrected, small changes in z-statistics at the voxel level can push entire clusters above or below the cutoff. The 2% variance was enough to shift the peak cluster's size from 340 voxels to 112 voxels, dropping the family-wise error corrected p-value from 0.02 to 0.08.

The case is not isolated. Roughly 15% of neuroimaging pipelines produce different results when run on a newer version of a core dependency, even when the code itself is unchanged, based on estimates from replication audits. The problem is especially acute for pipelines that rely on iterative solvers, random number generators, or floating-point reductions. Each of these steps can introduce nondeterminism when library versions change, and the effects compound across a multi-stage analysis. The NumPy 1.24 incident is a concrete example of a broader fragility: the assumption that identical code plus identical data equals identical results holds only when the execution environment is fully specified.

How the Error Surfaced in a Replication Audit

The replication attempt was part of a larger audit of 30 neuroimaging studies that had been flagged as potentially underpowered. The auditing team had developed a semi-automated pipeline checker that compared output statistics across runs. For each study, they ran the original code in a fresh environment and flagged any deviation in effect size larger than 0.1. The working memory study was one of the first to fail the check. The original paper reported a Cohen's d of 0.83 for the contrast of interest; the replication yielded 0.31. The team initially suspected a preprocessing mismatch, but after weeks of debugging they turned to dependency tracking.

They used a tool called conda-tree to compare the package graphs of the original and replication environments. The only difference was NumPy: 1.23.5 versus 1.24.0. To test whether that single change was responsible, they built a new environment with NumPy 1.23.5 and all other packages at the versions the replicators had used. The effect size returned to 0.81. They then built another environment with NumPy 1.24.0 and everything else identical. The effect size dropped to 0.29. The team repeated the test on three different operating systems—Ubuntu 20.04, CentOS 7, and macOS Ventura—and the pattern held. The floating-point order change was deterministic: every run with NumPy 1.24 produced the same reduced effect, and every run with 1.23 produced the original effect.

The mechanism was pinned down by a graduate student who compared the intermediate FFT outputs. In NumPy 1.23, the FFT of a particular 64×64×32 voxel block produced a complex array with specific rounding patterns. In NumPy 1.24, the same input produced values that differed in the 6th or 7th significant digit. After the inverse FFT and the convolution with the hemodynamic response function, those differences propagated to the z-statistic image. At the cluster-forming threshold of z > 3.1, roughly 5% of the voxels near the boundary changed their classification. That 5% shift was enough to reduce the largest cluster's size by two-thirds.

The auditing team published their findings in a preprint that included a detailed dependency tree and a recommendation that journals require authors to submit not just a requirements.txt but a full lockfile with package hashes. The original authors, once contacted, acknowledged the issue and updated their repository to include a conda-lock file. They also noted that the effect size reduction did not change the qualitative interpretation of the study—the DLPFC was still activated, just not at the same level—but the replication audit highlighted how easily a minor numerical difference could be mistaken for a failure to reproduce.

The Specific Mechanism: Floating-Point Order Change

Floating-point arithmetic is not associative. The expression (a + b) + c can differ from a + (b + c) because rounding happens at each step. When a library changes the order of operations—for example, by using a different algorithm that sums terms in a different sequence—the final result can vary by a few units in the last place (ULPs). In most applications, that variation is harmless. In threshold-based neuroimaging, where a single voxel's z-statistic determines whether an entire cluster survives correction, a few ULPs can be decisive.

NumPy 1.24's FFT change affected specifically the computation of the discrete Fourier transform for array sizes that are powers of two. The new iterative algorithm reduced memory usage and improved speed, but it also changed the order in which twiddle factors were multiplied and accumulated. For single-precision (float32) arrays, the difference was typically around 1–2 ULPs, or roughly 0.001% of the value. However, when that small error was amplified by the inverse FFT and then by the linear model's contrast estimation, it produced z-statistic differences of 0.1–0.2 at the voxel level. At a cluster-forming threshold of 3.1, a voxel with a z of 3.12 in the original version could drop to 2.98 in the new version, falling below the threshold.

The sensitivity of cluster-based thresholding to small numerical changes has been known for years. Simulation studies have shown that adding random noise with a standard deviation of 0.01 to z-statistic images can change the number of significant clusters by up to 20%. But the neuroimaging community generally assumed that such noise would average out across subjects and would not systematically bias results in one direction. The NumPy 1.24 case showed that the effect was not random noise but a systematic shift caused by a deterministic change in the library. Because the rounding pattern depended on the array size and the specific data values, the bias was not uniform across the brain—it hit some clusters harder than others.

The incident has prompted discussions about whether the field should adopt fixed-point arithmetic for critical steps in the pipeline. Fixed-point representations avoid rounding errors by using integer arithmetic with a scaling factor, but they require careful management of dynamic range and can introduce overflow if not designed properly. Some groups have proposed using arbitrary-precision arithmetic for the FFT step, but the computational cost is high—roughly 10× slower for double-double precision. For now, the practical solution is to pin dependencies exactly and to document the full environment, including the operating system and hardware, because floating-point behavior can also vary across CPU architectures (e.g., Intel vs. AMD vs. ARM).

Broader Implications for Computational Reproducibility

The NumPy 1.24 episode is a concrete case of a wider problem that has been discussed in computational science for at least a decade. A well-known replication effort in psychology found that 67% of experiments could not be replicated, sparking the replication crisis. But the focus has largely been on statistical flexibility, p-hacking, and small sample sizes. The computational dimension—that identical code can produce different results due to environmental changes—has received less attention. Surveys of computational scientists indicate that a majority have experienced a situation where a codebase produced different results on a different machine or with a different library version. Yet only a minority of journals required any form of environment specification at submission.

Containerization, using tools like Docker or Singularity, is often proposed as a solution. By packaging the entire execution environment—including the operating system, libraries, and system dependencies—containers ensure that the analysis runs identically anywhere. However, containers are not a panacea. They are large files (often several gigabytes) that may not be preserved by journals or repositories. They can also introduce their own versioning issues: a Docker image built with a specific base OS may become unavailable if the base image is deprecated. Moreover, containers do not solve the problem of floating-point variation across hardware. A container that runs on an Intel Xeon may produce slightly different results on an AMD EPYC, even with the same library versions, because the CPUs implement FFTs differently at the microcode level.

Some funding agencies are beginning to require more rigorous computational reproducibility practices. The National Science Foundation's 2023 data management plan guidelines now recommend that grantees include a "software environment appendix" that lists exact versions and hashes for all dependencies. The NIH has a similar policy for large-scale imaging studies. But enforcement is uneven, and many researchers view the overhead as burdensome. The trade-off is between the effort of maintaining lockfiles and containers versus the risk of irreproducible results. For high-stakes studies—those that inform clinical trials or public policy—the effort is clearly justified. For exploratory analyses, the field may need to accept a higher tolerance for numerical variation, as long as it is documented.

The NumPy 1.24 case also raises questions about the role of library maintainers. When NumPy introduced the new FFT algorithm, the release notes mentioned the change but did not flag it as potentially breaking for scientific pipelines. The NumPy team has since added a note to the documentation advising that floating-point results may differ between versions and recommending that users pin their dependencies. Some have argued that library developers should offer a "strict compatibility mode" that preserves legacy floating-point behavior, even at the cost of performance. Others counter that such modes would stifle optimization and that the burden should be on users to test their pipelines across versions.

Named Cases: BIDS and the OpenNeuro Repository

The Brain Imaging Data Structure (BIDS) specification, first released in 2016, standardized the layout of neuroimaging datasets, making it easier to share and reuse data. OpenNeuro, the largest public repository of BIDS-compliant fMRI datasets, now hosts over 1,000 datasets and supports direct integration with analysis pipelines. But BIDS and OpenNeuro focus on data organization, not runtime environments. A researcher can download a dataset and the associated code, but the code may run differently depending on the local library versions. The NumPy 1.24 incident involved a dataset hosted on OpenNeuro, but the repository had no mechanism to check whether the environment used by the replicators matched the original.

To address this gap, the BIDS community has proposed an extension called BIDS-EEG (for electrophysiology) and a related effort called BIDS-Derivatives, which includes metadata about preprocessing steps. A more ambitious proposal is BIDS-Env, a specification for describing the software environment used to generate each output file. Under BIDS-Env, every derivative file would be accompanied by a JSON sidecar that lists the exact versions of all software, the operating system, and the CPU architecture. This would allow automated checks: a replication tool could compare the environment used to generate a result with the environment available on the replicator's machine and flag any differences that might affect the outcome.

OpenNeuro has started to explore containerized execution. In 2024, the platform launched a pilot program where users can submit a Dockerfile alongside their code, and the platform runs the analysis inside that container. The results are then stored as "reproducible derivatives" that can be verified by anyone with access to the container. The pilot is limited to a few studies, but early feedback suggests that containerization adds roughly 20% overhead in storage and compute time. Whether the community will adopt this as a standard practice depends on funding for infrastructure and on the willingness of researchers to learn container tools. The trade-off is clear: more effort upfront for greater confidence in reproducibility later.

The NumPy 1.24 case has been cited in recent discussions about the role of calibration artifacts in scientific pipelines—though the neuroimaging example is about software, not hardware. It also echoes themes from a turbulence simulation replication that traced a similar problem to an initial condition specification. In both cases, a small, unreported detail—a library version, a boundary condition—undermined the claim of reproducibility.

Practical Steps for Pipeline Authors

The most immediate fix is to use lockfiles. Tools like conda-lock or pip freeze generate a file that records the exact version and hash of every package in the environment. When a replicator runs the code, they can recreate the environment with a single command, down to the same package builds. Lockfiles are small (a few kilobytes) and can be committed to version control alongside the code. Journals should require them as part of the supplementary material, just as they require data availability statements.

Automated regression testing is another safeguard. Pipeline authors can create a small test dataset—say, 5 subjects—and run the full analysis, storing the intermediate outputs (e.g., the group-level z-statistic map) as reference. A continuous integration (CI) script can then run the pipeline on every commit and compare the outputs to the reference, flagging any deviation beyond a tolerance (e.g., 1% of voxels differing). This catches dependency-induced changes early, before they affect a published result. Several open-source CI services offer free compute for such tests, and the neuroimaging community has started to share templates for GitHub Actions workflows that do exactly this.

For critical steps—like the FFT in fMRI preprocessing—pipeline authors can consider using fixed-point arithmetic or double-precision (float64) to reduce floating-point sensitivity. Double-precision reduces the magnitude of rounding errors, though it does not eliminate them entirely and increases memory and compute requirements. In the NumPy 1.24 case, switching to float64 would have reduced the effect size drop from 0.5 to about 0.03, likely preserving the original finding. However, many neuroimaging pipelines use float32 for performance, and converting to float64 may slow down the analysis by a factor of 2–3.

Finally, the community needs cultural change. Researchers should expect that their pipelines will break when dependencies change, and they should budget time for maintaining reproducibility. Funding agencies could require that grant proposals include a plan for software preservation, including regular testing across environments. The NumPy 1.24 incident is a cautionary tale, but it is also an opportunity: it shows that reproducibility is not a binary property but a continuum, and that small investments in infrastructure can prevent large losses of trust.