Same input, different answer

In December 2024, one commit changed MapMyCells' default bootstrap_factor from 0.9 to 0.5. It shipped in cell_type_mapper v1.4.0, whose release notes list four new features and do not mention the change (commit).

In March 2025 a user resubmitted the same file to the hosted service and got a very different distribution of confidence scores. They posted on the Allen forum under the title "Has MMC changed? Completely different results on identical datasets" (thread).

The maintainer replied within minutes:

• The old default "is so large that even cells with a poor quality mapping received bootstrapping_probability ~ 1.0".
• Under the new one, a score of 0.6 means about 60% correct.
• The share of cells assigned to the correct type had "almost no" change.

So the labels held, but anything filtered by confidence did not. The settings had been recorded all along, under a config key in the output file. Nobody had read them.

Asks of the design: pin algorithm parameters, and show them next to the answer instead of filing them in metadata (R-2, R-3).

In October 2017 a commit to AllenSDK changed CCF_VERSION_DEFAULT from ccf_2016 to ccf_2017. The changelog for that release, 0.14.3, does not mention it (commit). Any code that relied on the default got different labels after upgrading.

An analysis run for this project on Allen's published 100 µm annotation volumes finds that 27.8% of voxels carry a different label in 2017 than in 2016. About half of those are refinements into finer structures. But 11.4% of all voxels moved to an unrelated structure: the subiculum became CA1, and the corpus callosum became optic radiation. Between 2015 and 2016, 10.3% changed. Almost all of that was redrawn motor-cortex borders.

A further 10 µm release from 2022 exists on Allen's download server. It is not on the documentation page, and AllenSDK has no key for it. In that release, 105 of 1,327 structures differ and one nucleus goes from 9,484 voxels to none (forum).

Asks of the design: never inherit an SDK default. Pin the annotation and resolution, and say both in every answer (R-4).

Excel's default settings turn SEPT2 into "2-Sep" and MARCH1 into "1-Mar". A 2016 survey found the resulting errors in 19.6% of papers with Excel gene lists in the journals it covered (Ziemann et al.). A 2021 follow-up of papers from 2014 to 2020 found them in 30.9%, with no improvement over the period (Abeysooriya et al.). In 2020 the gene nomenclature committee renamed every symbol that converted to a date: SEPT1 became SEPTIN1, and MARCH1 became MARCHF1 (Bruford et al.).

The fix for one kind of drift created another. Checked against Ensembl release 116 for this project, the Allen Brain Cell Atlas gene table (release 20231215) still uses Sept4 and March1. In total:

• 2,400 of its 32,285 symbols differ from the current names.
• 609 of its ids no longer exist in Ensembl.
• 15 old symbols now belong to a different gene in the same table. Ugt3a1 and Ugt3a2 have swapped.

A join on symbols misses genes and, worse, can quietly join the wrong one.

Asks of the design: key genes on stable id plus annotation release, and resolve old symbols with the rename shown (R-5).

Glatard and colleagues ran the same binaries on the same data under two Linux distributions. Only the system libraries differed. For FSL FIRST, "only 12.7% of the classifications are identical", and agreement fell to a Dice coefficient of 0.59 for small structures such as the amygdala. Resting state ICA differed in a third of subjects. They traced the first divergence to a one-bit difference in the C library's expf(), amplified by a long pipeline (Glatard et al. 2015).

An earlier study found that FreeSurfer volumes changed by 8.8% on average, and up to 64%, between software versions. Repeated runs on one machine were identical (Gronenschild et al. 2012). Being deterministic on one machine proves nothing across machines.

Asks of the design: record the environment a computed answer ran in. R-13 was added to the requirements because of this case.

Between Seurat 4.4.0 and 5.0.0, the defaults of FindMarkers changed: logfc.threshold went from 0.25 to 0.1 and min.pct from 0.1 to 0.01. The 5.0.0 release notes mention neither (v4.4.0, v5.0.0).

A systematic comparison found "sizeable differences in sets of significant marker genes" between the two versions, driven by how log fold change is computed. It found the difference between Seurat and Scanpy comparable to the variability from sequencing fewer than 5% of the reads (Rich et al., Cell Systems 2026).

10x's Cell Ranger 7 started counting intronic reads by default. Its own aggregation step refuses to combine samples run with different settings. That is the right pattern.

Asks of the design: record the effective value of every threshold, and refuse to combine results whose settings disagree (R-2, R-1).

Wadi and colleagues tested the same gene lists against 2010-era and 2016 pathway annotations. With the old annotations, "74% of enriched 2016 terms were missed on average". Mostly that was because existing pathways had been updated. In 2015, 67% of about 3,900 surveyed papers cited tools with outdated annotations (Wadi et al. 2016).

Asks of the design: treat the reference release as an input, exactly like the data, and carry it with every answer (R-1, R-5).