Measured against ground truth.
We tested the production engine in 280 blind runs against known injected benchmark factors. See whether FactorPrism® found each factor, located it at the right level, and where performance begins to degrade.
Most tools cannot even see the place where the cause lives
Suppose margin fell because of something specific to Northeast Outerwear — one region's slice of one product line. In the tooling most teams actually use, that location is not a row on any screen. Slice by region and the problem smears across everything Northeast sells; slice by product and it smears across every region's outerwear. The cross-hierarchy intersection where the cause actually acts — the thing you need to see — is structurally invisible to one-dimension-at-a-time analysis. Tree-based screening functions touch such intersections, but emit them as overlapping fragments with no location in your hierarchy and no reconciliation to your number.
So the first question about any root cause tool is whether the answer is visible at all. And the second question follows immediately: when a tool does name a cause, how would you know if it's right? On real business data there is no answer key. The industry's track record here is poor, and documented: anomaly-detection platforms became famous for alert fatigue — hundreds of flagged segments, most of them noise — and automated insight tools surface "interesting" segments with no statement of accuracy at all.
This study answers both questions the only honest way: generate business data where the true causes are injected by construction — at known locations across the hierarchy lattice, including exactly the cross-hierarchy intersections ordinary tooling can't render — then run the production FactorPrism® engine against it blind, and score whether it finds each cause and the place where it acts. We also do something vendors rarely do: we report where performance degrades, and we publish the comparison against the naive method most tools (and most analysts) actually use.
Benchmark design140 datasets, known causes, a blind engine
Structure
Two hierarchies (Product: category → subcategory; Geography: region → state), four children per level — 256 leaf segments, each starting at ~10,000 units. Two periods: before vs after, the canonical "why did the number move?" question.
Injected causes (the ground truth)
3 causes per dataset, each a ~5% effect applied at a randomly chosen node of the hierarchy — sometimes a whole category, sometimes a single state. The benchmark scores whether the engine finds both the effect and the location where it acts, with the correct direction.
Noise sweep
Every leaf also moves randomly between periods, swept from 0.1% (stable enterprise aggregates) to 15% (extremely volatile segments). 20 independently generated datasets per noise level, 7 levels — 140 datasets, each run through two configurations (280 runs, zero failures).
The comparison arm
The same engine with its significance test switched off, keeping only a magnitude threshold — the standard "report every segment that moved more than X% of the change" approach used by dashboard tooling and manual analysis alike.
The numbers
A factor counts as a hit only if it names the injected location exactly — not a broader or narrower segment that contains it — and the correct direction. Recall = injected benchmark factors recovered; recall@top-5 = ranked where a reader will actually look; precision = returned factors corresponding to a real benchmark factor; magnitude error = how far the estimated impact is from the injected factor. Chance scores the same output against randomly chosen locations: it measures how much "recall" a factor list earns by coincidence alone.
| Period-over-period noise | Config | Factors returned (mean / max) | Recall | Recall@top-5 | Precision | Chance | Median magnitude error |
|---|---|---|---|---|---|---|---|
| 0.1% | FactorPrism® | 4.6 / 12 | 100% | 100% | 83% | 2% | 4.0% |
| 0.1% | magnitude-only | 6.5 / 44 | 100% | 100% | 82% | 2% | 4.2% |
| 0.5% | FactorPrism® | 4.0 / 10 | 100% | 100% | 84% | 2% | 2.8% |
| 0.5% | magnitude-only | 14.6 / 118 | 100% | 100% | 64% | 2% | 5.7% |
| 1% | FactorPrism® | 3.8 / 5 | 100% | 100% | 83% | 2% | 3.2% |
| 1% | magnitude-only | 28.6 / 167 | 100% | 100% | 45% | 2% | 8.3% |
| 2% | FactorPrism® | 3.8 / 6 | 98% | 98% | 82% | 2% | 6.0% |
| 2% | magnitude-only | 56.8 / 192 | 98% | 98% | 24% | 2% | 14.9% |
| 5% | FactorPrism® | 2.7 / 6 | 65% | 65% | 76% | 0% | 13.0% |
| 5% | magnitude-only | 109.3 / 229 | 83% | 73% | 5% | 3% | 34.8% |
| 10% | FactorPrism® | 1.1 / 3 | 22% | 22% | 74% | 0% | 20.7% |
| 10% | magnitude-only | 146.6 / 243 | 62% | 38% | 2% | 3% | 53.0% |
| 15% | FactorPrism® | 0.6 / 2 | 10% | 10% | 78% | 0% | 33.2% |
| 15% | magnitude-only | 162.8 / 250 | 40% | 25% | 1% | 5% | 49.2% |
20 datasets per noise level per configuration; FactorPrism® 5.15 (October 2026) at shipping defaults, no benchmark-specific tuning. Full method notes below.
What the numbers sayFour findings
1. In its operating regime, the engine is essentially exact.
At the noise levels typical of established business aggregates (up to 2% random period-over-period movement per segment), FactorPrism® recovered 98–100% of injected benchmark factors at their exact location, ranked every recovered one in the top 5, kept precision at 82–84%, and estimated each factor's impact within 3–6% of its true contribution — while returning a list of about four factors.
2. The factor list never floods.
This is the category's documented failure mode, and the headline result. By 2% noise, magnitude-only thresholding returns 57 factors on average (peaking near 200) — an alert flood in which the three real causes are items among dozens, at 24% precision. FactorPrism® returns about four, with the same recall (98% vs 98%). Its significance test isn't discarding truth to look tidy; it's discarding segments whose movement carries no evidence of signal beyond their own volatility.
3. When data gets very noisy, FactorPrism® goes quiet instead of guessing.
At 10–15% noise, a 5% cause sits at or below what two periods of data can distinguish. FactorPrism® then returns about one factor — and when it names one, it is right about three times in four. The magnitude-only arm "finds" 40–62% of causes only by returning ~150–160 factors at 1–2% precision: the truth is in the list the way a name is in a phone book. No tool should claim to work in this regime; what a tool should do there is say little, and be right when it speaks.
4. More periods move the line.
Because a factor is identified across the whole window, every extra period adds evidence. Run on the same kind of data over six periods instead of two, FactorPrism® recovers 77% of injected factors at their exact location at 5% noise and 55% at 10%, with precision around 90%.
What this means for your dataThe variable that matters is stability, not size
Performance is governed by the period-over-period stability of your segments — not company size or row count. Quarterly revenue for an established business, aggregated to segment level, typically behaves like the left side of the table; daily data for tiny, sparse segments behaves like the right. FactorPrism®'s pre-flight checks and significance gating are built around exactly this: report what the data can support, and stay quiet past that line.
Two structural properties hold at every noise level, because they're guaranteed by construction rather than estimated: the returned factors always reconcile exactly to the total change being explained, and every factor is located — attached to the level of the business where it acts, separating broad-based forces from problems localized to one segment.
Method notes and limitationsThe fine print, voluntarily
Synthetic by necessity
Ground truth doesn't exist otherwise. The generator is the same simulation framework we use for engine validation, not a showcase tuned for the engine: causes land at random hierarchy levels, in both directions, with random sizes.
Two-period comparison
The canonical variance question, and the hardest case. Multi-period analyses give the engine strictly more evidence to work with (finding 4).
Production code path
The engine seen by the benchmark is the code that ships in the Snowflake Native App — same defaults, no benchmark-specific configuration. Segments that launch from zero are handled by the same analysis but are not exercised here.
Scoring changed in October 2026
An earlier version of this page counted a returned factor as a hit if its location contained the injected one, and reported an earlier engine. Scored against random locations, that looser rule alone credited a long factor list with a third of the causes at high noise. These results use exact-location scoring and the current engine; the chance column is published so the difference stays visible.
Terminology
In this study, "cause" refers to the injected ground truth and "factor" to what the engine returns — the benchmark scores the second against the first. In technical prose like this we use the two interchangeably; in the product itself every reported item is a factor, located at the place in the business where it acts.
Run it on your own numbers.
The analysis runs inside your Snowflake account. Open a real Finding first, or bring one question to a guided evaluation.