Fluent and Wrong Can Look Exactly Like Fluent and Right
Why I built an AI analytics demo that refuses questions
Natural language makes analytics easy to ask and dangerously easy to fake, so I built a working demo that shows its work and refuses what it cannot verify.
The Demo AI Insights Dashboard Link
Analytics tooling has been climbing versions of the same ladder for roughly thirty years. Products such as Crystal Reports have been used since 1991, making recurring answers easier to distribute. Visual analytics moved more of the work into the hands of business users. Tableau was founded in 2003 around VizQL, its way of turning direct manipulation into database queries. Self-service tools and semantic layers then promised that governed definitions would let more people build the right views. Natural-language analytics did not arrive with the current model boom either: Microsoft introduced Power BI Q&A in 2013.
Each wave kept hitting the same wall, the gap between asking and trusting. Reports answered last quarter’s questions and many dashboards remained in place after the decision around them had moved on, while the question someone had that morning still required an analyst. Self-service made it possible for several people to bring several revenue numbers to the same meeting, each produced competently under a different definition. Natural-language query understood the words, but the words were never the whole problem. Revenue can mean the Finance view, net of contractual discounts, or the Sales Ops view used for territory scorecards. Both definitions can be legitimate. An interface that silently chooses one has made a policy decision while pretending to answer a language question without opinion.
The Insight Harness is a working demo of how I would balance tensions I expect to appear in a production system using modern AI: flexible language versus governed meaning, breadth versus refusal, explanation versus evidence and diagnosis versus causal attribution. Every numerical example below is computed by the running demo against a controlled, generated evaluation dataset, not production or customer data. I deliberately included source lag, metric variants and known causal effects so I could test those behaviors rather than merely stage them.
What LLMs genuinely changed
It is worth being precise about what is new, because both the hype and the backlash tend to skip it.
The first change is flexible intent. For years the burden sat with the person asking the question: learn the report catalogue, the dimension names, the filter syntax, the difference between two similarly named dashboards and the local dialect for dates. A language model can absorb much of that burden. Someone can type “which segments account for the revenue change” without knowing that the answer requires a comparison window and a pivot across several dimensions. This does not eliminate the need for governed definitions. It removes a substantial interaction cost that made many self-service systems feel less self-service than their name suggested.
The second change is tool orchestration. A model can translate a request and route it to deterministic machinery instead of trying to supply the answer from its parameters. The closest public analogue I found is Komodo Health’s Marmot, whose published architecture puts an LLM planner over SQL, cohort logic and curated tools, and exposes the interpreted question, plan, queries and returned data. Looker grounds conversation in a governed semantic model, while Databricks Genie exposes generated SQL and supports review and benchmark evaluation. That convergence narrows the claim. The demo’s contribution is a testable answer contract and stopping boundary within an established category.
The third change is synthesis. Given results from several tools, a model can assemble a readable account of what they show. Analysts have long done this connective work by hand. Models can reduce the time between calculation and explanation, particularly when the explanation must combine a metric result, its definition, its source and the limitations attached to both.
Those three gains share a root, and the root is also the hazard. A fluent wrong answer can look exactly like a fluent right one when all the reader sees is prose. Better models can reduce particular error rates, but prose alone still offers no reliable channel for detecting that a number came from the wrong table, the right table under the wrong definition, or a definition that changed since the previous meeting. The industry increasingly recognises the verification problem; for example, Power BI Copilot now attaches citations to report summaries. Citations help, but analytics often needs more than a link. It needs the resolved definition, source, calculation, data version and a way to reproduce the result.
That distinction matters. A citation can show which report or dataset the model consulted without showing which metric variant, filters, dates or aggregation produced the number. Those choices are often where analytics disagreements live. I use “receipts” to mean more than a source link: the structured intent, governed resolution, deterministic computation and versioned result have to travel together. The prose may change without altering the claim; if any of those inputs changes, the artifact should make the change visible. Verification has to reach the decision logic, not merely the document where the data was found.
The diagnosis is not mine alone. Benn Stancil argues that analysis is not self-validating, while Hamel Husain makes the product-design corollary directly: AI outputs should be built to be easy to verify. I agree with both. Their work frames why verification must shape the product and the build decisions below show how I would make that requirement concrete in a deliberately bounded demo.
The ladder of ambition
Not all analytics questions carry the same risk. Each rung buys more ambition and demands a higher standard of evidence.
The bottom rung is retrieval and counting. Show me the East-region accounts in the top revenue deciles with no activity in three months. The system returns eight accounts and prints the rule it applied: decile 8 or above on trailing-twelve-month revenue, no activity for at least three months, and region equal to East. This is cheap to check because the criteria are visible and every row either meets them or it does not. The cost is limited ambition. A governed filter can find a useful list, but it does not contribute meaningful explanation of the business.
The second rung is descriptive measurement, where definitions and sources become the battleground. “What is revenue in the South region?” sounds simpler than the account question, but the two registered sources and two registered definitions create four defensible configurations. The governed warehouse and net-revenue default returns 141.2 for June. The warehouse under the gross definition used by Sales Ops returns 164.9, which is 16.7 percent higher. The external panel feed returns 153.5 under the net definition and 177.9 under gross, both for May because that source lags the warehouse by a month. These values are in the evaluation dataset’s revenue units. None of the four is a calculation bug. A system that presents one without disclosing the choices has hidden the ambiguity, and that hidden policy choice becomes a credibility problem later.
The third rung is diagnostic decomposition, which is where much of the practical value lives. When someone asks why revenue moved, they often need to know where the movement sits before they need a causal theory. That part is arithmetic. In the March-to-June comparison, net revenue moved by minus 17.9, down 2.6 percent. The largest single contributor was the West region at minus 22.3, or 125 percent of the net change. A contributor can exceed the total because other regions moved in the opposite direction and partially offset it. The components reconcile to the total by construction, so a reader can inspect the result rather than accept a story about it.
The cost of decomposition is easy to miss precisely because the arithmetic is checkable. It shows where a change sits, not what caused it. West fell, and the mind supplies a reason before the next sentence arrives. Every decomposition that produces a breakdown carries that boundary in its output. If a registered event overlaps the March-to-June window, the interface points to the Causal Studio, but it does not transfer the decomposition’s certainty to a causal claim. Tableau draws a similar line in its own explanatory tooling: patterns and correlations are not evidence of causation.
The top rung is causal attribution. It requires a designed comparison, assumptions that can fail and human judgement about what the data does not contain. To test whether the causal workflow can recover a known answer, I created an evaluation scenario with an eight percent effect set before analysis: a partner enablement program beginning in the East region. A simple East before-and-after comparison reports plus 15.4 percent because it also absorbs everything else changing in the business. The registered difference-in-differences design compares East with the North and South control regions and estimates plus 9.0 percent. The estimate is close enough to the planted effect to pass this benchmark, but that does not turn synthetic recovery into proof of production causal accuracy.
The gap between 15.4 and 9.0 is why the top rung differs in kind. A model with the raw table can narrate the before-and-after figure persuasively. The narrative will not reveal the missing counterfactual. Causal systems need to expose their assumptions as part of the output; DoWhy’s approach centers the explicit modelling and validation of those assumptions. Controlled evaluation with a known effect is useful because ordinary observational data does not provide ground truth on demand, an evaluation challenge also addressed in work such as CausalImpact.
This ladder provides a practical test for any AI-analytics system. What evidence does it provide at its current rung, and what does it do when a question rises above the highest rung it can support?
What I decided, and why
I built this demo around a working assumption: each question belongs somewhere on that ladder, and the machinery should know where.
Answers carry reproducible evidence. Every computed answer is an artifact rather than a transient sentence. It contains the resolved metric definition and source, the engine that produced the result, a human-readable trace of the transformations, the underlying data version, a confidence tier, structured caveats and a stable twelve-character result hash. Refusals use the same artifact envelope but leave computation-specific fields empty while preserving the question class, reason, tier, data version and supported reframes. The trace is a reproducible recipe, not a claim that the interface has captured the literal Python statements executed inside every engine. Repeating the same question against unchanged data and configuration produces the same hash, and the standing evaluation runs every question twice and fails reproducibility when the hashes differ. When two people arrive with different numbers, the hash gives them a concrete place to start. The full artifact exports as JSON so someone with the matching code, data snapshot and registry can audit or reconstruct the calculation outside the interface. Accuracy, reproducibility and correct refusal are recorded across evaluation runs rather than treated as a one-time launch gate.
Definition and source variance stay visible. The South example has four configurations because source and definition are independent choices. The default answer follows the governed Finance-owned net definition on the internal warehouse. The default fork analysis recomputes each registered alternate one axis at a time, holding the other choice fixed and using the same engine and window. It flags a fork when the result moves beyond the governed materiality threshold, currently two percent. The demo allows both controls to be explicitly overridden to reach the fourth combined configuration, and the override is recorded. An alternate that cannot reproduce the same quantity is skipped rather than approximated. This is an important limit: showing a single decisive number may feel cleaner, but suppressing material variance converts a legitimate organisational disagreement into an invisible product decision. Finance and Sales Ops use their definitions for different decisions; in my design, the system’s job is to preserve which branch a downstream claim used, not to average the disagreement away. Governance changes are part of the record too. A changed materiality threshold or default variant is validated, persisted and timestamped so an earlier answer can be understood under the configuration that produced it.
Caveats come from registered facts and computed sensitivity. A model can append thoughtful-sounding caution to any number, but generic hedging soon becomes wallpaper. The harness builds ordinary answer caveats from concrete registry metadata, such as the panel feed’s one-month lag and known multiplicative bias, and from engine-aware divergence for descriptive and decomposition results. It says which month the source actually covers and how far a supported alternate moves the answer. It does not fabricate a like-for-like comparison when the alternate cannot support one. Causal designs are deliberately scoped differently: they do not receive the generic divergence treatment. Their source and definition sensitivity is computed inside the study brief alongside the design assumptions. When the system has no registered fact or calculated test to attach, it does not manufacture a caveat for atmosphere.
Refusal is a scored output. Ask the system to forecast next quarter’s revenue and it declines in these words: “This asks for a forecast, and no governed forecasting model is registered — any number would be a guess you couldn’t verify. The measured history is available instead.” Ask why morale is down and it declines for a different reason: the question asks for a causal claim no registered event supports. Ask for a customer happiness index and it reports that the metric is outside the registry. Each refusal offers re-askable questions the system can support, such as a measured trend or decomposition. The standing evaluation set contains seventeen questions, five of which should be refused. A run that helpfully invents a number for one of those five fails. The Reliability page records pass rate, reproducibility and correct-refusal rate across runs, making abstention part of measured product quality rather than an error state hidden from view.
Causal work begins with a design. A causal question proceeds only when it matches a registered event. The design itself runs on a supported metric, and any substitution is disclosed. For the enablement programme, the Causal Studio proposes difference-in-differences and computes the checks that can be computed:
· Parallel pre-trends: the slope gap between treated and control regions is plus 0.05 percentage points per month across the six months before launch, which passes the studio’s threshold.
· Post-period length: nine months of data follow the event, so the design does not receive a short-period flag.
· Sensitivity to definition: net revenue estimates plus 9.0 percent and gross revenue plus 9.0 percent, so the conclusion is stable across the registered variants.
· Sensitivity to source: the warehouse estimates plus 9.0 percent and the panel feed plus 8.3 percent, so the conclusion is also stable across the registered sources.
The displayed checks are a review surface, not an exhaustive proof that the design is valid. One crucial displayed check is left to a person and labelled for manual review: whether another event contaminated the control regions. The estimate stays at the Hypothesis tier before and after review. Analyst sign-off does not promote it to Verified or erase its assumptions. It adds an analyst-reviewed marker to the artifact and records the review in telemetry with the result hash and data version, leaving those provenance fields available for any later use of the estimate.
What I chose to give up
This demo costs something real. In the words of Thomas Sowell’s “A Conflict of Visions” book, “There are no ideal solutions, only trade-offs”.
In the demo, I gave up open-ended “why” on demand questions. If someone asks a causal question about an event nobody has registered a priori, the system will not answer it. That can be frustrating because a general chatbot will usually produce a paragraph, and the paragraph feels like progress. I gave up some conversational magic too. A demonstration of the Insight Harness is less theatrical than a demonstration of a system that answers everything. The constraints interrupt the smooth exchange at exactly the moment when the evidence runs out.
I also accepted the operational work this design would impose in a real deployment. A registry would need owners. Definitions would change. Events would have to be recorded before the causal workflow could use them. Materiality thresholds would need governance, evaluation questions would need maintenance, and flagged answers would need attention. Refusal is cheap only in the moment of answering; converting repeated refusals into safe capabilities takes engineering, analytical design and organisational agreement. The demo makes that work visible instead of making it disappear.
What those constraints buy is a basis for belief. Trust in an analytics product does not come only from an accuracy score. It grows when a reader can inspect an answer, reproduce it, see when an alternate definition changes it and identify the boundary where the system stopped. A clean refusal puts a floor under the product because it limits the damage of an unsupported claim. An always-answering interface has no comparable floor.
Expansion therefore has to be earned. In the demo, an answer or refusal can be flagged, and those flags accumulate in the correction record beside the evaluation history. When a useful class of question is repeatedly declined, the response is to build the governed machinery that makes it answerable, register it and add it to the standing set. Forecasting is not excluded forever. It remains excluded until a registered forecasting model exists and its errors can be measured. At that point the present refusal becomes false, and the right product change is to remove it.
The test I hold this demo to is simple. Does it shorten the path from a question to an answer someone can verify, rather than merely to an answer they can receive? Does it become more trustworthy with use as its record accumulates and its refusals are converted into governed capabilities, or less trustworthy as small unexamined errors compound?
That is the standard the Insight Harness has to earn, one checkable answer and one honest refusal at a time.