When an enterprise AI system gives an unreliable answer, the usual diagnosis is that the model was not good enough. The usual remedies follow: a stronger model, a longer prompt, more retrieved documents or a larger context window. Each can help. None addresses a failure that is both more common and harder to detect in analytical systems: a model can reason correctly over incorrectly selected context.
In practice, the arithmetic is correct and the explanation is coherent, yet the answer is wrong. The system chose the wrong evidence, followed the wrong relationship between tables or resolved an ambiguous business term into a definition the user never intended. The error was made before any calculations began.
This distinction matters as enterprises move from conversational AI experiments to systems that support business decisions. An answer can be logically consistent and fully traceable to a source while still being analytically wrong because the system selected evidence that did not match the question.
The challenge is not simply to make models reason better. It is to ensure they understand what needs to be answered, identify which evidence is eligible and follow a defensible path from the question to the result.
This article examines why that process breaks down, why it becomes harder as enterprise data estates grow and how context engineering can make analytical workflows more reliable. It draws on our survey-analytics work, presented at DataHack Summit 2026, and published research.
A Denominator Nobody Defined
Take a routine question: On average, how many hours does Cam work per day?
The calculation looks trivial, but the word “day” is not defined. It could mean:
- Every calendar day in the reporting period
- A standardized working year
- Each roster record
- Each distinct date on which Cam actually worked
Each interpretation could produce a defensible number. But only one may reflect what the business user intended.
Dividing total hours by calendar days will produce a lower average than dividing by days actually worked whenever Cam has days off. Neither denominator can be judged appropriate until the intended meaning is clear.
A conventional output check might confirm that the totals reconcile and the arithmetic is correct. It would not necessarily detect that the system selected the wrong denominator.
This is the distinction between validating a calculation and validating the analytical decisions behind it. The latter requires checking the population, filters, denominator and relationships between the data sources used.
The stakes rise when a question spans multiple metrics, reporting periods, respondent groups and sources. An incorrect assumption made at the outset can shape every subsequent step without triggering an obvious error.
Business Questions Are Compressed Analytical Programs
“Which brand improved trust this quarter?” reads like a lookup. Answering it requires a series of decisions that the question itself does not specify.
Which questionnaire item represents trust? Is it a direct response, a top-box score or a derived key performance indicator? Which survey waves make up “this quarter”? Who is in scope? Are the results weighted? What counts as improvement?
The question is effectively an analytical program whose parameters have not yet been resolved.
Consider what happens when an enterprise LLM receives this request. It identifies a table labelled “Brand Trust”, retrieves the latest available values and compares the two periods. The calculation runs correctly and the model generates a clear explanation.
But the selected table may measure the proportion of respondents who chose the highest trust rating, while the business user expects an overall trust KPI. The reporting period may also contain only some of the survey waves required for a quarter-on-quarter comparison.
The system has answered a plausible interpretation of the question, not necessarily the intended one. No arithmetic error is required.
Context engineering addresses this upstream problem by resolving the analytical choices before calculation begins. The system establishes what trust means in the study, which reporting periods qualify, who belongs in the respondent base, whether weighting applies and which source is authoritative. If any of these choices remains ambiguous, the system should seek clarification rather than silently commit to an interpretation.
The objective is not simply to help the model understand the words. It is to make the intended analytical operation explicit.
Survey data sharpens the problem further. A questionnaire may appear linear but behave as a conditional system: answers trigger skip logic, determine which brands appear later and change who is eligible for subsequent questions. Once the questionnaire is flattened into a wide export table, those relationships become difficult to see.
A model that sees only the exported columns may retrieve a relevant-looking value without understanding the conditions under which that value was collected.
Why Scale Makes It Worse
In a small dataset, the evidence is usually obvious. In an enterprise data estate, several tables may carry a “trust” metric. Multiple survey waves may resemble the requested period and a crosstab may exist for every market, segment, question variant and weighting scheme. The answer is present but hidden among alternatives.
The system therefore faces two distinct problems: discovery, or where the answer might be, and resolution, or which candidate actually matches what the user meant.
A top-ranked result is not necessarily a valid one. Similarity shows that a table resembles the question. It does not establish that the table is fit to answer it.
The research points in the same direction. Spider 2.0, an enterprise text-to-SQL benchmark of 632 real workflow problems, was released in late 2024. Its strongest system, an agent built on o1-preview, solved about 17% of tasks, against roughly 91% for leading systems on the original Spider benchmark [1]. The leaderboard has since moved considerably, with its Snow track listing a vendor-submitted score of 96.7% from a named agent system [2].
My reading is that the strongest entries are engineered agent systems rather than bare models. This supports the argument that much of the improvement comes from the systems built around the model. Such benchmarks primarily measure performance on specified tasks. They tell us less about whether a system can recognise that a business question is ambiguous before attempting to answer it.
Long context is no shortcut either. Chroma’s “context rot” study found that models became less reliable as input length grew, even on simple tasks, and that similar-looking distractors made the problem worse [3].
Giving an enterprise LLM more documents can increase the amount of available information without making the right evidence easier to identify. The challenge is not to supply everything the model might need. It is to establish which information matters, why it matters and how it relates to the question.
How This Differs from Context Engineering as Usually Framed
Anthropic describes context engineering as the natural progression of prompt engineering: managing an agent’s entire context state and treating context as a finite resource with diminishing returns [4]. That framing is sound. It concerns what enters the context window and how to keep it useful and focused.
The problem examined here sits one step earlier: what does the question mean and which evidence is eligible to answer it?
Retrieval finds content that resembles the question. A semantic layer can define metrics consistently. Neither necessarily determines which definition the user intends or which of several similar tables is appropriate for the current analytical task.
That decision requires a structured path from intent to evidence.
Context as a Path, Not a Pile
Under uncertainty, the reflex is to give the model more information. But additional information can multiply competing interpretations and force the model to reconstruct relationships the system already knows.
A more reliable approach is to organise context sequentially: the intent of the study, then the questionnaire structure, questions and definitions, followed by segments, markets, survey waves and weighting rules, and finally the crosstab that holds the evidence.
Each layer inherits constraints from the one above. A table earns its meaning from the sequence of context surrounding it, not from its column names alone.
The same ingredients, arranged as a pile and as a path, can produce very different outcomes. A pile gives the model multiple definitions and candidate sources and asks it to infer which combination is appropriate. A path establishes the dependencies that make a source valid before the model uses it.
A Five-Step Workflow for One Business Question
In practice, the difference can be implemented as a series of explicit decisions.
For the question, “Did trust in Brand X improve in India last quarter?”, the workflow could:
- Resolve the intent. Establish whether “trust” refers to a specific survey question, a top-box score or a derived KPI.
- Apply the study rules. Identify the eligible respondent base, relevant survey waves and weighting scheme.
- Identify the evidence. Retrieve candidate crosstabs whose definitions match those constraints rather than relying on table names or semantic similarity alone.
- Validate the route. Check that the selected source uses the intended metric, population and reporting period and that the comparison is methodologically consistent.
- Calculate and explain. Use an authorised tool to retrieve the values and compute the change, then present the result with its definition, respondent base, weighting and source.
Each decision narrows the next. Instead of asking the model to reconstruct analytical logic from a collection of documents, the system supplies the relationships and constraints needed to reach the correct evidence.
The exact implementation will vary across enterprise data environments. The underlying principle remains the same: resolve meaning first, select evidence second and calculate only when the analytical path is sufficiently clear.
The outcome is a navigable structure rather than a larger prompt. The model remains important, but it operates within a controlled decision process instead of improvising the entire analytical workflow.
Five Principles That Make the Path Governable
Implementation will vary, but five principles should guide the design.
1. Build Context in Sequence
Later definitions depend on earlier ones. A question label depends on the study objective. A crosstab description depends on the question, survey wave, weighting and market.
Make these dependencies explicit so that a layer is produced only after the necessary context above it exists. This reduces the risk of selecting a source before the system has established what makes that source valid.
2. Retain What Has Been Approved
Researchers may describe the same concept differently: “purchase propensity”, “intent to buy” or “conversion likelihood”. If labels are regenerated on every run, vocabulary can drift and previously resolved definitions may be interpreted differently.
Treat approval as durable feedback, scoped with care. One person’s vocabulary must not overwrite study-level definitions or leak across projects. Accepted terms should be versioned so that changes can be tracked and their impact assessed.
3. Place Human Review Where the Error Began
A final approval screen comes too late when a mistake originated upstream. Recalculating only the output preserves a flawed definition. Rebuilding everything is costly and can obscure what actually needs to change.
Return the workflow to the affected layer, apply the correction there and carry it forward. The reviewer should correct the system’s understanding, not just one answer.
This also makes feedback reusable. If a subject matter expert corrects the definition of a metric, the system should apply that correction to the relevant workflow rather than requiring the expert to catch the same error repeatedly.
4. Let Tools Supply the Numbers
Once the question is resolved, the model should not invent the mechanics of data access.
Give it narrow tools to find eligible tables, confirm metric definitions, retrieve values, compute a defined change and trace each output to its source. Every reported number should come from an authorised tool result.
Separating interpretation, evidence selection and calculation makes it easier to identify where an error occurred. It also narrows the scope for fabricated figures and unsupported numerical claims.
5. Know When to Stop and Ask
A retrieval system always returns a top candidate. That does not make it a good one.
A sound routing mechanism considers both the strength of the best candidate and how clearly it outperforms the next-best option. If the match is weak or two candidates are nearly tied, the system should pause and ask for clarification. If the route is clear, it should proceed.
Thresholds must be calibrated against actual enterprise queries and the cost of error. A low-impact exploratory question may tolerate more uncertainty than a figure used in a consequential business decision.
The principle outlasts any particular threshold: uncertainty at the routing layer must remain visible.
These controls carry a real cost in dependencies, retries, versioning and tool contracts. In return, they provide inspectable control over how meaning is accumulated, which a long prompt chain cannot easily deliver.
An Illustrative Case: Two Plausible Answers to the Same Question
A brand manager asks: “Did trust in Brand X improve in India last quarter?”
Two crosstabs are plausible matches. Both contain the trust measure for the correct market and quarter. One reports the top-box score among all respondents, weighted. The other reports it among respondents aware of the brand, unweighted. Their descriptions differ by only a few words, so a similarity-based retriever ranks them almost identically.
One shows a modest rise. The other shows a modest decline.
A system that selects the top-ranked candidate returns a confident, fully sourced answer that may well be wrong. The numbers are real and the computation is correct. The problem is that the system has selected the wrong population and weighting scheme, and nothing in the final output reveals that a choice was made.
A path-based system takes a different approach. It checks the candidates against the resolved constraints and identifies that they differ on the respondent base and weighting. If those details have not been established, the system pauses and asks the user which population and methodology the analysis should follow.
Once the ambiguity is resolved, it retrieves the eligible source, calculates the change and records the respondent base, weighting and source alongside the result.
The contrast is straightforward:
| Poorly contextualised query | Effectively contextualised query |
|---|---|
| Retrieve the closest trust table, calculate the change and return the result. | Establish the intended trust measure and respondent base, resolve the weighting and period, select eligible evidence, validate the route and calculate from the authorised source. |
The cost of the intervention is one clarifying question. The cost of its absence is a wrong figure that looks identical to the right one.
The lesson is not that retrieval should always stop when multiple sources exist. It is that the system must distinguish between candidates that are genuinely interchangeable and those whose differences change the meaning of the answer.
Practical Controls for Reliable and Traceable Outputs
A structured context path is only useful if enterprises can govern and inspect it. Reliability depends on controls across the analytical workflow, not just on the quality of the initial prompt.
Several practices are particularly important.
Maintain Governed Definitions
Document business metrics, respondent bases, filters and calculation rules in a form the system can retrieve and apply. Definitions should identify the conditions under which a metric is valid, not just provide a label and description.
Restrict Evidence Selection
Allow the model to select from authorised sources that meet explicit eligibility criteria. Do not treat the highest similarity score as sufficient evidence that a table is appropriate.
Separate Interpretation from Execution
Resolve the question and its analytical constraints before allowing tools to retrieve data or perform calculations. This reduces the risk of the system committing to an unsupported interpretation and building the rest of the workflow around it.
Preserve Provenance
Record the source table, metric definition, filters, weighting scheme, reporting period and calculation used to produce each result. Where relevant, retain the identifiers or versions needed to reproduce the query and inspect the evidence later.
Version Context Assets
Track changes to definitions alongside changes to the underlying data. When an upstream definition changes, reassess dependent outputs rather than allowing outdated context to persist.
Design for Uncertainty
Define the conditions under which the workflow must ask a clarifying question, request expert review or decline to produce a result. A system that can recognise insufficient evidence is more useful than one that always produces an answer.
Why Traceability Matters
These controls make an answer more than a number accompanied by an explanation. They provide a record of how the system interpreted the request, why it selected particular evidence and whether the resulting calculation can be reproduced.
Traceability matters because a plausible explanation is not proof that the correct evidence was used. An enterprise system must be able to show the analytical decisions behind the answer, not simply generate a convincing narrative about them.
Evaluate the Route, Not Only the Answer
Most AI evaluation focuses on the final answer. That is insufficient when the main risk is misrouting.
A model can reach the right figure through the wrong analytical path because two tables happen to agree. A system can also correctly decline to answer an ambiguous question, yet a basic accuracy metric may penalise it for not returning a number.
The unit of evaluation should therefore be the decision trajectory.
- Was the user’s intent represented correctly?
- Were the cohort, metric and denominator resolved correctly?
- Was the right table and relationship path chosen?
- Did values come only through authorised tools?
- Did the system stop when ambiguity was too high?
- Can every reported number be traced to a source?
- Did reviewer feedback change the correct upstream definition?
A practical validation programme should test three things separately.
Test Interpretation
Use representative business questions with known definitions and deliberately ambiguous wording. Check whether the system resolves the intended metric, population, period and denominator or recognises when clarification is necessary.
Include questions that use different terms for the same concept as well as questions that use the same term to mean different things in different studies. This helps establish whether the system is applying the correct definition rather than matching words alone.
Test Evidence and Execution
Verify that the selected source is eligible, joins and filters are appropriate, calculations use authorised values and each reported figure can be traced to its origin.
Include cases where two plausible sources produce conflicting results and cases where different sources happen to produce the same number. The latter are important because output accuracy alone may conceal an incorrect route.
Where expected results are known, compare the selected source, applied constraints and calculation with the validated analytical path. Where no single interpretation is justified, check whether the system appropriately asks for clarification instead of forcing a result.
Test the Review Process
Route unresolved definitions, competing evidence and high-impact findings to the appropriate subject matter expert. When a reviewer identifies an error, correct the layer where it originated and test whether that correction carries through to subsequent decisions.
Human oversight should be designed into the workflow rather than reserved for a final approval screen. Experts are most valuable when they validate the assumptions and evidence that determine the answer, not merely the wording of the final response.
Together, these tests help distinguish a lucky answer from a reliable system. They also make evaluation more useful for improvement: teams can identify whether a failure originated in intent resolution, context retrieval, evidence selection, execution or validation.
What Context Engineering Does Not Fix
A structured path can still be wrong.
Documentation may be incomplete, survey logic may change between waves, approved labels may entrench an outdated interpretation and confidence scores may be poorly calibrated. Sequential context can also propagate an early error. If the first layer is wrong, downstream consistency can make the result look more credible, not less.
Context engineering does not eliminate the need for data quality, sound analytical methodology or expert judgement. Nor does a traceable workflow automatically make an answer correct. Traceability makes it possible to inspect and challenge the decisions that produced it.
Several controls are therefore non-negotiable: version context assets alongside the data, preserve the provenance of generated and approved definitions, reassess downstream artefacts when an upstream definition changes, test routes with known and deliberately ambiguous questions and retain an execution trace for every published insight.
Validation must continue after deployment. Enterprises should monitor for changes in data structure, metric definitions and survey logic, as well as patterns of incorrect routing or unnecessary clarification. Human oversight remains essential, placed where expert judgement has the greatest leverage.
The aim is not to remove humans from the analytical process. It is to direct their attention towards the decisions where their expertise can prevent the greatest downstream cost.
Inspect the Path Before Replacing the Model
The lesson is not that models cannot support enterprise analysis. It is that model capability cannot compensate for an undefined analytical path.
When the data is bounded and the target is clear, an LLM can reason transparently. When intent is distributed across questionnaires, schemas, weights, waves and hundreds of near-identical tables, the central engineering problem changes. The system must establish what the question means, determine which evidence qualifies and validate the route before trusting the result.
For any answer that looks confident but turns out to be wrong, five questions can help locate the fault:
- What did the system decide the user meant?
- Which evidence did it treat as eligible and what did it exclude?
- Which analytical path did it follow?
- Where was that path validated?
- Did it know when not to proceed?
Reliable enterprise AI begins before generation. It turns intent into constraints, constraints into a route and the route into an auditable sequence of evidence and operations.
Before replacing the model, inspect the context path.
References
[1] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025). Project page: https://spider2-sql.github.io/
[2] Spider 2.0 leaderboard, Snow track (scores are submitted by the teams; accessed October 2026): https://spider2-sql.github.io/
[3] Chroma Research, Context Rot: How Increasing Input Tokens Impacts LLM Performance: https://research.trychroma.com/context-rot
[4] Anthropic Engineering, Effective context engineering for AI agents (29 September 2025): https://anthropic.com/engineering/effective-context-engineering-for-ai-agents
Mayank Chaturvedi is a Senior Manager, Insights & Analytics at Dentsu Global Services. Previously at Amazon, CommerceIQ, Olam and Bira 91, he has built AI and data functions from the ground up and authored two international white papers. He has been visiting faculty at MIT Sloan, NUS, ISB, Wharton and UC Berkeley.
Login to continue reading and enjoy expert-curated content.
