Semantic layer lifts AI accuracy on business questions

AtScale has published research suggesting that adding a semantic layer improves AI accuracy on business questions across seven models. The study found that, without the layer, leading models answered fewer than four in 10 business questions correctly.

The research used the independently developed BIRD-Interact benchmark, a test set with 600 tasks across 22 databases. Of those, 410 were business questions and 317 had a verifiable correct answer.

The results highlight a persistent problem for companies trying to apply large language models to finance, sales and operations. Models querying databases directly struggled with basic business concepts such as revenue, margin and active customer when they lacked governed business definitions.

Across the 410 business tasks, adding the semantic layer increased accuracy by 72% to 137% for all seven models tested. On the 317 business questions with a verifiable answer, Claude Sonnet 5's accuracy rose from 46% to 93%.

The research also suggested that open-weight models can remain close to better-known proprietary systems when given the same business context. With the semantic layer, Kimi K2.5 scored 49.3% on business questions, within 1.2 percentage points of OpenAI's GPT-5.

One of the sharper differences emerged in follow-up questions. Google's Gemini 3.1 Pro asked more clarifying questions than any other model in the test, yet without a semantic layer it passed only 10.2% of follow-up questions, the lowest result among the models assessed.

With the semantic layer, Gemini 3.1 Pro reached 45.6% on those follow-up questions, which the research described as a 345% increase. AtScale said the benchmark was designed to reflect ambiguous prompts, messy data and simulated user conversations resembling real enterprise workflows.

Business definitions

The findings come as companies look for ways to move AI from pilot projects into operational work, where answers must be consistent and auditable. Financial reporting, forecasting and other business processes can depend on common definitions that remain stable across teams, dashboards and applications.

AtScale argued that a semantic layer can serve as that common reference point by applying shared definitions and governed computation before an answer is returned. In practice, the model handles the language of the question, while the business logic sits in a separate governed layer.

Chris Lynch, Chief Executive Officer at AtScale, said semantic layers will alter the economics of enterprise AI, according to the company's account of the research. The benchmark also highlighted four areas where AI systems tend to guess on business questions: disambiguating prompts, resolving vague semantics, handling dirty data and calculating complex multi-pass metrics.

Jay Schuren, President at AtScale, said: "Large language models are probabilistic by design. Finance and operations demand consistent, auditable answers. That mismatch is one of the biggest barriers to putting enterprise AI into production. A semantic layer bridges the gap: the model reasons about the question, while governed business definitions and deterministic computation produce the answer. That's how AI agents move from experimentation into the workflows that run the business, and how model providers unlock the next wave of enterprise adoption."

Methodology questions

AtScale described the findings as preliminary and outlined differences in how some of the tests were run. The reported results used the best of three runs per model with the semantic layer, following the BIRD-Interact leaderboard convention, compared with one run per model without it.

For the 317 answerable questions, the company said it applied the same grading corrections to both approaches and compared the results with the average of three runs without the semantic layer. It added that those figures are not comparable with the leaderboard's success rate.

The research underlines how difficult it is to assess whether general-purpose AI models are ready for business-critical use without additional controls around data meaning and calculation rules. Even after improvement, the reported scores indicate that many models still answered fewer than half of the tested business questions correctly.