Excellent Demo, Unproven Solution
It’s been a week since go-live for your latest conversational analytics solution. The semantic view was built in a few days, and your agents answered all the demo questions, so the business should be happy, right? Then an analyst asks an unexpected question that wasn’t part of your test set, and the number comes back wrong. An engineer finds the source of the error, and a fix is live in an afternoon, but now there’s a new problem. A question the agent used to get right now responds with false information. This raises a big concern: are there other questions that used to work but are broken now? There are dozens of test questions, with dozens more variations. Manual testing would take far too long.
Answers Don’t Mean Facts
Semantic views are not new, and most data teams can describe one by now: a schema-level Snowflake object holding logical tables, relationships, dimensions, and the formulas behind metrics, so Cortex Analyst answers from deterministic definitions instead of guessing from column names. That ground is well covered. The Snowsight UI even has a built-in AI tool, Semantic View Autopilot, for generating a semantic view from your existing data in Snowflake.
With semantic views being simple to create now, a new problem has reared its head. Agentic tooling made building semantic views far faster, without making the results more trustworthy. Being wrong faster still means you’re wrong. Outputs can be plausible, well-formatted, and confidently incorrect, which is hard to catch with a quick skim of the results. Gartner predicts that by 2030, half of AI agent deployment failures will be attributed to insufficient governance and runtime enforcement rather than to the models themselves (Gartner Announces Top Predictions for Data and Analytics in 2026, 2026). Vibe-coded semantic layers are one of the ways teams become one of these statistics.
The Business Challenges It Solves
Three problems sit between a semantic view that demos well and one you would put in front of a CFO.
- No Reliable Metrics
Many teams would answer the question “How accurate is your semantic layer?” with at worst an anecdote or at best a spreadsheet of manual test results. Anecdotes are an obvious red flag, but manual test results are more subtle. They are time-consuming to run and are invalidated the moment the semantic layer changes, so they are incredibly difficult to scale. - Hidden Domino Effect
Adding a synonym or tightening a metric to fix one error may very well create new ones. Without a suite of tests run every time the semantic view changes, every fix simply trades known issues for an uncertain number of uncertain errors. - Upstream Drift
A semantic view can be perfect and still return wrong answers when the tables beneath it go stale. Running Snowflake native data metric functions (DMFs) introduces proper data-quality measurements that can flag foundational data issues before they break a conversational analytics solution.
Building It with CoCo
CoCo (formerly Cortex Code) bundles a semantic-view skill just for this work: creating and refining semantic views over gold tables, defining metrics and dimensions, and mapping relationships. Point it at a set of tables, describe the use case, and it analyzes the schemas, infers the relationships, drafts dimensions and metrics, and generates verified queries – natural-language questions paired with the SQL that correctly answers them.
Two things to get right at this stage. The first is a proper, human-in-the-loop review. An SME needs to sign off on definitions, metric formulas, join keys, and anything else contained in the semantic view, because a missing join can silently break things downstream. The second is a narrow scope. Snowflake recommends five to ten tables and one business domain, and starting too big can slow down the overall development process by making initial errors more difficult to diagnose.
Scoring It with Cortex Analyst Evaluations
This is the integral step that separates a production-grade conversational analytics solution from a snazzy demo.
Cortex Analyst evaluations test a semantic view against the records in its verified query repository (VQR). During a run, Snowflake builds a temporary copy of the view with the evaluated queries removed, so the model cannot grade its own homework. Evaluation is done using an LLM-as-judge approach, where a separate judge model is given instructions on how to grade Cortex Analyst’s output and returns a consistently structured score. Results are retrieved with GET_ANALYST_AI_EVALUATION_DATA, which returns a table carrying the input, the generated output, the ground truth, and an aggregate score per record.
Each evaluation run returns an accuracy percentage, the expected versus generated SQL for every question, latency, and an explicit list of regressions, meaning queries that used to pass and now fail. Scoring runs on sql_correctness, which is currently the only metric Cortex Analyst evaluations support, so the accuracy number means one specific thing rather than whatever you decided to measure. The judge model versions are pinned, so a score from this week is comparable to a score from last month without worry. In addition, the whole thing is scriptable. Run it from the Evaluations tab in Snowsight while you iterate, then call EXECUTE_AI_EVALUATION with a staged YAML config and wire it into CI, so no semantic view change ships without a scored run behind it.
That is the difference between “we built a semantic view” and “our semantic view answers ninety-four percent of the verified set, up from eighty-one, with zero regressions this release.” One is built on blind trust; the other is built on verified metrics.
Above and Below the Semantic Layer
Upstream, DMFs run on a schedule and surface data quality violations in Snowsight, both per table and account-wide. DMFs show whether the data answering those questions is current, complete, and within expectations by measuring metrics such as null-counts, freshness, and row-count drift, or custom metrics your engineers can define. A quality, proven semantic layer won’t matter if the data foundation is full of cracks.
Downstream, Cortex Agent evaluations score the reasoning rather than only the natural-language-to-SQL conversion. Using a Goal-Plan-Action framework, this evaluation suite measures tool selection accuracy, tool execution accuracy, answer correctness, and logical consistency across an agent’s steps, with the two tool-level metrics still in public preview. This is also where custom, LLM-scored metrics of your own can sit alongside those four. CoCo’s cortex-agent skill covers creating, debugging, evaluating, and optimizing those agents with Snowflake’s own best practices and patterns. Once an agent is choosing between Cortex Analyst, Cortex Search, or a custom tool, there’s a whole other layer of evaluations to run.
7Rivers Data Native® Model
This loop spans the whole Data Native® Model rather than living in one stage of it. DMFs are Foundation work, the semantic view and its verified query repository are Insights, and agents with their own evaluation suites are Actions. Most clients arrive with pieces of all three and no measurement connecting any of them, and that gap is where 7Rivers works.
A typical conversational analytics engagement runs in order: discovery is done on the tables in the target domain, semantic views are built with CoCo, reviewed by the business owner who actually owns those definitions, hardened with verified queries drawn from questions people really ask, then scored to establish a baseline before anything gets wired into CI. As a Snowflake Elite Partner, 7Rivers treats that first semantic layer as a template rather than a deliverable, because the second domain is where the discipline either holds or falls apart.
From Vibes to Verified
Generating a semantic view is now the cheapest part of the project, but this doesn’t make it a build-and-forget process. When anyone can produce something plausible in an afternoon, plausible stops being worth much, and the teams that succeed are the ones who can prove their semantic views enable Cortex Analyst to respond with the right SQL, not just runnable SQL.
The toolkit to do that is already sitting in your Snowflake account. Now, it’s simply a matter of being the team that stands up a scored, regression-tested semantic layer, instead of being the one that builds on a foundation of guesswork and hope.
If you are building conversational analytics on Snowflake and want a production-grade solution, you don’t just need to generate a semantic view and plug it into Cortex Analyst; you need to validate it with a robust evaluation suite that proves your solution works, which will build trust with your userbase.
If your semantic layer has never been scored, that is the place to start. 7Rivers runs a semantic layer readiness assessment that begins exactly there: one domain, a verified query set built from real questions, and a baseline you can improve against.
Sources & Further Reading
