Continuous assurance for the agent your board thinks somebody already checked.
The platform keeps an invariant suite compiled from your own dbt manifest and schema, runs it inside your warehouse on every pull request and every week in production, and delivers a signed assurance case with dollar-denominated exposure — so production clearance is a decision made on evidence, and remains one after every schema change.
Built for one behavioural trigger, not a firmographic
Demographic filters do not predict assurance demand. One thing does: an LLM agent that generates SQL against a production warehouse and whose output someone downstream trusts.
An agent in production, or about to be
Snowflake Cortex Agents or Cortex Analyst, Databricks Genie, or a custom LangChain / LlamaIndex Text-to-SQL pipeline, answering questions whose answers reach a dashboard, a report, a customer or a regulator.
Someone who owns the go/no-go
A CDAO, VP Data or governance lead who has to sign a residual-risk acceptance and would rather sign it against a signed assurance case than an internal demo.
Insufficient access or definitions
We cannot verify what we cannot see or what nobody can define. Without read-only schema metadata and a named Business Definition Owner the engagement is declined, professionally and in writing. Full scope & exclusions →
Compiled from your metadata. Executed in your compute. Signed every week.
Metadata & manifest ingestion
We read your dbt
manifest.jsonand warehouseINFORMATION_SCHEMA— model grain, foreign keys, tests, effective-dated columns, row access policies. No table contents. On every schema change the suite is re-synthesised, so coverage does not silently decay.Automated invariant generation
Join fan-out traps, null foreign keys, temporal range boundaries, SCD-2 point-in-time joins, tenant isolation predicates and sargability checks are synthesised automatically — the target is 70% of a standard suite in under 15 minutes PLANNING ASSUMPTION. Human authoring is reserved for your proprietary high-consequence logic.
Risk-adaptive sequential execution
Low-risk formatting invariants run 5 trials, medium-risk aggregations 20, and high-risk financial or authorisation invariants use sequential probability ratio testing up to 59 trials — enough to say, at 95% confidence, that the true failure probability is under 5%. A challenge-suite hash and an ephemeral nonce are committed before execution so results cannot be manufactured.
Adjudication into four states
PASS,DEFECT_CONFIRMED(reproducible trace and counterfactual proof captured),EXCLUDED_UNSUPPORTED(out of contracted surface or dialect) orINDETERMINATE(ambiguous definition → your Business Definition Owner’s 48-hour SLA). Sev-1s go to two blinded reviewers and a lead adjudicator.Signed bundle, in your perimeter
A dual-signed HTML + JSON audit bundle lands in your S3 bucket or Snowflake internal stage. Only its hash reaches our stateless notary. Anyone you choose can verify it without seeing a byte of your data.
Numbers that change a decision, reported with their uncertainty
Every metric is reported with a 95% Wilson interval computed by standard libraries (scipy.stats, statsmodels), never a hand-derived formula.
Incremental Consequential Finding Yield
Share of critical and high defects we find beyond your native platform evaluation. Contract target ≥ 40%. If it is under 20% across five customers, our own kill rule fires.
Consequential Defect Discovery Rate
Critical findings per 100 evaluated agent traces, tracked release over release so you can see whether the agent is getting safer.
Risk-Weighted Assurance Coverage
Share of the pre-frozen business schema surface covered by active invariants. The denominator is frozen before execution and signed by both parties, so coverage cannot be inflated after the fact.
Decision Change Rate
Share of findings that altered a go/no-go or prompt-engineering decision. Target ≥ 30%. Findings that change nothing are a signal that we are testing the wrong surface.
Finding Overturn Rate
Share of reported defects overturned during adjudication. Target under 5%. Published in every bundle, because an assessor that is often wrong should be visibly often wrong.
Consequence-tier suite allocation
Tier 1 financial/payroll agents: ≥50% independent adversarial cases. Tier 2 operational: ≥40%. Tier 3 exploratory BI: ≥30% — the contractual floor below which no engagement proceeds.
Meets your data team where they already work
evalqaPython CLI. Runs the suite in GitHub Actions or any CI against a Snowflake staging branch on every pull request; exits non-zero on Sev-1.- Snowflake native stored procedure.
CALL EVALQA_RUN('finance', 'tier-1')executes the evaluation entirely inside your Snowflake compute — zero infrastructure egress. qabitOpenTelemetry adapter. MIT-licensed. Streams runtime agent spans (prompt, generated SQL hash, tool calls) for production observability of the agent in service.
Three evidence modes. Zero raw data out by default.
Local-only automated execution
100% automated inside your VPC or Snowflake sandbox. Only hashes, invariant status, row counts, latency and the signed manifest leave. No human review.
Sanitised evidence escalation
For exception triage: pseudonymised AST snippets and execution plans, 14-day TTL, encrypted under a per-tenant KMS key that is destroyed at contract end.
Customer-hosted restricted VDI review
External expert review on sensitive schemas happens inside a VDI you control, clipboard and downloads disabled. Remote visual inspection is treated as regulated processing.
Evidence modes, assurance levels and the runner adversary model → · Security & data handling →
Before the call
The person accountable for an analytics agent that already executes SQL against a production warehouse, or is about to: usually a Chief Data & Analytics Officer, a Head of Data Platform, or the AI governance lead who has to sign the residual-risk acceptance. The trigger is behavioural — a Cortex Agent, Genie space or custom Text-to-SQL pipeline in production — not company size.
Read-only access to your schema catalog (INFORMATION_SCHEMA) and your dbt manifest.json, or a one-off run of our local metadata extractor. A named Customer Business Definition Owner who can resolve INDETERMINATE findings within 48 hours. And a consequence tier for the agent, which we agree together in week one.
It is recorded as INDETERMINATE, never guessed. The contract gives your Business Definition Owner a 48-hour window to resolve the semantic definition ("active customer", "net revenue"); the invariant is then re-run and lands in PASS or DEFECT_CONFIRMED.
No. Developmental separation is part of the Independence Charter: EvalQA personnel never author production prompts or dbt models for an audit client. Remediation is done by your team or a Qualified Implementation Partner; we re-test it.
Thirty minutes. No deck.
Tell us which agent worries you. We will look at how it is evaluated today, say which of the 18 failure domains apply to its schema, and tell you honestly whether a Baseline Sprint would surface anything you cannot already see. If the answer is no, we say so.
- Reply within one business day
- Design partner cohort: 3 slots, card-billed, on the Common Paper DPA v1.3
- Or email [email protected]
Received.
We will reply within one business day with two or three times for the review.