What it does
Receives what Tier 1 could not settle
Deterministic AST and invariant checks clear 70–80% of items. The remainder — a query that is syntactically different from the reference but may be semantically equivalent, an aggregation whose grain is ambiguous — goes to EvalML.
Compares plans, not text
Generated SQL is parsed to a SQLGlot AST, normalised, and compared against the valid equivalence set for the invariant — the family of formulations that satisfy it. Execution plans from the warehouse are compared where available.
Emits a calibrated probability
Not a verdict: a probability that the item violates the invariant, with the features that drove it. Above a routing threshold it goes to a Tier-3 reviewer; below it, the item passes with the probability recorded.
Is scored like a reviewer
EvalML has a Brier score and a Brier skill score against adjudicated truth, tracked per dialect and per failure family, exactly as a human reviewer does. When a human overturns it, that is data.
Why it is not sold separately
EvalML is not sold separately. It is the engine behind every sprint and every platform run, and its cost is inside those fees. Running it against your own traffic, in your own infrastructure, is an enterprise agreement.
The market publishes per-call prices for LLM judges that differ by an order of magnitude on the same nominal unit. That spread means the unit is not commoditised and, more importantly, that a price quoted before the judge’s accuracy is published is exactly the kind of unbacked number this site argues against. Before we quote anything for EvalML, three numbers go on the methodology page:
- Judge–human agreement per failure family and dialect, with bootstrap confidence intervals
- P50, P95 and P99 latency at a stated load
- The routing thresholds that decide when a human sees an item
When those exist, the price follows them. Not before.
What it is not
- Not a foundation model evaluator. It judges query equivalence against a customer’s invariants, not model quality in the abstract. We publish no leaderboards.
- Not the last word. Every Sev-1 goes to humans. EvalML routes; it does not sign.
- Not trained on your data. It is built on synthetic and internal fixtures. Customer opt-in for de-identified calibration examples trains reviewers, not the model.
Where it sits in tiered review → Ask about an in-infrastructure deployment