Evaluation datasets
JSONL evaluation cases turn agent behavior into repeatable test inputs instead of one-off manual prompting.
CASE STUDY · AI RELIABILITY · OPEN SOURCE · 2026
AGENTGATE
An evaluation and CI quality-gate platform for AI agents — built to make answer quality, grounding and regression risk testable before agent changes are merged or deployed.
VIEW SOURCE CODE ↗WHY I BUILT IT
AI agents need regression tests that understand behavior, not only deterministic code output.
Traditional unit tests work well when the same input should always produce the same output. Agent systems are different: generated answers, tool calls and judged quality can vary while still needing a reliable release process.
AgentGate turns that problem into an engineering workflow. Evaluation datasets define expected behavior, an evaluator produces metrics, thresholds decide whether the run passes, and CI can block a change when quality drops below the configured gate.
EVALUATION PIPELINE
01Agent ChangePrompt or implementation
→02DatasetRepeatable evaluation cases
→03EvaluatorHeuristics + optional LLM judge
→04Quality GatePass or block CI
CORE SYSTEMS
Evaluator API
A FastAPI service scores cases with deterministic heuristics and can optionally use an OpenAI LLM-as-a-Judge provider.
CI quality gates
Threshold failures return a non-zero CLI exit code so regressions can block a pull request or deployment pipeline.
Developer CLI
The agentgate eval command provides a small developer-facing surface for running evaluation suites locally or in CI.
Trace-ready data model
Projects, datasets, runs, cases and traces are modeled in PostgreSQL/Supabase for persistent evaluation history.
Dashboard foundation
A Next.js dashboard presents production-style metrics and trace previews while keeping the evaluator as a separate service.
DEVELOPER WORKFLOW
01change agent
02run evals
03score cases
04check thresholds
05merge / block
The same evaluation suite can run locally through the CLI and inside GitHub Actions, keeping the release criterion explicit instead of relying on subjective spot checks.
TECHNICAL STACK
Python · FastAPI · heuristic scoring · optional LLM judge
Node.js CLI · JSONL datasets · reusable GitHub Action
Next.js · React · TypeScript
PostgreSQL · Supabase schema · evaluation runs and traces
GitHub Actions · evaluator tests · CLI syntax checks
ENGINEERING DECISIONS
Deterministic first
Local heuristic scoring keeps the foundation testable without requiring an external model for every evaluation run.
LLM judge is optional
Model-based judging extends the evaluator for semantic quality while keeping the system usable when no AI provider is configured.
CI-native failure
Threshold failures map directly to process exit codes so evaluation quality can become a required engineering gate.
Extensible boundaries
Dashboard, evaluator, CLI, action and persistence are separate surfaces so each part can evolve without collapsing into one monolith.
WHAT THIS PROVES
The project treats AI quality as a measurable software concern rather than simply adding a model call to an interface.
The product includes an evaluator API, CLI and GitHub Action designed around an actual engineering workflow.
Agent regressions can be evaluated before merge or deployment through explicit thresholds and repeatable cases.
Evaluation, persistence, UI and CI integration are separated into clear architectural responsibilities.
NEXT CASE STUDYPULSEOPS AI ↗