Supabase Releases Evals: An Open-Source Benchmark for AI Agents
Supabase has made its benchmark and framework, Supabase Evals, available as an open-source tool. This release allows developers to test the performance of various coding agents on real-world tasks, providing a comprehensive evaluation of their capabilities.
Evals is designed to assess how well AI agents can build, debug, and maintain applications using Supabase’s services. The benchmark runs against actual containerized stacks, not mock environments, ensuring that results are accurate and reliable.
The framework scores the performance of coding agents like Claude Code, Codex, and OpenCode on a range of tasks, including building schemas, debugging Edge Functions, and fixing broken RLS policies. These scenarios cover various dimensions, such as products (database, auth, storage), topics (RLS, security, migrations), and stages (build, deploy, investigate).
Supabase defined the smallest scenario set that touches each dimension once, grounded in support tickets, bug reports, and GitHub issues. The benchmark scenarios are published for public evaluation, while regression scenarios cover known failure modes and refresh daily.
The framework boots a hosted-like stack and a local CLI project in containers, allowing agents to call the actual MCP server and CLI. A platform-lite runtime exposes a Management API-compatible surface backed by @supabase/lite. Scoring combines deterministic checks with LLM-as-a-judge, giving one retry before grading.
Each eval directory contains PROMPT.md (task plus frontmatter), EVAL.ts (the scorer), and optional remote/ and local/ starting states. Shipping a local/ workspace or declaring interface: cli boots a Docker sandbox with the real CLI installed.
The results show that agents pass most scenarios without loaded skills, but skills significantly improve performance for smaller models. Opus 5 and Kimi K3 scored 100% unaided in the Build stage, while Sonnet 5 rose from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89% with skills loaded.
However, three weaknesses surfaced: agents hand-write migrations instead of using declarative schemas, prompting a skill guidance update; agents verify auth by hand rather than reaching for @supabase/server, prompting a package selection guide; and docs usage varies sharply between coding agents, even when skills are loaded.
The release of Supabase Evals marks an important step in evaluating the performance of AI agents on real-world tasks. By providing a comprehensive benchmark, developers can better understand their capabilities and limitations.