Agent and Model Evaluations Now Generally Available in Gemini Enterprise Agent Platform

Evaluating the quality of agents is crucial during development, but it’s equally important to measure their performance after launch. The Agent Platform has introduced agent and model evaluations as a generally available feature, allowing users to assess and compare agents at both stages on one engine with consistent metrics.

When using consistent quality scoring for local experiments and live traffic, any drift in production points to an issue with the agent rather than its measurement method. This ensures that issues are identified promptly and addressed accordingly.

The primary unit of work in evaluation is the experiment, which consists of a dataset of eval cases and a set of metrics and scores to run and review as you iterate. The evaluation service comes equipped with a flexible UI for defining metrics, kicking off new runs, and reviewing results down to individual failures, where users can access the agent’s full trace and session log.

Experiments can be run locally for rapid iterations or on existing sessions and traces if an agent is already deployed on Agent Platform with telemetry enabled. The system also supports issue clustering, grouping eval failures into interpretable clusters against a user-defined taxonomy of failure reasons.

For large evaluation jobs, the service uses pre-built taxonomies such as adaptive rubrics that cover common ways agents go wrong. More than 20 pre-built metrics are included in the service, with computation-based metrics scoring deterministically against ground-truth references like ROUGE for summarization and BLEU for translation.

Beyond these metrics, an advanced LLM-judge metric workflow co-developed with Google DeepMind’s research partners creates case-specific pass/fail tests from eval cases, developer instructions, and tool declarations. This adaptive rubric grades traces against the generated rubrics, providing verdicts and rationales per rubric.

The evaluation service includes a range of metrics for assessing agent performance, including Task Success, Tool Use Quality, Safety, Trajectory Quality, Final Response Quality, Hallucination, Grounding, image and video quality. Natural-language guidelines steer rubric generation toward specified criteria, with resulting rubrics reviewable and reusable across agents.

When pre-built metrics don’t fit a specific use case, users can bring their own code-based or LLM-as-a-judge metrics. Code-based metrics are Python functions covering exact text matches and JSON-shape checks, while LLM-as-a-judge metrics carry custom criteria, rating scales, and judge models.

Regardless of how metrics are defined, they land in the same versioned registry running unchanged across offline experiments and online monitors. The system is actively expanding coverage to more specialized tasks and input modalities, with a welcome for user feedback.

The evaluation service meets users where they build agents, being part of the Agent Platform SDK with operations available over REST API for languages and pipelines not reached by the SDK. For command-line work, agents-cli makes eval a first-class command alongside deployment and telemetry inspection tools (ADK-Python agents today).

Users can even run evaluation loops from their coding agent using reusable skills that walk Claude Code and similar tools through the full agent-quality flywheel. For teams building on ADK, evaluation is built into the framework – define eval sets and run them against your agent locally as you develop, including inside pytest for CI.

The Evals Worksheet web UI provides a grid to review runs without writing code. The system also integrates with Cloud Trace, allowing users to evaluate existing traces directly with one click in the traces UI. Better still is setting up online monitors to grade live traffic as it arrives, using built-in sampling and targeted filters to manage costs.

Every evaluation needs test cases to run against, which can be generated by the service’s case generator. This tool seeds synthetic eval cases from agent instructions and tools, reducing manual effort and capping coverage at what users thought they could type out. For ADK agents, it pairs with a user simulator to reach scenarios hard to produce manually.

The environment simulator stands in for systems an agent calls, allowing users to test how the agent handles failing or slow backends without touching production. Agent and model evaluations are now generally available, with supported regions and enterprise security features listed in the regions and security features table.