Evaluating Live and Voice Agents with Native Evaluation Tools

Picking up the pace of AI development, Google’s ADK has introduced a suite of native live evaluation tools to help developers assess their live and voice agents more efficiently.

The traditional method of evaluating live agents – scripted demos or limited simulations – often failed to capture real-world interactions. Now, with this new feature in place, that all changes.

Developers can create a live evaluation loop within ADK by using a simulated user that speaks its turns as audio and scores the spoken replies.

This process is integrated into the same eval loop used for text agents, streamlining testing and providing more accurate results. The developers say this helps identify potential issues early on.

The example provided by Google demonstrates how to build a complete live evaluation loop using graph-based workflows. They set up three single-purpose live agents, each running on gemini-live-2.5-flash-native-audio.

This setup allows developers to test their agent’s performance across multiple stages of interaction – like handling a conversation that spans several topics or tasks.

Each stage in the workflow is an ordinary live agent, with the workflow orchestrating them and carrying output from one stage to the next. The user never notices anything unusual happening because control moves seamlessly between agents.

The accumulated session state and conversation history are carried forward so each agent picks up where it left off rather than starting cold. This makes testing more like real-world interactions.

Developers test their agents using a JSON file containing test cases, which can be decoupled from how they run. This flexibility allows for different styles of testing: conversation scenarios and fixed conversations.

In the first scenario, developers describe a goal and persona in the JSON file, with the user simulator improvising turns based on that plan. The simulated user’s communication is shaped by the user_persona attribute in the JSON file.

The ADK ships with built-in personas, including NOVICE, which tells the simulator to share high-level goals and wait for the agent to ask for specifics – helping test how well the agent drives conversations.

Personas are prompt-driven rather than hardcoded, allowing developers to extend the set with their own custom personas. The simulator ends a scenario on its own once the conversation_plan is satisfied, letting developers script the goal and decide when the call is finished.

As an added safeguard against run-off conversations, max_allowed_invocations caps the total number of turns. Developers can also author fixed conversations by scripting user turns verbatim – with static cases valid inputs to live runs as simulated users are.

This flexibility allows for more comprehensive testing and helps ensure agents perform well in real-world scenarios. The configuration file, test_config.json, contains critical settings like enabling live mode via the live_model_config field.

Omitting this setting or using a standard text model means running the exact same test cases – just not in live voice mode. Model vs. audio_model distinction is also important: model powers simulated user’s turn-taking logic while audio_model synthesizes those turns into speech.

Adjusting voice_name and language_code allows developers to test agent performance against different voices and accents, further testing its real-world adaptability. Criteria configures metrics and pass/fail thresholds – useful for multi-agent graphs where more complex interactions are involved.

A conversation can be correct in hundreds of different phrasings, but natural-language rubrics judge intent the way a human reviewer would: captured once and applied automatically across every conversation in your suite.

This ensures agents not only accurately respond to questions but also effectively handle real-world interactions – with no single solution working for all scenarios. With their agent, eval set, and configuration ready, developers can run the evaluation from the CLI using uv run adk eval.

The same pipeline can be called programmatically via AgentEvaluator, making it easy to drop live voice evaluations into CI/CD pipelines – catching regressions before shipping, so development stays on track.

For interactive debugging, ADK Web now natively supports live evaluations. The run setup dialog includes a Standard | Live mode toggle: selecting Live reveals input modality options and voice settings for the simulated user.

Once the run completes, ADK rebuilds the live audio stream into a clean transcript – each turn rendered in its own message bubble complete with inline playable audio clip and text, allowing developers to evaluate how their agent sounded as much as what it said.