ADK Adds Native Live Evaluation for Voice Agents

Google ADK now supports native evaluation for real-time voice agents. The Google Developers Blog announcement published on August 24, 2026, introduces live audio simulation, multi-turn trajectory scoring, and playback inside the existing ADK testing workflow.

You can use the capability to test a single voice agent or a multi-agent graph, verify tool calls and business rules, and run the same scenarios from the command line in CI/CD. The workflow uses an LLM-driven caller simulator instead of fixed text prompts, so the agent must handle interruptions, turn-taking, spoken responses, and changing conversation context.

What live evaluation adds to ADK

Traditional text-based agent tests validate messages and tool calls against predetermined inputs. Live evaluation adds the audio path to that harness. ADK can generate a caller’s spoken turns, stream them to the agent under test, reconstruct the resulting conversation, and score the complete trajectory.

The main component is the llm_audio user simulator. A model such as gemini-3.7-flash manages the simulated caller’s turn-taking logic. Text generated for each caller turn is converted to speech through a model such as gemini-3.1-flash-tts-preview, then streamed to the live agent.

This distinction matters for systems built on multimodal models such as gemini-live-2.5-flash-native-audio. A transcript-only test can pass while the live system mishandles pauses, turn boundaries, or spoken context. Audio evaluation exposes those failures within the same test case.

For background on broader agent test design, see this guide to evaluating and testing AI agents. If your application uses several specialized agents, the explanation of multi-agent systems provides useful context for structuring the graph under test.

Step 1: Define the agent or multi-agent graph

Start with the production-shaped workflow you want to evaluate. The example architecture described for ADK uses a greeter_agent, a dob_verifier_agent, and a goals_agent. These agents run on gemini-live-2.5-flash-native-audio and can call a tool such as validate_date_of_birth.

Keep the graph identical to the version used by your application wherever possible. Live evaluation is most useful when it exercises the actual routing, session state, tool execution, and guardrails rather than a simplified test double.

A sequential graph can validate a fixed call flow, such as greeting the caller, verifying their date of birth, and collecting goals. More complex routing can test whether the correct specialist receives a turn, whether control returns to the coordinator, and whether state survives each handoff.

Treat each agent boundary as an evaluation point. A final success score does not tell you which agent failed, while turn-level messages and tool execution logs can show whether the problem occurred during routing, identity verification, or response generation.

Step 2: Author scenarios as eval cases

Create eval cases in JSON with the caller details and the goal of the conversation. Each case should describe a realistic interaction, not a list of exact sentences for the simulator to repeat.

Useful scenario fields include the caller’s identity information, the desired outcome, facts the caller is allowed to disclose, and conditions that should trigger a refusal or escalation. For example, a case can require the agent to verify the caller’s date of birth before sharing sensitive information.

Separate the conversation objective from the expected wording. Live agents can satisfy the same goal with different phrasing, and a rigid transcript comparison would classify valid responses as failures. Rubrics are better suited to judging whether the required behavior occurred.

Include cases for normal completion, missing information, incorrect verification data, ambiguous requests, interruptions, and attempts to bypass a guardrail. The simulator can then exercise the agent across multiple turns instead of stopping after a single response.

Step 3: Configure live audio and scoring

The live behavior is controlled through test_config.json. Set live_model_config for the live session, including timeout_seconds: 300 when a conversation needs a five-minute evaluation window.

Set user_simulator_config to use llm_audio. Configure the caller simulation with the selected turn-taking model, the text-to-speech model, a voice profile, and a locale. The configuration can specify voice_name: "Kore" and language_code: "en-US" when you want a US English caller voice.

The important configuration choices are shown below.

Configuration area	Example value	Purpose
User simulator	llm_audio	Generates spoken caller turns and manages multi-turn interaction
Turn-taking model	gemini-3.7-flash	Determines how the simulated caller responds during the call
Speech synthesis model	gemini-3.1-flash-tts-preview	Converts caller turns into streamed audio
Voice profile	Kore	Selects the simulated caller voice
Locale	en-US	Sets the caller language and regional voice configuration
Live timeout	300 seconds	Limits the duration of a live evaluation session
Trajectory rubric	rubric_based_multi_turn_trajectory_quality_v1	Scores the complete conversation trajectory

Use natural-language rubric criteria to express business requirements. A criterion such as “verify caller identity before sharing sensitive information” evaluates the sequence of actions, rather than checking only whether a phrase appeared in one response.

Rubrics can also score individual turns and tool function executions. That lets you check whether a response was appropriate at a specific point, whether validate_date_of_birth was called with the correct information, and whether the graph continued only after verification succeeded.

Rubric-based judging introduces an important tradeoff. LLM judges are more flexible than exact string comparisons, but their scores can vary with ambiguous criteria. Write each rubric as an observable requirement with a clear pass condition. Keep separate criteria for safety, business logic, tool behavior, and conversational quality so a single aggregate score does not hide a critical failure.

The full parameter set and current setup requirements are available in the live voice evaluation documentation.

Step 4: Run the evaluation from the CLI

Run the evaluation suite through the ADK CLI with adk eval. This keeps live tests in the same automation path as other ADK evaluations and makes them suitable for regression checks in CI/CD.

Run a small set of representative cases during local development. Reserve larger scenario collections for scheduled or pre-release runs because every case involves model inference, speech synthesis, live audio handling, and rubric judging. A failure can therefore indicate a change in the agent, a tool, routing logic, audio behavior, or evaluation configuration.

Use stable scenario identifiers and retain the rubric results for each run. Compare failures at the case and criterion level instead of relying only on a single suite-level pass rate. This makes it easier to identify a regression such as a new prompt causing the agent to skip identity verification while all other behavior remains correct.

For teams already building an automated quality process, this live harness fits naturally with a quality flywheel for agent evaluation. Add live cases to the same release gate as text and tool-use tests, but keep audio-specific criteria distinct from text response criteria.

Step 5: Inspect results in ADK Web

After the run, open ADK Web and select the live evaluation results. The run setup dialog includes a Standard | Live mode toggle, which distinguishes ordinary evaluations from live audio sessions.

ADK Web reconstructs the recorded stream into a structured transcript. Each turn appears as a message bubble with transcript text and an inline playable audio clip. Review both representations. The transcript shows what the system recognized and generated, while the audio reveals delivery problems such as awkward pauses, clipped responses, or an unsuitable voice configuration.

Inspect the tool execution logs alongside the conversation. A caller may sound as though they provided the required information, yet the agent may have extracted it incorrectly or invoked a validation tool at the wrong time. The combination of transcript, playable audio, tool logs, and rubric scores provides the evidence needed to isolate that failure.

Playback also helps distinguish model behavior from graph behavior. If the agent gives a correct response but routes the next turn to the wrong specialist, the transcript and execution trace point to orchestration. If the route is correct but the spoken response is incomplete, the audio and turn-level score point to the live model or speech handling.

Tradeoffs to account for

Live evaluation offers higher fidelity than static text testing, but it also consumes more resources and introduces more moving parts. The simulator model, speech synthesis model, live model, timeout, network path, and rubric judge can all affect a result.

Keep the caller persona and evaluation criteria consistent when comparing agent revisions. Changing the voice, locale, simulator instructions, or rubric at the same time as the agent makes the result difficult to interpret.

Use text evaluations for fast checks of deterministic tool behavior and basic instruction following. Use live evaluations for turn-taking, spoken interaction, multi-agent handoffs, audio quality, and end-to-end business rules. The two modes complement each other rather than replacing one another.

Start with a compact eval set covering successful calls and the highest-risk guardrails. Run it with adk eval, inspect failed turns in ADK Web, then expand the scenario set as new production failure modes appear.