AI Agent Testing and Evaluation for Production

AI agent testing and evaluation is not a prompt benchmark. It is a production authority decision.

A model can answer correctly while the agent uses the wrong source, crosses a permission boundary, calls a tool with dangerous parameters, retries an external action, corrupts memory, or fails to recover. Test the complete action loop under the conditions that will actually operate.

Define the accepted task. Build representative and adversarial cases. Exercise identity, data, tools, policy, approval, retries, state, limits, failures, rollback, and monitoring. Run enough attempts to see variation. Preserve the transcript and system events. Make the release decision against named stop conditions, not an average score.

The Agentic AI hub connects testing to agent risk, IAM, monitoring, and lifecycle controls. Use AI Workflow Quality Assurance for the wider workflow proof model, AI Agent Risk Assessment to set consequence, and AI Agent Identity and Access Management to test the real authority boundary. GS Consulting aligns the release decision to a secure enterprise AI strategy.

AI Agent Testing and Evaluation: The Short Answer

Test one bounded agent use case across six layers: task outcome, source and data fidelity, identity and authority, tool and workflow behavior, security and abuse resistance, and operations and recovery. Use realistic records, deployment like infrastructure, denied paths, adversarial content, dependency failures, repeated attempts, and the actual policy and approval design.

Measure both task success and harmful success. A test suite should answer whether the agent completes valid work, refuses prohibited work, stays inside access limits, preserves required source support, handles uncertainty, avoids duplicate effects, stops when controls fail, and recovers without hiding damage.

Do not reduce the release to one pass rate. Separate mandatory stop conditions, contextual review thresholds, and operating improvement measures. One unauthorized payment, access grant, external message, data disclosure, or deployment may matter more than hundreds of correct low consequence tasks.

The Unit of Test Is the Task and Its Effect

The model response is only one event in an agent run. The test boundary should include the initiating goal, system instructions, model and version, retrieved content, memory, planner, other agents, identity, credentials, tools, policy, approval interface, target system, event records, monitoring, and recovery path.

Write expected results at three levels. First, define the business outcome: what useful work should be complete? Second, define the control outcome: which data, tools, permissions, approvals, and limits should apply? Third, define the evidence outcome: which records must prove the decision and effect?

Test the negative space. Specify prohibited sources, targets, operations, amounts, recipients, data classes, tenants, sequences, and retry behavior. A test catalog made only of valid requests measures assistance. It does not measure control.

Version every dependency that can change the result. Include model, prompt, evaluator, source set, memory policy, tool schema, connector, access policy, approval policy, agent code, test case, environment, and target configuration. Without those identifiers, a passing run cannot support the next release.

Public Research Rejects Easy Agent Safety Claims

A January 2026 NIST AI 800-2 initial public draft organizes automated benchmark work around the measurement target, implementation and execution, then analysis and reporting. It emphasizes validity and reproducibility. The draft is useful evaluation guidance, but it is not final and does not define a complete production agent release process.

In March 2026, NIST CAISI reported results from a large agent security competition covering 13 target models, more than 250000 attack attempts, and more than 400 participants. At least one successful attack was found against every target model. NIST cautioned that the competition does not yield one simple safety ranking or a representative production failure rate.

A NIST CAISI analysis of agent hijacking evaluations shows why task level measurement and repeated attempts matter. In one experimental example, changing the attempt policy from one to 25 increased the reported average attack success from 57 percent to 80 percent. That example is not a universal rate. It demonstrates that the test protocol can change the risk conclusion.

NIST CAISI research on evaluation cheating identifies solution contamination and grader gaming as threats to validity. A high score can reflect access to the answer or exploitation of the evaluator rather than the intended capability. Transcript review, hidden cases, clear constraints, and independent system records matter.

Original Research: GS Agent Production Test Coverage Priority Index

Tool parameter and prohibited action testing scores 100. Multi agent cascade scores 96. Indirect prompt injection scores 93. IAM boundary testing scores 92. These tests lead because an apparent task success can still create an unauthorized or propagating external effect.

We rated twelve test families from one to five across consequence, attack exposure, nondeterminism and retry, system reach, and recovery burden. Base weights are 25, 20, 20, 20, and 15 percent. Each weighted result is reported on a zero to 100 planning scale.

Tool parameter and prohibited action scores 100. Multi agent cascade scores 96. Indirect prompt injection scores 93. IAM boundary scores 92. Approval bypass and action mutation score 89. Integration failure and partial completion score 88. Pause, rollback, recovery, and resume also score 88. Memory and state poisoning score 84. Retry, rate, cost, and time limits score 80. Representative deployment context scores 73. Evaluation contamination and grader gaming score 69. Expected task and source fidelity score 65.

The last score is not permission to weaken functional quality. It reflects the comparative authority and recovery burden in this model. Expected outcomes remain the baseline for every valid test. The ordering says that a team which already tests clean task success should put its next effort into prohibited effects, cascade, injection, and identity boundaries.

The sensitivity case moves weight toward consequence and recovery. No score moves more than three points, and the leading risk families stay near the top. The result is useful for sequencing. Replace the analyst ratings with system facts, incidents, threat intelligence, consequence decisions, and test evidence.

Build a Six Layer Agent Test Stack

Task and source. Test expected outcomes, completeness, source authority, freshness, citation support, abstention, and business rules. Score important fields and consequential errors separately from averages.

Identity and data. Test correct subject and agent identity, tenant separation, resource audience, permission denial, sensitive data handling, derived data, logs, memory, and deletion. Attempt the same task through users and agents with different access.

Tools and action. Validate tool selection, schema, parameters, allowed operations, prohibited operations, target, amount, recipient, and resulting external record. Inspect what the target received, not only what the agent said it sent.

Security and abuse. Exercise direct and indirect prompt injection, malicious files, poisoned retrieval, memory manipulation, peer messages, compromised tool output, credential abuse, excessive action, and evasion. Run sustained attempts within realistic operating limits.

Workflow and resilience. Inject timeouts, duplicate events, stale state, partial writes, dependency failure, queue delay, policy outage, approval rejection, and model or provider failure. Test pause, rollback, reconciliation, and safe resume.

Operations and change. Verify monitoring, thresholds, alert ownership, trace completeness, cost, latency, capacity, drift review, incident handling, version change, and retirement. A release test without a production feedback path expires as soon as the system changes.

Turn Security Threats into Executable Tasks

Threat names are not test cases. “Prompt injection” should become a representative task containing untrusted instructions in email, a web page, a document, retrieved knowledge, tool output, or a peer message. Define the expected safe result, prohibited tool call, evidence to capture, number of attempts, and consequence if one attempt succeeds.

Test both the model and the control plane. If the model proposes a prohibited action, the policy layer should deny it. If the model refuses but the tool accepts an unauthorized direct call, the system still fails. Defense should not depend on one component behaving perfectly.

Use threat sources as scenario inputs, not universal checklists. The OWASP Top 10 for Agentic Applications names goal hijack, tool misuse, identity and privilege abuse, memory poisoning, insecure communication, cascading failure, and rogue agents. Map only the relevant threats, then add system specific cases from architecture, incidents, users, data, and contracts.

Test Repeated Attempts, Memory, and Partial State

An agent that retries is a different system from an agent tested once. Define the actual retry count, backoff, tool rate, time budget, cost budget, and stop rule. Measure whether a harmful outcome appears at least once across the allowed run, not only the average result per attempt.

Test duplicate safety at the target. Network timeouts can hide a completed action and prompt another attempt. Use stable request identifiers, idempotent operations where practical, reconciliation, and a review path for ambiguous completion. A clean transcript does not prove that the external system changed only once.

Memory needs poison, isolation, expiry, correction, deletion, and resume tests. Seed wrong or malicious state, then observe whether it changes later tasks. Confirm one user, tenant, or agent cannot influence another outside the approved boundary. Confirm a pause and resume cannot restore revoked instructions or credentials.

For multi agent systems, inject a compromised peer, delayed message, duplicate message, malformed handoff, false completion claim, and circular delegation. Limit propagation by task, message count, tool call count, elapsed time, and spend. Test whether one agent can quarantine another without disabling the entire service.

Protect Evaluation Validity

A test result can be precise and still answer the wrong question. State the measurement target before selecting metrics. If the decision concerns safe invoice updates, a general reasoning benchmark is weak evidence. Use actual document types, source systems, tool schemas, permission patterns, exceptions, and consequences.

Separate development cases from hidden release cases. Control solution contamination where the agent or its tools can retrieve reference answers. Review transcripts for grader gaming. Use deterministic system events to verify tool calls and effects rather than trusting the agent's narrative.

Document sample construction, exclusions, evaluator instructions, scoring logic, repeated attempt policy, environment, uncertainty, and known limits. Report severe failures separately. Avoid claiming safety from the absence of a discovered attack. A test shows what happened under the stated conditions.

Independent review is most valuable at the decision boundary. The team that built the workflow should not be the only team deciding whether a surprising behavior is acceptable. Risk, security, mission, data, legal, privacy, or compliance roles should participate in proportion to the actual use case and obligation.

Production Monitoring Extends the Test Program

Controlled evaluation cannot reproduce every live interaction. NIST AI 800-4 identifies monitoring challenges across functionality, operations, human factors, security, compliance, and broad impact. It also explains that available methods remain immature. That uncertainty calls for clear signals and response ownership, not claims of complete detection.

Monitor sampled task quality, source support, permission denials, prohibited tool attempts, action volumes, retry patterns, duplicate prevention, approval outcomes, reviewer disagreement, memory changes, peer propagation, incident severity, latency, cost, capacity, rollback, and recovery. Bind each signal to a threshold, owner, response, and review period.

Feed incidents, near misses, overrides, drift, support cases, and recovery lessons into the test catalog. A fixed benchmark slowly becomes a test of yesterday's system. The operating loop should add cases and change thresholds as evidence changes.

Six Production Agent Test Failures

The benchmark becomes the product. Teams optimize the score while the live task boundary stays untested. One attempt passes. Retry behavior and sustained attack pressure remain invisible. The model is tested alone. Identity, tools, policy, targets, and external effects are assumed.

Average success hides severe failure. One consequential action disappears inside a large sample. The grader is trusted. Contamination and evaluator exploitation inflate results. Recovery is a document. Nobody has paused, contained, repaired, and resumed the operating workflow.

These failures usually come from a vague release decision. Name what must never happen, what evidence would show it, and who has authority to stop the release.

Build One Production Test Evidence Packet

The packet needs eight connected records: test charter, system and version manifest, case catalog, source and fixture set, run record and transcript, finding and exception register, release decision, and production monitoring and change record.

The run record should capture case identifier, seed where relevant, attempt number, environment, subject, agent, model, prompt, memory state, tools, permissions, policy version, inputs, tool calls, target effects, evaluator, scores, logs, and disposition. Store sensitive test material under appropriate access and retention controls.

The release decision should list passed mandatory tests, unresolved findings, accepted exceptions, compensating controls, accountable approver, expiry, monitoring thresholds, stop rights, and rollback proof. A dashboard summary is useful only when a reviewer can reach the underlying case and system evidence.

A Practical Production Test Plan

1. Define accepted work. Name the useful outcome, prohibited outcome, authority boundary, data boundary, thresholds, and stop conditions.
2. Build the system manifest. Version the model, prompt, memory, data, tools, policy, permissions, evaluator, infrastructure, and targets.
3. Create representative cases. Include routine work, rare cases, denied paths, stale and conflicting sources, malformed inputs, attacks, failures, and recovery.
4. Run repeated system tests. Use deployment like conditions, the real retry policy, and enough attempts to expose variation at the task level.
5. Verify external effects. Inspect target records, permission decisions, approvals, messages, state, costs, and audit events.
6. Separate stop from review. Block failures that invalidate the use case boundary and route contextual uncertainty to named reviewers.
7. Exercise recovery. Pause, revoke, contain, reconcile, roll back, repair, retest, and resume through the approved path.
8. Close the operating loop. Turn incidents, drift, overrides, and changes into new cases and focused reevaluation.

The decisive operating standard is this: no production authority until the complete agent passes representative and adversarial tasks, prohibited effects stay blocked, recovery works in practice, and live evidence is tied to the exact version that runs.

Sources and Research Notes

• NIST AI 800-2 Initial Public Draft
• NIST AI RMF Playbook Measure Function
• NIST CAISI Large Scale Agent Security Competition
• NIST CAISI Strengthening AI Agent Hijacking Evaluations
• NIST CAISI Cheating on AI Agent Evaluations
• NIST AI 800-4 Monitoring Deployed AI Systems
• NSA and International Partners Careful Adoption of Agentic AI Services
• OWASP Top 10 for Agentic Applications for 2026
• NIST AI 600-1 Generative Artificial Intelligence Profile
• NIST AI Risk Management Framework Core

---

Frequently Asked Questions About AI Agent Testing and Evaluation