Agentic AI | | 24 min read
AI Agent Risk Assessment: Authority, Data, and Recovery
An AI agent risk assessment should start with the job you're about to approve. I'd ask the team to show which records it can read and what it can change. Then try the failures you're concerned about. The owner needs those results, the safeguards, and the remaining risks before deciding how much access to allow.
Key Takeaways
What the assessment needs to show
Cascading failure scores 89
In the GS planning model, cascading failure comes first at 89. Broad consequences combine with gaps in controls, difficult detection, and hard recovery when agents depend on one another.
All 13 target models had a successful attack
NIST reported at least one successful attack against each of the 13 target models in a competition with more than 250,000 attempts. That result needs the testing context explained below.
Show that recovery works
Before granting permissions with serious consequences, have the team exercise pause, revocation, rollback, repair, and safe resume. I would want that proof before approving more access.
An AI agent risk assessment starts with the records the agent could read and the changes it could make. The review also needs to account for whoever can put instructions in front of it. If an agent makes a bad change while nobody is watching, someone has to detect it and work out what can still be repaired.
I'd take one use case into that discussion, with the person who owns it. What is the goal, and who are the users? We need the agent identities, data, memory, tools, and permissions on the record. We also need the external inputs it might encounter, the actions it could take, the conditions for stopping, and the way back after a failure. Then we can try the failures we are worried about and see whether the controls prevent, detect, contain, and reverse them.
There are several related guides if you're working through the design. The Agentic AI hub brings together lifecycle, access, monitoring, and evidence. For identity and delegation, I'd use AI Agent Identity and Access Management; for the full action loop, AI Agent Testing and Evaluation. Multi Agent AI Security addresses jobs passed between agents. Put those choices into the wider secure enterprise AI strategy so the assessment has an organizational owner.
Plan an agent risk review
If you're considering more production access for an agent, GS Consulting can help map its authority and work through the failure cases. We can help define the control evidence and exercise recovery before the new permissions are used.
Plan an Agent Risk ReviewWhat is an AI agent risk assessment?
The owner needs to decide whether this agent should do this job. Follow what it can reach and do, then test how it fails. The five-part process below puts evidence behind that decision, including reasons to limit or refuse the work. A score alone won't do it.
Reading public research and changing a financial record are different jobs. So are sending a message, deploying code, granting access, and directing another agent. I would bring the approval back when the job changes, rather than assume the old decision covers the new authority.
Frame the workflow before choosing a tool
The two-page GS AI Decision Boundary Checklist is a place to start if the team hasn't agreed on the scope yet. Use it for one workflow, working through the data, approval, and recovery questions. Keep the owner's name and supporting evidence with each decision so the next review has something to work from.
Download the ChecklistWhat belongs inside the assessment?
Suppose the request says the agent will draft an account change. I'd still ask the developer to show the credentials. Can they read the directory? Write the entitlement? Send a notification? Try again after a denial? Nobody needs to have used those capabilities in a demonstration for them to belong in the assessment.
The model endpoint is only one part of that path. The application and prompt bring in retrieved material; the agent framework and its memory can carry it forward. Tools and credentials turn the resulting request into action. Include the approval interface, message bus, observability stack, operators, and downstream systems as well. Otherwise there may be a route nobody in the review has examined.
For the map, I'd follow a real kind of job from its starting goal to its effect. Identify the user or service starting it and the agent identity doing it. For each context source, peer agent, and tool call, record the calling identity, target, and returned result. Where are the persistent copies? Add the people who can approve or stop that work. A reviewer also needs to see the trust and tenant boundaries and the points where untrusted content can enter.
For the risk logic, I'm using NIST SP 800-30 Rev 1: what threats and vulnerabilities are involved, how likely are they, and what would the impact be? We need those answers to choose a response. NIST AI RMF 1.0 extends the discussion into governance, context mapping, measurement, and continuous management. I wouldn't go to either looking for a universal agent score. The organization has to decide what boundary it can defend and explain why it used it.
What are the risks of AI agents?
An agent may retrieve another customer's records, act on instructions hidden in a document, or use a tool with more authority than the user requesting the work. Persistent memory can carry harmful instructions into a later task. A retry can repeat a change that already succeeded.
The consequences depend on the job and its permissions. Include sensitive information assembled from several sources, actions taken without meaningful approval, and failures that nobody can detect or repair. The evaluation checks below cover normal work, prohibited requests, and broken controls; the recovery review addresses what happens after a failure.
What the public research tells us about testing
I would be cautious about approving an agent on the strength of one clean test. Look at what NIST CAISI reported in March 2026: more than 400 participants made more than 250,000 attack attempts against 13 frontier models. They found at least one successful attack against every target model. That's a reason to examine the testing, not a simple verdict that tells you which model is safe. NIST noted differences between models, uneven transfer of attacks, and the need for evaluation to keep adapting.
The earlier NIST analysis of agent hijacking evaluations makes the practical problem clearer. What happened on the task that matters to you? An aggregate success rate can hide that task's consequences. How often can somebody try? A result based on one attempt may look different when attempts are repeated. If our deployed agent can retry, that is the version of its behavior I'd want tested.
Commenters summarized in NIST AI 800-5 widely agreed that agents present novel security threats and that familiar cyber practices need adapting. For scenario ideas, the OWASP Top 10 for Agentic Applications includes goal hijack and tool misuse, identity and privilege abuse, memory poisoning, insecure communication, and cascading failure. Which of those paths exists in our system? We still have to answer that locally; the taxonomy doesn't mean every system faces equal risk.
The GS Agent Risk Assessment Priority Index
Where should the assessment team spend its time first? We built the GS Agent Risk Assessment Priority Index to help with that question. In this derived model, cascading failure, delegated authority abuse, supply chain compromise, and recovery failure come out at the top. I would use that result to decide where to investigate, rather than to predict an incident.
The inputs are ten plausible scenarios, each rated from one to five on six ordinal factors. We gave threat exposure 20 percent of the weight and authority consequence another 20 percent. Data impact, control gap, detection difficulty, and recovery burden each receive 15 percent. We report the weighted result on a zero to 100 planning scale.
Here are the modeled scores, so the order can be checked. Cascading multi agent failure is 89 and credential or delegated authority abuse is 87. Supply chain or tool compromise is 86, as is pause, rollback, or recovery failure. Agent hijack from external content is 84. Sensitive data disclosure comes to 83; audit and attribution gap, 82. The remaining results are 79 for memory or context poisoning, 78 for excessive tool action, and 77 for approval bypass or misplaced trust.
What would change those numbers for your organization? I'd start with the architecture and controls somebody has actually tested, then bring in incident history and the risk owners' evidence. Replace the GS ratings with those local facts. A high result means that the exposure, authority, data, gaps, detection, and recovery deserve earlier treatment and stronger evidence. It doesn't tell you that the scenario will occur.
We tried shifting weight toward authority, data, and recovery. Every scenario stayed within two points of its original result, and the highest priorities stayed high. That gives some support to the sequence under that sensitivity test. It doesn't establish a set of weights everybody should use.
Keep the status of the model clear when sharing it. This is a GS Consulting derived planning tool. It isn't an official NIST, NSA, OWASP, legal, audit, compliance, certification, or regulatory determination. Nor do the public sources give us a representative benchmark for the modeled scenarios.
What evidence supports the six ratings?
For threat exposure, I'd start with who gets to put material in front of the agent. A direct user is one route. So are email, a website, a document or code repository, retrieved content, another agent, or a tool response. Poisoned memory and a compromised supplier belong in that review too. How often can they try? Put the path and test case beside the rating so somebody can see what the prompt-injection label means here.
Then look at authority consequence. With these credentials, could the agent read, create, change, send, approve, or execute something? Could it purchase, grant or revoke access, disable, or delete? I want to see delegated permissions and credentials acquired during the task as well. Show what is possible before explaining how the controls narrow it in operation.
Data impact takes more work than reading the source labels. Follow source records into the combined context and derived output. Include messages, traces, memory, artifacts, logs, backups, and support records. What does the assembled answer reveal? Joining individually routine records may make the answer sensitive even when no single label warned the reviewer about it.
For control gap, go through what is supposed to protect that path. Does it prevent unsafe input and enforce identity and permissions? Does it constrain the tools and gate consequential actions? Check memory isolation, message validation, limits on propagation, and the way to stop the system. I would ask to see a current test before giving a designed safeguard the same credit as one shown to work.
Detection difficulty is partly a question for the person who will investigate. Can they find the goal and context, identity and policy decision, plan, tool call, parameters, target, action, and result? Can they join peer messages, approval, and exceptions to that same work? With different identifiers in every component, that person has to guess how the records fit.
Finally, how much work would recovery take? Include pausing jobs, revoking identities and credentials, and containing affected agents and tools. Someone may have to roll back actions, repair data, notify owners, reconcile state, retest, and authorize a safe return. I'd estimate that burden when scoring the initial risk. Once the agent has acted, the team may have a much harder job reversing it than they would have had preventing it.
How to do an AI risk assessment?
I'd take the proposed job into the assessment with its owner and the people building it. The five steps below give them a way to work through what they're willing to allow. Keep the evidence, because a later change may mean the approval has to come back.
1. I'd begin by getting the owner to describe what success would look like in a test. Record that business outcome with the owner, users, affected people, boundary, data, tools, and environments. Which actions are allowed and prohibited? Put the success measures and stop conditions in writing too. If a simpler rules based workflow would do the job, why are we proposing an agent?
2. Then I'd ask someone to show each connection. For the agent, service, user, tool, and target, explain the identity and permissions, delegation, audience, scope, duration, and revocation. I also want to know which identity the downstream record will show. Trace input through memory and action on the data map, and include the logs, artifacts, backups, and deletion. That lets the reviewers see where the authority and data meet.
3. Try the failures the system could encounter. Include malicious external content and ambiguous goals, unauthorized tool requests and stale credentials. What about excessive retries, a crafted peer message, poisoned memory, the wrong tenant, or an approval rejection? Add provider failure, gaps in monitoring, and recovery while the system is under load. Those tests should describe real tasks this system might face.
4. Work through what the results mean. Each scenario's rating needs a written basis, current controls, and test evidence. Which controls are missing? Who is treating each gap, and by when? Record any residual risk the owner accepts. I'd keep a severe failure visible even if several less consequential tasks went well.
5. At the end of the review, the accountable owner has to say what they're prepared to allow. They may approve, limit, defer, or reject the use case. Write the launch conditions and monitoring thresholds, stop rights and exception handling, change triggers, review cadence, and retirement criteria with their decision. If the material assumptions change, that approval has expired and the owner needs to look again.
Follow the authority for each action
I'd take an action table into the review. Put the initiating user or service beside the agent identity and delegated authority, then the tool and target resource. Include allowed parameters, data classes, approval rule, rate limit, evidence, revocation and rollback methods, and owner. Now the reviewer can ask why a particular action has broad credentials or an external effect nobody had discussed.
Separate identities can help where you need separate attribution or selective revocation. With one shared service credential, ten agents can look like one actor. During an incident, disabling just one of them becomes harder. The February 2026 NIST NCCoE concept paper is preliminary, but asks useful questions about proving authority, conveying intent, delegation, binding human approval, and verifiable action records.
The agent's ability to reach a source doesn't mean the user may see it. Retrieval, tools, memory, and downstream services can all extend the access path. Check current authority at retrieval and again at action time. Where the architecture permits, bind tokens to the intended resource and task, and expire them soon enough that a finished job doesn't leave continuing access behind.
The reviewer needs the actual goal and source, affected resource, proposed change, consequence, uncertainty, and safer alternative. They must be approving the exact action. If the agent can change the target or parameters afterward, that approval isn't controlling the action they reviewed.
How do I evaluate my AI agent?
Ask the agent to do the intended job first. I want to see it finish with the right sources cited or recorded and the right identity used. Did it stay inside the data and tenant boundaries? Did approval happen at the right point, was the result verified, and did the job stop cleanly? Without that normal path working reliably, we may spend the security review chasing unrelated failures.
Now try to get it to do something it shouldn't. Ask it to use an unauthorized tool or reach the wrong tenant. Try an amount above the permitted limit, expired access, and another attempt at a denied request. Give it instructions hidden in external content or a false peer message. Try to keep prohibited data in memory and get around approval. I care about what the downstream system actually denied, even if the model was prepared to do it.
If somebody can try a task thousands of times, why would we draw a conclusion from one attempt? Agree on attempt counts and acceptable thresholds. Change the wording and context, and the model temperature where that is relevant. Keep the exact prompts and inputs, environment, model and framework versions, controls, outputs, actions, and results. Otherwise the next person won't be able to repeat what we're discussing.
I'd then ask how we behave when a control breaks. With authorization for the test, take away the policy service. Delay revocation or remove a log source. Return malformed tool data, fill a queue, break a callback, and simulate a provider outage. Does the system deny work, pause, move to a safe mode, or continue with reduced visibility? The owner needs to know which one happened.
The production follow-through is in Continuous monitoring for AI agents. For the investigation record, AI audit trails and activity logging covers how the event chain needs to stay together.
Exercise recovery before increasing access
An email the agent has already sent won't come back just because we stop it sending another. I'd use that kind of consequence when working out the recovery plan. Separate the reversible actions from effects that need a compensating action and effects we can't reverse. An exposed record, purchased item, disabled account, published code change, or external decision may each leave a different repair job. Consequential, irreversible work needs strict gates before it happens.
In the recovery exercise, someone has to declare the failure and say which goal and task were affected. Identify the agents, credentials, tools, data, actions, users, and time window. Then pause new work and revoke or narrow the authority involved. Keep the relevant evidence without unnecessarily copying sensitive content. Peers and downstream jobs that received the same message, state, or artifact need containing too.
Who will authorize a return? I would ask them to see the cause, the repaired control, reconciled state, and the defined tests rerun. Check connected agents and tools as well, and record the approval to resume. Retrying the task quietly with the same context and authority wouldn't address why we stopped it.
The joint NSA and international partner guidance on careful agent adoption recommends incremental deployment and continuous assessment, containment and rollback, rigorous monitoring, and human oversight. I think that is a useful operating standard. Its recommendations don't automatically create a specific legal obligation for a given organization.
Six shortcuts that leave the assessment incomplete
A completed asset entry doesn't necessarily tell the reviewer what the agent could do. The model name, vendor, owner, and data label may all be there while credentials, tool authority, peer agents, approvals, external effects, monitoring, and recovery remain unexamined. I'd follow those paths before treating the inventory as an assessment.
I'd also ask what a portfolio average has hidden. One rarely used task could have a severe consequence. Keep scenarios at the action level when their consequences differ so that task is still visible.
What did the person clicking approve actually get to decide? They need context, time, authority, and a safe way to reject or correct the action. Without those things, the click can make the action appear controlled without giving the reviewer meaningful control.
Look at the output after the agent joins the data. Routine facts may become a sensitive inference or a more complete record. Include that assembled answer and every persistent copy in the assessment.
And if the task can recur, I'd test repeated attempts in varied contexts. Agents are probabilistic and attackers can try again. We need to understand more than the result of one run.
Finally, try recovery. Someone needs to show that credentials revoke, peer jobs halt, actions reconcile, data restores, and safe operation can resume. A stop button in a document doesn't establish those outcomes.
Keep the evidence connected to the decision
Could a new reviewer follow this assessment without asking everyone to reconstruct the project history? I'd put the use case boundary at the front. For the authority, show the identities and permissions, scopes and delegation, audience, expiry, and revocation. For the data, show the source records and sensitivity, joins and outputs, memory, logs, retention, and deletion. Keep the threats' paths, consequences, and assumptions with those maps.
For a control test, the next person will need its method, cases, versions, results, thresholds, limits, and owner. For the decision, they need treatment and accepted residual risk, the approver, date, and review trigger. Can they trace the goal and context through policy, action, result, actor, and exception in the operating events? The recovery exercise should show how pause, revoke, rollback, repair, reconcile, and resume worked.
I'd store it where the people running the workflow can update it. If a presentation explains the decision, have it link to the current technical and operating records. A screenshot can support a point. It can't stand in for the machine records, tested configuration, interviews, and repeatable procedures that the reviewer needs to check it.
AI agent lifecycle management connects that evidence to ownership, review, change, and retirement. For the approval itself, Human in the loop workflow architecture discusses how the reviewer can make a consequential decision at that point.
Before the pilot becomes production
Suppose the team gets to launch week and still hasn't agreed on how much access the agent may have. I'd rather settle that while designing the workflow. Name the approver and record the maximum authority, data boundary, required tests, monitoring coverage, acceptable residual risk, and recovery objective before people are counting on a release date.
A narrow, read only job with little consequence may be ready with good source controls, evaluation, observation, and an easy stop. Changing records, granting access, executing code, moving money, communicating outside, directing peers, or touching regulated data calls for stronger identity, approval, isolation, tests, evidence, and recovery.
Have the team demonstrate the proposed expansion before granting it. I want to see who authorized the agent and what it could reach, then the reason that particular action was allowed. Show the result and the recovery as well. I would hold the additional access if the team couldn't show me those things.
Research sources and method
The source list below is where I'd go to check the public observations. It includes NIST, NIST CAISI, NIST NCCoE, the joint NSA partner guidance, and OWASP. The scenarios, ratings, weights, and derived scores are GS choices. They shouldn't be attributed to those organizations. The complete source register and data dictionary are in the article research package. You'll also find the model inputs and formulas there, along with the sensitivity analysis, figure data, and workbook if you want to check how the results were calculated.
- NIST SP 800-30 Rev 1: Guide for Conducting Risk Assessments
- NIST AI Risk Management Framework 1.0
- NIST Generative Artificial Intelligence Profile
- NIST CAISI agent hijacking evaluation research
- NIST CAISI agent security competition analysis
- NIST AI 800-5 security considerations response summary
- NIST NCCoE agent identity and authorization concept paper
- NSA and international partner agent adoption guidance
- OWASP Top 10 for Agentic Applications for 2026
AI agent risk assessment FAQ
How is AI agent risk different from ordinary AI risk?
Look at what follows the answer. An agent may call a tool, keep state, delegate, and continue taking steps toward a goal. Those steps can affect other systems. I'd follow its action authority and delegated credentials through the work, then look at repeated attempts, state, propagation, revocation, rollback, and accountability. Model quality and output risk remain part of the review too.
What should an AI agent risk assessment include?
Could another person follow the approval decision? Give them the use case boundary, owner, users, and agent identities, then the data and permission maps, including delegation. Keep the threat scenarios with the control tests and human decisions. They also need event records, the accepted residual risk, monitoring thresholds, stop rights, and a recovery path someone has exercised.
How should AI agent risk be scored?
I'd score one scenario at a time and put the assumptions beside the ratings. This guide uses threat exposure, authority consequence, data impact, control gap, detection difficulty, and recovery burden. Change the weights and see whether the priorities move. That helps plan the work. It doesn't turn the result into a universal benchmark or a compliance grade.
Does human approval make an AI agent low risk?
How much could you decide if the approval screen didn't even show which account would change? The reviewer needs the goal and source, proposed action and target, consequences, uncertainty, and safer alternatives. Give that person a way to refuse or correct the action. I wouldn't give the approval click much weight if the reviewer couldn't do those things.
How often should an AI agent risk assessment be updated?
Revisit the assessment when an assumption behind approval changes. That could be the goal, model, prompt, tools, permissions, data, memory, provider, or protocol. It could also be the approval flow, monitoring, recovery design, or operating environment. Incidents, near misses, and new threat findings need a review too. I'd set a recurring cadence based on the consequences, so the next assessment doesn't depend on someone reporting a change.
Related reading
- Agentic AI Resource Hub
- Multi Agent AI Security
- Securing AI Agents
- AI Agent Lifecycle Management
- Continuous Monitoring for AI Agents
- AI Audit Trails and Activity Logging
- Secure Enterprise AI Strategy
Discuss an agent risk assessment
If you're considering a pilot or an increase in an agent's authority, GS Consulting can help define the use case, test the failure paths, and examine the controls and recovery evidence. Bring those findings to the owner who will make the risk decision.
Request an Agent Risk Assessment