AI pentesting has quickly become a crowded category as the AI offensive-security boom brings more tools and approaches to market. Some products automate familiar scanner workflows, some put a language model in front of existing tools, and some are building systems that can plan and execute extended testing tasks. The claims can sound similar even when the products, their intended use cases, and their operational controls are very different.
Frontier models are increasingly able to understand code, call tools, and suggest credible next steps. As a frontier model by itself falls short of a pentest, security leaders still need to establish whether the surrounding system can turn that capability into an authorized, repeatable security test that produces evidence an engineering team can act on.
The evaluation should therefore focus on what the system is allowed to do, how it makes decisions, and what it can demonstrate after a run.
Start with the environment you need to test
Buyers should ask a vendor to describe the environment it can actually assess, including whether it operates from an outside-in perspective, supports authenticated application testing, or expects access to source code and internal systems. A black-box assessment of an internet-facing application answers a different question from a white-box review of source code or an internal network assessment. Broad claims about autonomous testing mean little until the vendor sets these coverage boundaries and identifies where a manual specialist or another security control remains the appropriate choice.
Ask how a lead becomes a verified finding
The most useful test results give engineers enough evidence to understand what happened, reproduce the issue safely, and decide how to fix it. That standard matters because a successful HTTP response can be a login page, a generic application shell, a cached error, or the expected content from the account that made the request.
In an agentic pentesting system, the model may recognize a promising response and suggest a follow-up. The surrounding system has to decide what evidence would prove the hypothesis, run a bounded check, and reject the candidate when that proof is absent. Without that discipline, the receiving engineering team inherits the validation work.
Ask to see a finding that was investigated and dismissed, alongside one that was verified. The comparison exposes how the system handles uncertainty and whether the report records the request, response, affected asset, and proof required for remediation. Proof-backed findings are a useful standard because they make the vendor's conclusion inspectable rather than asking the customer to accept it at face value.
Ask to see the controls that enforce scope
Every vendor will say that testing stays in scope. The more revealing question is how that promise is enforced when a model or tool proposes a next action that falls outside the agreed boundary.
Controls should govern the actual request path. They may restrict target hosts, methods, tools, identities, traffic volume, and actions with a potentially irreversible effect. Buyers should be able to see which actions were allowed, which were rejected, why that decision was made, and how the customer can stop a run. This matters when a system investigating cross-tenant access encounters a page where deleting a record is technically possible. A safe test prevents that action before the click takes place.
Use authentication as a technical reality check
Authentication is often where a seemingly convincing testing demo meets the conditions of a real application. Logging in through a browser does not automatically give every tool a usable session, and each application may carry identity through a different combination of cookies, headers, tokens, device checks, or workflow state.
A vendor should be able to explain how its system establishes and refreshes a session, distinguishes one controlled user from another, and protects shared test accounts during multi-agent runs. For an application where authorization is central to the risk, ask the vendor to use two controlled accounts and synthetic data, then show whether one account can cross a boundary into the other.
Find out how the system works when the application changes
The value of a pentest depends on how closely it reflects the system currently in production. Harness’s The State of DevOps Modernization 2026 found that 71% of surveyed enterprise engineers and managers deploy to production at least weekly, while 35% deploy daily or more. In that environment, a report produced months ago can remain useful context, yet it cannot be assumed to describe the current attack surface or the current authorization logic.
Ask what triggers a new test, whether the system can retest a confirmed exposure after a fix, and how it distinguishes a previously verified result from a new outcome after a release. The answer should cover the operational loop between change, testing, evidence, remediation, and verification. That loop is where agentic pentesting can support a continuous security practice, provided the test remains controlled and the reported result remains explainable.
Ask how it uses attack-surface context
Testing begins with an understanding of what exists and what has changed. Continuous attack-surface management can provide the external asset inventory, technologies, exposed services, and known issues that direct an investigation toward the places worth testing. Buyers should ask whether that context is current, how the vendor handles assets with uncertain ownership, and whether the system can preserve what it knew at the time of a particular run. Coverage claims should reflect any incomplete or stale context.
Be precise about the role of people
Autonomy should not obscure accountability. People need to authorize an engagement, decide the acceptable scope, provide appropriate test identities, and determine the priority of a confirmed exposure in the context of customers, contracts, and operational risk. Human offensive-security specialists can also be essential for deep business-logic review. The middle of the process can be agentic when the task is bounded, so ask where people intervene, how a customer can halt or adjust a run, and when a human reviews the result before it reaches an engineering team.
Make the proof of concept an evaluation of the system
A proof of concept should test more than whether a product can identify a known issue in a prepared environment. It should reveal how the system behaves when evidence is ambiguous, authentication is stateful, an action approaches a safety boundary, or a result needs to be retested after a change.
Set controlled objectives with the vendor and agree in advance on what would count as proof. For example, two synthetic tenants with distinct invoices can establish whether the product can identify and verify unauthorized cross-tenant access without exposing real data. Require the vendor to show the run log, the finding evidence, the actions that controls rejected, and the result of a follow-up verification after the issue is addressed.
That approach gives a security team a view of the vendor’s engineering discipline, not only the model’s ability to produce a compelling narrative about a test.
The buyer takeaway
Agentic pentesting can give teams a faster way to investigate changing applications and external exposures. Its value comes from the system that directs the model’s work, limits its actions, validates its conclusions, and preserves a record that security and engineering teams can inspect.
The strongest vendor conversations therefore begin with practical questions. What is the system permitted to do? How are those limits enforced? What evidence does it require before reporting an exposure? Can we see what it tried, what it rejected, and what a retest looks like after remediation?
For a deeper evaluation framework, read What to look for in AI-powered pentesting.
