AI and LLM Penetration Testing: From Prompt Injection to Data Leakage
An AI penetration test covers the LLM application, its prompts, tools, data sources and the surrounding web and API layer; the report follows within 5 business days.
What is tested and why it matters
Validating the security of LLM applications, agents and their integrations.
An LLM application does not fail like a normal web application. Its input is natural language, so the usual input validation does not apply; an attacker instructs the model instead of exploiting a parser. Prompt injection through a document, an e-mail or a web page can make the model leak its system prompt, reveal data from the retrieval store, call a tool with the wrong parameters or send information to an external address. Excessive agency, where the model may act without a human check, turns a language flaw into a business action. At the same time, the application around the model still has authentication, authorisation, APIs and sessions that fail in the usual ways. A penetration test covers both layers and shows which combinations lead to real impact before customers or regulators find them.
Who needs this assessment
Organisations that benefit most.
This assessment is designed for organisations that expose an LLM-based feature to customers or employees, or that let an agent act on internal systems, and need evidence that it cannot be turned against them.
SaaS providers that added a chatbot, copilot or AI assistant to their product
Organisations running an internal assistant on company documents through retrieval-augmented generation (RAG)
Teams deploying agents that can call tools: send e-mail, query databases, create tickets or trigger workflows
Customer service, banking, insurance and healthcare organisations where the model handles personal or financial data
Development teams preparing an AI feature for production and seeking independent validation before launch
Organisations that must show due diligence for AI systems to customers, auditors or under the EU AI Act
Where these assessments are relevant
- Financial services: customer portals, payment processing, and cloud-hosted transaction infrastructure.
- SaaS and technology: application validation before production deployment or enterprise procurement.
- Healthcare: digital systems handling patient data, clinical records, and connected medical devices.
- Retail and e-commerce: web applications processing transactions, customer accounts, and payment data.
What the assessment produces
The insight and evidence delivered at the conclusion of the assessment.
Every assessment concludes with a structured report covering technical findings, severity ratings, evidence, and remediation guidance mapped to the OWASP Top 10 for LLM Applications. Reports are delivered through the Pentrox Portal and are available for download at any time.
- Independent validation of the LLM application against the OWASP Top 10 for LLM Applications and the classic OWASP Top 10
- Proof of which prompt injections succeed, through which channel, and what they leak or trigger
- A map of tool and data access: what the model can reach, with which rights, and where a human check is missing
- Concrete fixes in the prompts, the retrieval layer, the tool permissions and the surrounding application
- Portal access with real-time findings, retest workflow, and report downloads for the lifetime of the client account
How an assessment runs
- Scope and Intake
- Planning
- Assessment
- Reporting
- Findings & Retest
Retesting of remediated findings is included as standard with every assessment.
Assessment methodology
Five phases from architecture mapping through verified remediation.
Each phase builds on the previous one. Architecture mapping defines the trust boundaries, prompt and data testing probes the model layer, tool testing follows the actions the model can take, and the application layer is tested like any web application.
Architecture and trust boundary mapping
The components are mapped: model and provider, system prompt, retrieval sources, tools and function calls, memory, and the web or API layer. Each boundary where untrusted content meets the model is recorded as a test target.
OWASP LLM Top 10, NIST AI RMF, MITRE ATLASPrompt injection and data leakage
Direct and indirect prompt injection are tested through every channel: user input, uploaded documents, retrieved pages, e-mails and tool outputs. The goal is system prompt leakage, disclosure of retrieval data and instructions that cross the trust boundary.
OWASP LLM01, LLM02, LLM07, LLM08Tools, agency and output handling
Every tool the model may call is tested for abuse: wrong parameters, other users' data, actions without confirmation and privilege escalation through chained calls. Model output is followed into the application to find injection through improper output handling.
OWASP LLM05, LLM06, OWASP ASVSApplication, API and abuse controls
The web and API layer around the model is tested for authentication, authorisation, session handling and injection. Rate limits and cost controls are checked for unbounded consumption; the supply chain of models, plug-ins and libraries is reviewed.
OWASP Top 10, OWASP WSTG, OWASP LLM03, LLM10Reporting with OWASP LLM mapping
Findings are documented with the exact prompts, evidence, risk classification and a mapping to the OWASP LLM category. Critical findings are communicated immediately during testing.
OWASP LLM Top 10, OWASP Risk Rating Methodology (severity rating)Scope and deliverables
What is included, what is excluded, and what is delivered.
The statement of work defines the exact application, model integrations and testing approach before the assessment begins. The lists below reflect the standard scope for an AI or LLM penetration test.
In scope
- Direct and indirect prompt injection through user input, documents, retrieved content and tool outputs
- System prompt leakage and disclosure of sensitive data from the retrieval (RAG) store or the training context
- Tool and function-call abuse, excessive agency and missing human confirmation on actions
- Improper output handling: injection into the application through model output (XSS, SQL, command)
- Authorisation between users and tenants: whether the model or its tools reach data of other users
- Rate limiting, cost controls and denial-of-wallet through unbounded consumption
- The web application and API layer around the model, including authentication and session handling
- Supply chain review of models, plug-ins, vector databases and libraries in the application
Out of scope
- Training or fine-tuning a model, and formal evaluation of model accuracy or bias
- Security of the model provider's own infrastructure (OpenAI, Anthropic, Azure OpenAI, Google and others)
- Denial-of-service testing against production systems unless explicitly agreed
- Social engineering of the client organisation
What you receive
- Executive summary Risk posture of the AI feature, severity distribution and remediation priorities written for product owners and leadership.
- Detailed findings Each vulnerability documented with the exact prompts and inputs, evidence, reproduction steps, severity rating and OWASP LLM category.
- Tool and data access map What the model can reach, with which rights, through which tools, and where a confirmation step is missing.
- Remediation plan Prioritised by exploitability and business impact, with fixes in the prompts, the architecture and the application code.
- Portal access Real-time findings, report downloads, retest workflow, and team management through a dedicated platform.
- Findings meeting A scheduled session to align on risks, priorities, and the remediation approach.
- Retest report Verification results after remediation, documenting each finding with its updated status.
Read more
AI versus human in penetration testing
Where automation helps a tester and where judgement still decides the result.
API security testing beyond the OWASP Top 10
Authorisation, rate limits and business logic in the APIs that models and agents call.
What a penetration test costs
Indicative prices by service, what moves the number, and what the price includes.
When independent assessment is the right step
An independent assessment is required ahead of an audit, a product launch, or a funding round, and the organisation needs documented evidence from a third party.
Digital platforms handle revenue, customer data, or service delivery, and the security of those platforms has not been validated by an external party.
Existing security controls have been implemented but have not been tested from an attacker perspective to confirm they work as intended.
Timeline
Illustrative durations to support planning. Confirmed in the statement of work.
The duration of an AI or LLM penetration test depends on the number of tools and data sources the model can reach, the number of user roles and tenants, and the size of the application around the model. The calibration bands below are indicative. Exact duration is confirmed in the statement of work.
What determines duration
- Number of tools and function calls the model may use
- Number of retrieval sources and the sensitivity of the data in them
- Number of user roles and tenants sharing the same model
- Size of the web application and API layer around the model
- Whether an agent can act without a human confirmation step
Actual duration is confirmed during scoping and depends on the number of integrations and the depth of testing on the application layer. A statement of work confirms the exact scope and timeline before the assessment begins. The final report is delivered within five business days after completion of testing.
Why Pentrox
The operational standards behind every AI assessment.
These operational standards apply to every AI and LLM assessment Pentrox delivers, regardless of the model provider or the size of the application.
OWASP LLM Top 10 mapping
Findings tied to the LLM risk categories.
Every finding references the OWASP Top 10 for LLM Applications category it relates to, next to the classic OWASP category where the application layer is involved. The report can be used directly in a security review or a customer questionnaire.
Manual adversarial testing
Prompts written by testers, not only a scanner.
Automated prompt suites find the known patterns. Senior testers write the prompts that fit your application, chain injection with tool calls and follow the output into the application, which is where the real impact lives.
Both layers in one test
The model layer and the application around it.
A leak through the model is often only exploitable because the application trusts the model output or lacks authorisation on a tool. Testing both layers in one assessment shows the complete attack path.
Portal-delivered results
Findings visible in real time, not buried in a PDF delivered weeks later.
The Pentrox Portal provides remediation tracking, retest requests, report downloads, and team access management. Role-based access control is enforced on every account.
Retest included
Verification that fixes are effective, at no additional cost.
Remediated findings are flagged through the Portal and scheduled for verification. Each finding is re-evaluated individually. Retest cycles repeat until all findings are resolved.
Frequently asked questions
Practical answers for product owners, security teams and procurement.
What is an AI or LLM penetration test?
A penetration test of an application that uses a large language model: a chatbot, a copilot, an internal assistant or an agent. Senior testers try to make the model do what it should not: leak its instructions or data, call tools with the wrong parameters, act on behalf of another user or inject content into the application. The test also covers the normal web and API layer around the model. The result is a report with the exact prompts, evidence and fixes.
What is prompt injection?
Prompt injection is input that makes the model follow the attacker's instructions instead of the developer's. Direct injection comes from the user in the chat. Indirect injection hides in content the model reads: a document, an e-mail, a web page or the output of a tool. Indirect injection is the more dangerous form, because the victim never types anything; opening a document is enough. It is number one in the OWASP Top 10 for LLM Applications.
Which OWASP list applies to AI applications?
The OWASP Top 10 for LLM Applications, 2025 edition: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. The classic OWASP Top 10 still applies to the application around the model. The report maps each finding to both where relevant.
Do you test the model itself or the application?
The application, including how it uses the model. Training, fine-tuning and bias evaluation are out of scope, and so is the infrastructure of the model provider. What is in scope is everything you control: the system prompt, the retrieval sources, the tools, the permissions, the output handling and the web or API layer. That is also where the fixes are.
Is retesting included, and at what cost?
Retesting of remediated findings is included as standard with every assessment, within 90 days after report delivery. There are no additional charges for verifying that fixes have been implemented correctly. The retest is managed through the Pentrox Portal, where you flag remediated findings and request the retest.
How is production impact managed?
All testing is performed in a controlled, non-destructive manner. Pentrox recommends testing in a non-production environment with the same model configuration where possible. Where production testing is required, testing windows, excluded tools and actions, spending limits on model usage and escalation contacts are agreed in the statement of work before the assessment begins.
Is the report delivered in Dutch or English?
Reports are delivered in English by default. Dutch-language reporting can be arranged when required. The statement of work confirms the reporting language before the assessment begins.
What happens when a critical vulnerability is found during testing?
Critical or blocking vulnerabilities are communicated immediately through the channel agreed at kickoff. Findings are published in the Pentrox Portal as they are confirmed, so remediation can begin before the final report is delivered. The escalation path is documented in the statement of work.
Does the EU AI Act require a penetration test?
The AI Act sets risk-based obligations, with the strictest ones for high-risk systems, including requirements for accuracy, robustness and cybersecurity. It does not name a penetration test. Like ISO 27001 and NIS2, it asks for evidence that the system is resilient against attempts to manipulate it; a penetration test report with findings and retest results is that evidence. For most organisations the customer questionnaire arrives before the regulator does.
What do we need to provide before the test?
Test accounts per user role, a description of the tools and data sources the model can reach, the system prompt if a white-box test is agreed, and a test environment or a spending limit for the production model. Materials are due three business days before the start; a working environment on the first morning keeps the testing days in testing.
Related assessment categories
Web application and API testing
The application layer around the model: authentication, authorisation, APIs and business logic.
Source code review
Reviewing the prompt handling, tool wiring and output handling in the code itself.
Cloud security review
The cloud configuration behind the model APIs, vector databases and keys.