What does penetration testing for LLM products include?

When an LLM is integrated into a product, it is necessary to test not only the external shell. It is important to check how the request passes through the application, model, internal services, and third-party integrations.
Specialists analyze:

Scenarios in which the model has access to confidential data require separate attention. This refers to the risk of Sensitive Information Disclosure from OWASP Top 10 for LLM Applications. In such cases, the model can accidentally disclose personal data, financial information, commercial information or other sensitive data.
For the company, this is not only a technical problem. If this risk is not controlled, the company can lose customer trust, violate internal security rules, and face issues during an audit or regulatory inspection.
Penetration testing of LLM solution integration assesses not only the vulnerability but also its impact on business processes, customer data and the company’s reputation.
Testing also covers scenarios where the LLM response is automatically passed to other system components. For example, the model generates an SQL query, command, workflow decision, or API instruction. If the output is not checked, the model’s response can turn into a really dangerous action.

What risks arise for companies?

The main risk is loss of control over the agent’s actions. If the AI component can create requests, change data, work with files, launch processes, or transfer information to external systems, its compromise can have negative business consequences.
For fintech, e-commerce, banking services, and SaaS products, this is especially critical. An incorrectly configured agent can disclose customers’ personal data, act without proper authorization, bypass business rules, or transfer information about the internal infrastructure to an attacker.
If the system does not control and limit the model’s output data, the LLM response can lead to a really dangerous action: data leakage, unauthorized operation, service failure or violation of security requirements.
In the worst-case scenario, this can end in complete system compromise:

A separate aspect is audit and compliance checking.
An AI agent is subject to audit if it works with corporate or customer data, has access to internal systems or can perform actions in business processes.
To assess the agent’s security, it is not enough to describe its general role. The customer must provide the auditor with information about the agent’s role, data sources, access levels, connected systems, permitted actions, established restrictions, logging and the results of testing risk scenarios.
Without this, it is more difficult to confirm that the company controls risks related to data, access and autonomous actions of the LLM.

Where to start the check?

The first step is to describe the architecture of the AI component. It is necessary to determine where the LLM is used, what data sources it sees, the tools that can be called, the roles of users who interact with the agent, and the actions performed by the system automatically.
The second step is to check the agent’s rights. Often, the AI component receives excessive access “for development convenience.” At the launch stage, this seems like a quick solution, but in a production environment, it creates unnecessary risk. The agent must have only those permissions that are needed for a specific scenario.
The third step is to contact specialists who conduct penetration testing of LLM solutions according to OWASP Top 10 for LLM Applications. They check not only the web interface and API, but also specific risks for systems with integrated AI: prompt injection, leakage of sensitive data, context isolation, connected APIs, processing of the model’s response and restrictions on the agent’s autonomous actions.
Logging settings should also be checked separately. The security team must see what requests the agent receives, what tools it calls, what decisions the system makes and what actions are performed after the model’s response. Without this, incident investigation can become almost impossible.