AI integrations security training for developers

Part 3 of the SecureFlag AI training series

When an application uses AI to process customer messages or perform tasks, it also needs to account for the possibility that someone might try to manipulate what the model does. These risks grow when the AI feature accesses customer data or interacts with other systems. 

This post looks at the main risks of integrating AI into an application and the skills developers need to address them.

A user’s message reaches an AI feature that uses tools connected by MCP and reads customer records, but private data can leak out.

Who this training is for

Training is aimed at developers who integrate large language models (LLMs), AI agents, and other AI services into applications delivered to customers. It could involve adding a support assistant to an existing product or developing an agent that carries out tasks on a user’s behalf.

This is different from Part 2 of the series, which focuses on developers who use AI to help write code. Here, AI is integrated into the product itself, where it processes customer input and influences how the application behaves.

What changes when AI becomes part of a product

One of the biggest differences is how an application handles the information it receives. Traditional applications generally follow logic defined by their developers, whereas AI models interpret text and generate responses based on the content they receive, such as directly from a customer or an uploaded document.

This creates opportunities for attackers to influence how the application behaves, including through techniques that traditional application security testing may not cover. An AI feature can behave correctly during normal testing but still be vulnerable to carefully crafted input. 

Developers therefore need to understand the security principles that apply to traditional applications alongside the additional risks introduced by AI.

What developers learn

Training covers four topics, each focusing on a different part of an AI feature.

LLM risks and prompt injection

Attackers can use prompt injection to influence how an AI model behaves. They might include malicious instructions in their own messages or hide them in content the application retrieves or processes, such as a support ticket or web page. 

In agentic applications, this can lead to Agent Goal Hijack, a risk identified in the OWASP Top 10 for Agentic Applications, where an attacker redirects an agent away from its intended goal. Attackers may also try to make the model reveal its system instructions or disregard them.

If attacks are successful, they could make the application reveal sensitive information or perform actions the user isn’t authorized to request. Developers need to learn how untrusted content ends up in a prompt and why filtering input alone cannot prevent every attack.

Data leakage and safe defaults

Another concern is how much information an AI feature can access and where that information goes. Models often need application data to answer questions or complete tasks, which could involve customer records and internal documents.

Depending on how the integration is configured, some of that data could also appear in logs or be sent to an external model provider. Securing sensitive information starts with limiting what the model receives to what it needs for the task.

Developers should make sure secrets such as API keys aren’t included in prompts, that one user’s context isn’t exposed to another user’s session, and that logging and data retention settings don’t create additional opportunities for sensitive information to leak.

Securing AI integrations

As AI features become more capable, they often need to connect to other parts of an application to retrieve information or perform tasks. These connections introduce additional security risks, especially when data and instructions pass between components with different access levels.

Retrieval-augmented generation (RAG)

A common way to give an AI feature access to relevant information is through RAG, which retrieves documents before the model generates an answer. 

However, those documents aren’t necessarily trustworthy, and an attacker could add instructions to content that the model could then follow. Developers should make sure only authorized users can add or modify documents in the knowledge base.

It’s also important to limit who can access the retrieved information. If a user asks about another customer’s account, the application needs to make sure the retrieval process respects access permissions.

Both the content being retrieved and the process that determines which users can see it need to be secured. 

Tool calling

Things get more complicated when a model can use connected tools to interact with other systems. Aside from returning information, it could generate a request to retrieve a customer record or update an account.

Developers need to confirm that each request is valid and that the user is allowed to make it. Tool calls can also introduce vulnerabilities such as command injection and requests to unauthorized internal resources, so input validation and access controls are essential. 

Depending on the operation, additional safeguards such as human approval may also be necessary.

MCP connections

The Model Context Protocol (MCP) provides a standard way for AI models to connect to external tools and data sources, making it easier to extend what an AI feature can do. However, those connections also introduce another source of potential manipulation.

A malicious or compromised server can change how the model behaves, and tool descriptions or results can carry hidden instructions aimed at the model. Developers need to assess the security and trustworthiness of third-party servers and dependencies before connecting them to an application. 

The same applies to the models and packages an AI feature relies on, since they could be tampered with before they reach the application.

Model output

While much of the focus tends to be on the information going into a model and the systems it can interact with, the output it generates needs just as much attention. Once that output is passed to another part of an application, it should be handled like any other untrusted input.

Showing it on a web page without proper safeguards can lead to cross-site scripting (XSS), and putting it straight into a database query can allow SQL injection.

These risks apply wherever model output is used, so developers should never assume it’s safe just because it came from an AI service.

Permissions for AI components

Even if an attacker succeeds in misleading an AI model, the damage should be limited by what it is allowed to access and do. It’s why AI components need to follow the principle of least privilege.

For example, an AI feature that retrieves information may only need read-only database access. When an agent performs an action on behalf of a user, it should use that user’s permissions rather than a shared account with broader access.

Agents should also be restricted to the tools they need, rather than being given open-ended capabilities such as running arbitrary shell commands. Rate limits and monitoring of model and tool activity also help prevent excessive usage and detect unwanted actions.

What teams get from it

With a better understanding of these risks, developers can make more informed decisions about how AI features interact with application data, the systems they can access, and what actions they can perform. They can also identify security issues that might not show up during ordinary functional testing, helping teams build AI functionality with security in mind from the outset.

A SecureFlag MCP lab example

AI integrations security training with SecureFlag

SecureFlag supports teams at each stage of building an AI feature. During design, ThreatCanvas offers risk templates for the OWASP Top 10 for LLMs and the OWASP Top 10 for Agentic Applications, so teams can identify likely problems before any code is written.

Once development starts, hands-on labs for the OWASP Top 10 for LLMs, agentic AI, MCP, and skills let developers find and fix weaknesses in realistic AI features. Agentic Coding Labs add practice with integrations, and learning paths on prompt injection and secure RAG development cover those topics in more detail.

Book a demo to see SecureFlag in action.

Continue reading