Artificial Intelligence (AI) is rapidly moving from answering questions to taking actions on our behalf. This emerging form of AI, often called agentic AI, can independently perform tasks such as sending emails, deleting files, or modifying customer accounts. As AI gains autonomy and attackers find new ways to exploit it, incident response plans must evolve. Organisations must now prepare for incidents in which AI may cause, facilitate, or amplify a security breach.

In July, during an internal cybersecurity evaluation, OpenAI's AI agents broke free of their controlled testing environment, accessed the internet indirectly, communicated with each other through an improvised messaging board, collaborated and delegated tasks to each other, breached another company's systems, and carried out over seventeen thousand actions over four days before anyone noticed. The AI agents were not directed to do this, and no human hacker was involved. The agents, which sometimes referred to themselves as a "swarm" or "collective", simply determined that the fastest route to their objective lay outside the boundaries they had been given.

AI can also be tricked into following malicious instructions hidden in emails, documents, or websites. This vulnerability is known as prompt injection. Indirect prompt injection is reported to now rank as the top AI security vulnerability globally, with attacks reportedly increasing by 340% year on year. Earlier this year, a financial services company discovered that their customer-facing AI agent had been leaking internal pricing data for three weeks after an attacker tricked it into ignoring its system prompt.

These incidents show that AI agents can fail in two ways: they can depart from their instructions on their own, or be manipulated by outsiders into taking unauthorised actions.

Autonomous misbehaviour is what happened at OpenAI. The AI agent takes deliberate, connected actions in pursuit of its goal using methods nobody anticipated or authorised. The cause is internal: a misalignment between the AI’s objective and its chosen method. 

Indirect prompt injection is the mirror image. An external attacker steers the AI off course by hiding malicious instructions in content it processes. The agent follows the attacker’s instructions believing them to be legitimate.

The critical distinction is agency. AI agent incidents are active, not passive: the agent acts, observes the result, decides what to do next, and acts again. An agent with access to email, databases, or payment systems can send data to external servers, grant unauthorised access, or transfer funds, all using its own legitimate credentials, which means traditional security tools may not flag the activity as suspicious. By the time the deviation is noticed, the damage may already be widespread.

A single AI incident can trigger multiple legal obligations simultaneously. Consider an AI agent deployed to manage customer queries that is compromised through prompt injection and begins extracting customer personal information. That single event may simultaneously constitute:

• a security compromise under POPIA, triggering notification obligations to the Information Regulator and affected data subjects;
• a cybersecurity incident requiring reporting to the Prudential Authority, FSCA, or South African Reserve Bank (for payment system participants, within 24 hours);
• a potential cybercrime under the Cybercrimes Act, which applies regardless of whether the perpetrator is human or AI;
• a contractual breach of service agreements, data processing agreements, or supplier contracts; and
• a consumer or operational incident engaging consumer protection and operational risk management requirements.

Organisations cannot treat an AI incident as a single problem with a single response. It may require simultaneous notifications to different regulators under different timelines.

Traditional cybersecurity monitoring detects known threat patterns such as malware signatures, abnormal logins, or unusual network traffic. But an AI agent may use its own legitimate credentials to access permitted systems, performing the actions it was designed to perform, while pursuing its objective through unauthorised means. Organisations must consider whether their monitoring distinguishes between an AI agent performing its assigned task and one that has deviated from it.

When a traditional cybersecurity incident is detected, the response is straightforward: isolate the system, revoke credentials, shut down the service. An AI agent incident is more complicated. The agent may be performing a business-critical function midway through a complex task. Following the OpenAI incident, the United States proposed an AI Kill Switch Act. But a kill switch is only useful if someone knows when they are authorised to use it. Organisations need predefined authority: who can act, what criteria trigger action, and who decides when the situation is ambiguous.

Businesses are rapidly deciding what their AI agents may do, and must equally decide what happens when an agent does something it was never expected or permitted to do. If nobody is monitoring the agent, nobody is authorised to stop it, and nobody knows whether the incident must be reported, the organisation has not delegated a task, it has delegated risk.

The events of July 2026 moved AI risk from the theoretical to the actual. AI agents that can reason, act, and adapt are now capable of causing real harm, not through malice, but through pursuit of their objectives. Prompt injection attacks are turning AI agents into unwitting tools for external attackers, and this threat is growing rapidly.

For South African organisations, the regulatory landscape is already in place: POPIA, the Cybercrimes Act, the Joint Standard for financial institutions, and sector-specific obligations all apply. What remains to be determined is whether your organisation is ready to meet its obligations when the incident is caused by your own AI system.

Written by Tebogo Sibidla, a Director at Werksmans Attorneys