Everywhere we look, every day, we see new AI tools competing for our attention. Whether it is social media or by word of mouth. It is everywhere. Every new product promising businesses more efficiency, cost and time savings. However, most entrepreneurs and procurement teams are failing their organisations by treating AI like a standard subscription service—focusing on uptime and seat counts while completely ignoring the "black box" risks of confidentiality leaks, unlawful data processing, and algorithmic bias.  AI is not ordinary software. 

By rushing into these agreements without specialised legal and technical scrutiny, you are creating a massive liability gap. To integrate AI safely, you must move beyond the "Ordinary Software" delusion and address the reality of how these systems handle your most precious asset; your data.

The central contracting mistake is assessing an AI product solely on price, functionality, and uptime. Traditional software is deterministic and predictable; AI is probabilistic and often opaque. You and your procurement team would be failing if you are not auditing the entire data supply chain. You must look under the hood to understand exactly what information enters the system, how it is processed, and where it travels. If a system produces a biased, infringing, or illegal output, your organisation cannot simply point to the vendor. 

Accountability must be architected into the contract, not added as a footnote. Before appointing an AI vendor, an organisation must understand what information enters the system, where that information travels, how the system produces its outputs and who remains accountable when something goes wrong.

Your Data is the Fuel 

Most AI vendors use "service improvement" clauses as a Trojan horse to train their models on your proprietary information. A critical technical nuance that many leaders miss is that the vendor's data grab is not limited to the files you upload. Your contract must expressly prohibit the use of telemetry, user feedback, and derived data for model training. If a vendor uses your prompts to train a shared model, your intellectual property may be incorporated into future products for other customers, making deletion technically impossible. 

A hidden "opt-out" in an account settings menu is not a legal safeguard; you need a hard contractual prohibition that requires prior written authorisation for any data usage beyond the immediate service delivery.

You Cannot Outsource Accountability (The POPIA Reality)

Under the Protection of Personal Information Act 4 of 2013 as amended (“POPIA”), your organisation remainsthe "Responsible Party" for any personal information processed via AI. While the vendor acts as an "Operator," you carry the ultimate legal burden. Under POPIA, an organisation that determines why and how personal information is processed will ordinarily remain the responsible party. The vendor may act as an operator, but outsourcing the processing does not outsource accountability. You cannot contract your way out of statutory responsibility. 

Furthermore, if the vendor processes data unlawfully or transfers it across borders without a lawful basis, the Information Regulator will hold you accountable first. 

"Industry Standard" Security is a Meaningless Phrase

Generic promises of "industry-standard security" provide no protection against AI-specific threats like prompt injection or data exfiltration. Furthermore, POPIA Sections 19 to 21 mandate that you, as the responsible party, must ensure your operators maintain specific security measures through a written contract. To be compliant, your contract must include measurable obligations, including:

• Encryption:  Mandatory at rest and in transit.
• Access Controls:  Multi-factor authentication (MFA) and strict limits on privileged administrator access.
• Data Isolation:  Physical or logical separation from other customers’ data.
• Defensive Safeguards:  Specific protections against prompt injection and adversarial attacks.
• Development Standards:  Evidence of secure development practices and vulnerability management.
• Audit Rights:  The right to request independent assurance reports and summaries of penetration tests. Critically, the vendor must notify you of a security incident the moment they become aware of it. Waiting until they have completed a weeks-long internal investigation is a breach of your own regulatory obligations.

The "Black Box" and the Right to Representations

AI systems can inadvertently reproduce discrimination using "proxies" like postal codes or employment history. When a vendor hides behind "trade secrets" to avoid explaining a model's logic, they are exposing you to massive regulatory risk. This is a direct violation of the spirit of Section 71 of POPIA, which restricts automated decision-making. Your contract must mandate "explainability" and provide the tools for affected individuals to make representations and contest decisions. 

A vendor’s refusal to provide evidence of bias testing or risk limitations is a non-starter for any high-impact deployment in recruitment, credit, or healthcare.

You Can Own the Output and the Lawsuit Simultaneously

"Owning the output" is the most dangerous myth in AI contracting. Ownership does not guarantee the output is original, protectable, or non-infringing. If your AI tool generates content that infringes on third-party intellectual property, your "ownership" simply means you own the legal liability. 

You should guard against blindly accepting a standard liability cap for data breaches or IP infringement in an AI context. It is recommended that users seriously consider:

• Uncapped Liability:  For breaches of confidentiality, security incidents, and unlawful data processing.
• IP Indemnity:  The vendor must protect you against third-party claims arising from their model's output.
• Filtering Tools:  Requirements for the vendor to provide citation or filtering features to prevent plagiarism.

The "Hotel California" Problem (The Illusion of an Exit)

AI systems often become so deeply embedded in your operations that leaving becomes impossible. This is the "Hotel California" of tech: you can check out, but your data stays behind. Your exit strategy must be settled before you sign. This includes:

• Usable Formats:  Exporting data and configurations in a format another vendor can use.
• Certification of Deletion:  A written guarantee that all prompts, logs, and fine-tuning data are wiped from the vendor's environment and backups.
• Business Continuity:  You must have a plan for model unavailability. If the vendor goes insolvent or the model is discontinued, your business processes should not grind to a halt.

Conclusion

AI contracting is not about achieving zero risk; it is about maintaining control, transparency, and accountability. The goal is to ensure that you are the one steering the technology, not the other way around. Governancemust begin before the first byte of data is uploaded. 

As you review your current vendor list, ask yourself one critical question:  If the Information Regulator conducted a snap audit of your AI tools tomorrow, could you provide the written contracts and evidence of human oversight required by law? 

Written by Nicolene Schoeman-Louw; Specialist Technology, Commercial and Contract Law; SchoemanLaw Inc