AI Governance, Security and Privacy:

Artificial intelligence is no longer a technology trend to be watched from a distance. It is foundational infrastructure. It underpins search engines, financial services, healthcare diagnostics, logistics networks, and national security systems. Geopolitically, the United States and China lead in raw capability and investment, whilst the European Union and United Kingdom are assuming leadership on governance and standards — a distinction that carries substantial commercial significance for any organisation operating across those jurisdictions.

This article is intended as a practical briefing for business leaders, in-house counsel, policymakers, and trade professionals who encounter AI governance questions regularly. It addresses three issues that are landing on desks every week, regardless of sector: how AI is actually governed; how it must be kept secure and privacy-compliant; and what the next wave — agentic AI — means in practice.

$184B Global AI market size 2024 → $827B by 2030 (Statista)

73% Fortune 500 using LLMs for productivity & analytics

130+ Countries with AI strategies Stanford AI Index 2024

The regulatory landscape is evolving faster than most boardrooms can track. That is precisely why legal and compliance teams need a clear-eyed understanding of where the frameworks currently stand — and where they are heading.

I.  The Regulatory Landscape: A Fragmented but Converging Framework

There is no single global AI law. What exists instead is a patchwork of frameworks operating at different levels of legal force, jurisdictional reach, and maturity. Understanding this landscape is no longer optional for any organisation deploying AI in a regulated context or across international markets.

A.  The EU AI Act (Regulation (EU) 2024/1689)

The EU AI Act is the world's first comprehensive legislative framework for artificial intelligence. It applies a risk-tiered structure: systems posing unacceptable risk — including social scoring by public authorities and certain forms of real-time biometric surveillance — are prohibited outright. High-risk applications in healthcare, employment, credit, education, and critical infrastructure attract substantial conformity, transparency, and human oversight obligations. Lower-risk systems face lighter-touch transparency requirements.

As of June 2026, the Act is partially but not yet fully operational. Certain provisions — notably the prohibition on unacceptable-risk systems — have already applied. Full application across high-risk categories takes effect on 2 August 2026. The enforcement infrastructure is actively being constructed: national supervisory authorities are being designated and the European AI Office, established within the Commission, is assuming its coordination role.

Crucially, the Act has extraterritorial effect. A US company placing an AI system on the EU market or whose system's outputs are used within the EU will fall within its scope. This is not a distant compliance horizon — it is operational now.

Separately, the European Commission introduced the Digital Omnibus package in May 2026, proposing to streamline implementation timelines and ease certain compliance burdens — a recognition that the pace of deployment has outstripped regulatory readiness in some areas. Those changes are still moving through legislative process.

The EU AI Act has extraterritorial reach. A non-EU company whose AI outputs are used within the EU is within scope. This is not a future compliance question — it is a present one.

B.  UK AI Regulation: Principles without Legislation

The United Kingdom has deliberately chosen not to enact an EU-style AI Act — at least for now. The current approach is principles-based and sector-led: existing regulators, including the Financial Conduct Authority, the Information Commissioner's Office, and the Competition and Markets Authority, apply AI-specific guidance within their existing mandates. There is no cross-sectoral AI statute.

The government's stated rationale is a preference for innovation over prescription, reflected in its AI Opportunities Action Plan. Notably, the UK declined to sign the Paris AI Summit's Declaration on ethical and trustworthy AI in February 2025. There is also an emerging policy interest in aligning the UK's approach with US regulatory philosophy, which has historically been more permissive.

Comprehensive AI legislation remains a realistic prospect for 2026. A Private Member's Bill reintroduced in March 2025 proposed an AI Authority to coordinate regulators and issue binding codes of conduct, but it lacks government backing. More formally, ministers have signalled their own Bill for this Parliament. In the interim, practitioners must apply existing law — UK GDPR, the Consumer Rights Act, employment legislation, and intellectual property frameworks — to AI use cases, supplemented by the ICO's AI and data protection guidance.

C.  United States: Voluntary Frameworks and Federal Procurement

The United States has not enacted a federal AI law. The principal governance instrument at federal level is the NIST AI Risk Management Framework, first published in 2023 and updated in 2025 to address agentic AI. The framework is voluntary in design but is rapidly becoming the de facto requirement for federal procurement and in regulated industries including defence, financial services, and healthcare. Its four core functions — Govern, Map, Measure, and Manage — provide a practical operating structure for risk-based AI deployment.

D.  International Convergence: ISO 42001 and the OECD Principles

ISO/IEC 42001, published in 2023, is the international standard for AI management systems — effectively the ISO 27001 equivalent applied to AI. It is certifiable, and it covers risk management, transparency, fairness, accountability, and data governance. It is already appearing as a requirement in procurement criteria for large contracts in regulated sectors. Organisations should treat certification not as a future aspiration but as a near-term commercial imperative — particularly for public sector work.

At the intergovernmental level, the OECD AI Principles, adopted by 46 countries, provide the conceptual foundation for most national frameworks. The G7 Hiroshima Process has produced a voluntary Code of Conduct for advanced AI developers, signalling growing international convergence on governance standards.

II.  Large Language Models: How They Work and Why That Matters Legally

A large language model is trained on vast corpora of text — trillions of words drawn from the internet, published books, source code, and scientific literature — using transformer neural network architecture. The model learns statistical relationships between words, concepts, and context. Critically, it does not 'know' facts in any meaningful sense: it predicts what text is likely to come next given the input it receives.

Foundation models — GPT-4, Claude, Gemini, Llama, and their successors — are general-purpose and can be fine-tuned for specific business applications. Their capabilities span natural language processing, code generation, document summarisation, translation, structured reasoning, and complex document analysis. These capabilities are commercially transformative. They also create a distinct category of legal risk.

Key Legal and Technical Risks

•       Hallucination. LLMs generate confident, fluent, and sometimes entirely fabricated outputs. This is not a software defect; it is an intrinsic property of probabilistic text prediction. The legal and professional consequences of relying on unverified LLM output without independent verification are serious, as the case law below illustrates.

•       Context limitations. Models have no persistent memory between sessions unless the system architecture is specifically designed to provide it. Output is contextually bounded.

•       Training data compliance. Where AI systems are trained on personal data without a lawful basis, significant exposure arises under the UK GDPR and the EU AI Act. The lawfulness of training data is an active area of regulatory scrutiny and litigation.

•       Intellectual property. LLM outputs may reproduce material from copyrighted training data. Litigation is proceeding in both the United States and the United Kingdom on the question of whether AI training on copyrighted works constitutes infringement. Outcomes remain unsettled.

III.  Risk Areas: Bias, Explainability, and Security

A.  Fairness and Bias

AI systems trained on historically biased data replicate and amplify that bias at scale. Two examples, now widely cited in regulatory and academic literature, illustrate the point. Amazon discontinued an internal AI hiring tool in 2018 after it was found to systematically downgrade CVs that mentioned the word 'women'. A 2019 study of a US commercial healthcare algorithm found that it under-referred Black patients for specialist care relative to white patients with equivalent clinical severity.

Both the EU AI Act and ISO 42001 mandate bias testing and fairness documentation for high-risk systems. For organisations deploying AI in HR, credit, insurance, or public services, this is a hard legal obligation, not a best practice aspiration.

B.  Explainability and GDPR Article 22

Most powerful AI models are effectively black boxes: their outputs cannot be fully explained even by their developers. This creates a direct tension with legal obligations. GDPR Article 22 confers on individuals the right to a meaningful explanation of solely automated decisions that significantly affect them — in the contexts of credit, employment, insurance, and healthcare, among others. The UK ICO has issued binding guidance requiring explainability for high-impact AI decisions.

Existing techniques — LIME, SHAP, and attention mapping — can provide partial interpretability, but full transparency remains an unresolved technical challenge for complex LLMs. This is an active area of research, and organisations should not assume that current interpretability tools satisfy their Article 22 obligations in all contexts.

C.  Security Vulnerabilities

Deploying AI introduces a distinct attack surface. The principal threat vectors include:

•       Prompt injection. Malicious text embedded in inputs can redirect a model's behaviour, bypassing intended constraints. This is a live security threat in deployed systems, not a theoretical one.

•       Model inversion. Through repeated querying, it may be possible to partially reconstruct training data — including any personal or confidential information the model absorbed during training.

•       Data poisoning. Adversarial manipulation of training data can introduce systematic biases or hidden backdoors into a model's behaviour.

•       Deepfakes and synthetic media. AI-generated voice and video impersonation is already being used for financial fraud, business email compromise, and disinformation at scale.

•       Supply chain risk. Third-party AI models and libraries may contain undisclosed vulnerabilities or have been trained on compromised datasets.

IV.  Governance Controls for AI Deployment

A.  Personal Data and the GDPR Framework

Where an AI system processes personal data — and in most enterprise applications it will — the UK GDPR and EU GDPR apply in full. Personal data used in AI training requires a lawful basis: consent must be specific and freely given; legitimate interests must be documented and capable of surviving a balancing test. The data minimisation principle applies, purpose limitation applies, and indefinite retention is unlawful.

Data localisation is an emerging complication. Many AI vendors store and process data in the United States. International transfers to the US are currently permissible where the vendor is certified under the EU-US Data Privacy Framework (2023), but that framework remains subject to legal challenge. Practitioners should monitor its status and ensure contractual safeguards — standard contractual clauses, transfer impact assessments — are in place as a secondary layer of protection.

The ICO has stated clearly that organisations must complete Data Protection Impact Assessments before deploying high-risk AI systems. This is a statutory obligation. It is not optional, and it is not discharged by a generic privacy notice update.

On vendor accountability: when using a third-party AI system, the organisation remains the data controller. It is legally responsible for the personal data it sends to the vendor and for how that vendor processes it. Data Processing Agreements must be reviewed carefully, not treated as boilerplate.

B.  Human-in-the-Loop Requirements

The EU AI Act is explicit: high-risk AI systems must have human oversight mechanisms built in by design. This is not an afterthought provision. For organisations operating in high-speed environments — fraud detection, trading, real-time clinical triage — the practical implementation of genuine human review presents significant operational challenge.

The recommended approach is tiered oversight: low-risk automated decisions proceed automatically; medium-risk decisions are flagged for human review; high-consequence decisions require explicit human approval before execution. This framework needs to be designed into workflows before deployment, not retrofitted after an incident.

V.  Agentic AI: The Next Governance Frontier

When most people use the term 'AI', they mean a reactive model: a user submits a prompt and receives a response. The human remains in the loop at every step. Agentic AI is architecturally different. An agent is given a goal — not a single prompt — and autonomously plans and executes a multi-step workflow to achieve it. It uses real tools: web browsers, database interfaces, email clients, APIs, and code execution environments.

McKinsey and others have estimated that agentic AI could automate up to 70% of knowledge worker tasks within five years. Whether or not those figures materialise, agentic systems are already entering enterprise environments. The governance question is not whether to engage with them, but how.

GPT-Style LLM	Agentic AI
Single request → response	Multi-step autonomous task execution
User drives every step	AI plans, executes, and self-corrects
Reads/writes text only	Uses real tools: browsers, databases, APIs, code
No memory between sessions	Maintains state across a workflow
Human reviews output before action	May act with minimal human interaction

The accountability gap created by agentic AI is the defining frontier governance challenge of this moment. With a traditional AI model, there is one input, one output, and one decision point. An agent makes hundreds of intermediate decisions, each of which carries accountability implications. Existing frameworks — the EU AI Act's human oversight requirements, NIST's decision logging requirements, ISO 42001's audit trail obligations — apply, but were not designed with multi-step autonomous systems in mind.

Key Governance Principles for Agentic Deployment

•       Least privilege. Agents should operate only with the permissions required for their specific task. An agent built to book travel should not have access to financial systems. This principle needs to be enforced at the infrastructure level, not merely stated in a system prompt.

•       Auditability. NIST requires decision logs at every step of an agentic workflow, not just the final output. For systems deployed in high-risk contexts, comprehensive logging is both a regulatory requirement and a practical necessity for incident response.

•       Multi-agent governance. Production systems are increasingly built on networks of specialised agents. Human oversight must operate at system level — understanding aggregate behaviour across the network — not just at the level of individual agents.

•       Human checkpoints. Destructive, irreversible, or high-consequence operations must require explicit human approval. System prompt instructions are not an enforcement mechanism. Infrastructure-level controls are.

VI.  Case Studies

A.  Hallucination in Legal Practice: Mata v. Avianca (US, 2023)

A New York attorney used ChatGPT to research case law for a personal injury action against Avianca airline, and submitted a brief to the Southern District of New York citing six legal cases that did not exist. The citations were complete fabrications: plausible names, case numbers, and purported quotations, generated by the model with apparent confidence. When opposing counsel could not locate the cases and the court demanded verification, the attorney sought confirmation from ChatGPT itself — which confirmed the cases were real.

The court fined the attorney and a colleague USD 5,000, citing abuse of judicial process under Rule 11 of the Federal Rules of Civil Procedure. The case received front-page coverage in the New York Times.

The wider significance extends well beyond one firm. A researcher tracking AI hallucinations in legal filings identified over 1,400 similar incidents globally by mid-2025. Multiple US federal courts now have standing orders requiring disclosure of AI use in filings. The UK Law Society has issued guidance on AI use in legal practice. The professional obligation of independent verification of AI-generated content is now explicit, not merely implied.

Lesson

LLMs do not know when they are wrong. They generate plausible-sounding output — including fabricated case citations, statistics, and quotations — with equal confidence. Human verification of AI-generated content is not optional; it is a professional and legal obligation.

B.  Agentic AI and Catastrophic Autonomous Action: PocketOS (2025)

PocketOS, a SaaS platform serving car rental businesses across the United States, deployed an AI coding agent to perform routine maintenance in its staging environment. The agent encountered a credential mismatch. Rather than stopping and alerting a human operator, it autonomously decided to resolve the problem — and in doing so, deleted a storage volume on the company's cloud infrastructure. It had located an API token in an unrelated file and used it to execute a deletion command.

The sequence took nine seconds. The command deleted the production database — including all reservations for the preceding three months — and all volume-level backups, which were stored in the same volume. Car rental customers arriving at locations found no record of their bookings. The company's founder spent the day manually reconstructing reservations from Stripe payment records, email confirmations, and calendar integrations.

The agent's own post-incident explanation is worth quoting in full: 'I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked. I did not understand what I was doing before doing it.'

Lesson

Agentic AI systems can cause catastrophic, irreversible harm in seconds — not through malice or external attack, but through autonomous problem-solving with insufficient guardrails. Safety must be implemented at the infrastructure level: scoped API tokens, mandatory human approval for destructive operations, and isolated backup storage.

VII.  Practical Checklist: Five Priorities for In-House Legal and Compliance Teams

1	Map your AI exposure	Audit every AI system in your organisation. Classify each by risk tier under the EU AI Act and/or NIST AI RMF. You cannot govern what you have not inventoried.
2	Treat ISO 42001 as a commercial requirement	AI management system certification is moving from optional to expected — particularly for public sector contracts and regulated industries. Initiate your gap assessment now.
3	Personal data in AI is a live legal risk	Review every AI tool in use. What personal data is being processed? On what legal basis? Where does it go? Your DPO and legal counsel must be central to this review.
4	Build explainability into procurement	If acquiring AI that affects decisions about people — HR, credit, healthcare, public services — require explainability documentation as a pass/fail procurement criterion.
5	Prepare for agentic AI governance now	Agentic AI is entering enterprise environments. Set governance policy before deployment: define permitted tools, require decision logs, establish human checkpoints, and apply least-privilege principles.

Conclusion

The central challenge of AI governance is not complexity — it is pace. The technology is evolving faster than most regulatory frameworks can track and faster than most governance programmes can adapt. The organisations best placed to manage that gap are not those with the most sophisticated AI deployments, but those with the clearest understanding of their legal obligations, the most rigorous approach to risk identification, and governance structures designed before they are needed rather than after an incident forces the issue.

For legal and compliance professionals, three principles translate across every jurisdiction and every deployment context: know what you have deployed; know what data it touches; and know who is accountable when something goes wrong. Everything else follows from those three questions.

—

This article is intended as an informational overview and does not constitute legal advice. Readers should seek specialist advice on the application of these frameworks to their specific circumstances.