Definition
AI governance is the standing machinery — rules, roles, documentation and approval gates — that decides which AI systems an organisation may build, buy or switch on, and what evidence must exist before it does. What applies to you is settled by three questions, not by your values statement: which role you play (provider, placing a system on the market, or deployer, using one under your own authority), which use case the system serves, and whose territory its output lands in. Under the EU AI Act (Regulation (EU) 2024/1689) two duties reach nearly everyone using AI in the Union regardless of tier — AI literacy under Article 4, and the Article 50 disclosures that require telling a person they are dealing with a machine and marking synthetic content. Everything more expensive is triggered by the use case, never by the technology: a large language model is not regulated; using one to screen job applicants is.
This is the half of responsible AI that happens before anything goes wrong; accountability is the half that happens afterwards, deciding who answers and who pays. Governance is measured by the decisions it changed: a system scoped down, a launch delayed pending evaluation, a vendor rejected for refusing to supply documentation.
The failure mode is the norm: an organisation adopts principles — fairness, transparency, human oversight — publishes them, and wires them into no gate, no owner and no veto. The test is blunt. Name a system this framework stopped, delayed or changed; if nothing comes to mind after a year, what exists is a document, not a governance regime.
How It Works
Working governance runs a pipeline: inventory every system, classify each one, derive the obligations that follow, gate the release on evidence they are met, and keep monitoring afterwards. Inventory comes first and is where most programmes stall — you cannot classify what nobody has listed, and shadow use inside business units routinely exceeds what the central team knows about.
Classification is where the cost is decided. The EU AI Act sorts systems into four bands: prohibited practices under Article 5 (social scoring, untargeted facial-image scraping, emotion inference in workplaces and schools, and others, banned since 2 February 2025); high-risk, reached by two routes — a safety component of a product already covered by EU product legislation and subject to third-party conformity assessment, or one of the eight use areas of Annex III (biometrics, critical infrastructure, education, employment, essential services including creditworthiness, law enforcement, migration and border control, and justice and democratic processes); transparency-only systems under Article 50; and everything else, which carries no product obligations at all. Article 6(3) offers a narrow escape from Annex III for systems doing a purely procedural task, improving completed human work, flagging deviations from prior decision patterns, or doing preparatory work — but any system that profiles natural persons stays high-risk regardless.
That classification then unfolds into a fixed list of artefacts. The provider owes a lifecycle risk management system (Article 9), data governance over training, validation and test sets (Article 10), technical documentation matching Annex IV (Article 11), automatic logging (Article 12), instructions for use (Article 13), designed-in human oversight (Article 14), accuracy, robustness and cybersecurity measures (Article 15), a quality management system (Article 17), a conformity assessment before market placement (Article 43 — self-assessment for most Annex III categories, a notified body for most biometrics), CE marking and registration in the EU database (Article 49). Deployers owe less but not nothing: use in line with the instructions, plus an Article 27 fundamental rights impact assessment if they are a public body, a private provider of public services, or score creditworthiness or price life and health insurance — with the result notified to the market surveillance authority.
General-purpose models are governed by a compute number rather than a use case. Chapter V obligations have applied since 2 August 2025, and Article 51 presumes a model carries systemic risk when cumulative training compute exceeds 10^25 floating-point operations. Using the standard 6ND approximation (6 × parameters × training tokens), a 70-billion-parameter model trained on 15 trillion tokens costs about 6 × 7×10^10 × 1.5×10^13 ≈ 6.3×10^24 FLOP — below the line. A 405-billion-parameter model on the same corpus reaches roughly 3.6×10^25 — above it, carrying model-evaluation, systemic-risk mitigation and incident-reporting duties and fines under Article 101 of up to 3% of worldwide annual turnover or €15 million, whichever is higher. The tier is fixed by a training-run budget approved long before any policy team sees the model.
Real-World Applications
The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) is the voluntary US counterpart and the vocabulary most internal policies borrow: four functions — GOVERN, MAP, MEASURE and MANAGE — across 72 subcategories (19, 18, 22 and 13). GOVERN runs continuously; the other three are per-system. Nothing is certifiable and nothing is enforced, which is why organisations pair it with something that is.
ISO/IEC 42001:2023 is that something: the first certifiable AI management system standard. Its Annex A carries 38 controls across nine objectives, and certification requires a Statement of Applicability justifying every control included or excluded, then an external audit and surveillance visits. What is certified is the management system, not any model.
Sectoral model-risk rules predate AI governance and still outrank it in regulated finance. The Federal Reserve and OCC's SR 11-7, issued 4 April 2011, set the template — a model inventory, independent validation, "effective challenge" by people empowered to say no, and board-level ownership. It was superseded on 17 April 2026 by SR 26-2, Revised Guidance on Model Risk Management, which the Fed describes as most relevant to banking organisations with over $30 billion in total assets. A bank deploying a model starts from the validation function it already has, not from the AI Act.
Frontier labs govern by release gate. Anthropic's Responsible Scaling Policy activated ASL-3 protections in May 2025 alongside Claude Opus 4, combining weight-security measures with deployment restrictions aimed at chemical and biological misuse. OpenAI's Preparedness Framework version 2 (15 April 2025) tracks biological and chemical capability, cybersecurity and AI self-improvement, with an internal Safety Advisory Group advising leadership on whether deployment is safe. Google DeepMind's Frontier Safety Framework version 3 (22 September 2025) added critical capability levels for harmful manipulation and for models resisting shutdown. Mechanism matters more than labels: a measured capability threshold triggers pre-committed mitigations, binding the release to an evaluation result rather than a launch date. See AI safety for what those evaluations test.
Challenges
Reach surprises organisations outside the EU. Article 2 extends the AI Act to providers and deployers in third countries where the output of the system is used in the Union — so a US vendor screening EU applicants is in scope without shipping software into Europe. That, plus the cost of two parallel model pipelines, produces the Brussels effect: firms apply the strictest regime globally because a second compliance track costs more than one over-compliant one. The effect is real but partial, and weakening as the US federal posture moves the other way.
Fragmentation and the pacing problem are best shown by one law. Colorado's SB 24-205 was signed in May 2024 with a duty of care, a risk management programme and impact assessments; SB 25B-004 pushed its start from 1 February 2026 to 30 June 2026; then SB 26-189, signed 14 May 2026, repealed and re-enacted it as a narrower disclosure regime for automated decision-making, effective 1 January 2027. A programme built clause-by-clause against the 2024 text was obsolete before that statute had ever applied to anyone. The EU timetable has moved too — see the Digital Omnibus deferral described under accountability.
The lesson is structural. Rules are written against a technology whose capability profile changes faster than a legislative cycle: the AI Act was proposed in April 2021, before general-purpose chat models existed as a consumer category, and its GPAI chapter was added late in negotiation. Capabilities that survive text changes — a current inventory, a defensible classification record, evaluation results, a named approver per release — are worth more than a checklist keyed to article numbers that shift.
Future Trends
The centre of gravity is moving from inspecting models to auditing management systems. European harmonised standards drafted by CEN-CENELEC will carry a presumption of conformity for AI Act duties, and the Article 17 quality management system was among the first to reach public consultation. Expect certification, not inspection, to be what buyers ask suppliers for.
Two pressures will reshape the internal function. Agentic systems break the unit of governance: approving "a model" means little when the deployed thing plans, calls tools and acts with standing permissions, so review is shifting to the tool and data access a system holds rather than the weights behind it. And evaluation is becoming the gate itself — capability thresholds, red-team results and bias audits are the evidence that unlocks a release, which loads more weight onto benchmarks than they are yet built to carry.