AI Governance.

What it is and why it matters

Governing AI means deciding who may do what, within which limits, and to whom they are accountable. It includes public policy, organizational decisions and controls over individual systems. The unit of analysis is the whole service: people, data, model, interface, incentives and context. An accurate model can still be embedded in an unfair process.

We do this to deliver demonstrable benefits while protecting rights, safety and agency. It also helps reduce errors, manage supplier dependence and enable complaints. Compliance is one part; a written policy without authority, resources or evidence does not govern actual behavior.

Three levels of governance

At the public level, institutions establish rights, responsibilities, oversight and cooperation. Public procurement, competition, concentrated power and representation of affected people matter. Organizations decide which uses to accept, fund, restrict or reject. Individual systems implement and test those commitments.

Legitimate governance needs reasons for decisions, a voice for those experiencing their consequences and routes to remedy. Participation must be capable of changing the design; consultation with no possibility of changing anything is inadequate. No individual company can resolve unequal access or cross-border impacts alone.

From principles to controls

Privacy requires limited collection, defined purposes and retention, and controlled access. Fairness requires identifying harms and affected groups rather than choosing a convenient metric. Transparency calls for understandable notices and useful explanations. Accountability needs a person with authority to correct, stop and answer for the system.

Turn each commitment into a control with an owner, evidence and a review date. For example, test an assistant for leakage, explain its limits and provide an incident channel before release. Record residual risks, disagreements and the option of not automating.

Inventory and intended use

Start by identifying internal models, external APIs, AI embedded in products and informal employee use. Record purpose, users, affected people, owner, supplier, version, data, permissions, location, criticality, status and review date. A controlled spreadsheet can be sufficient initially.

Classify uses by their effects and context. The same model that summarizes public documents may require different controls when used in employment decisions. Distinguish information, recommendation and execution. Specify excluded uses and conditions requiring reassessment, including supplier changes.

1. DefinePurpose, non-AI alternative and excluded uses.
2. IdentifyData, affected people, supplier and owner.
3. AssessHarm scenarios and applicable obligations.
4. TestControls, boundary cases and reproducible evidence.
5. DecideAccept, restrict, modify or reject with reasons.
6. MonitorOutcomes, complaints, changes and incidents.
7. ReviewCorrect, reassess or retire.

Impact and risk assessment

Describe scenarios: what could go wrong, who could be harmed, through which pathway and with what consequences. Assess severity, scale, reversibility and uncertainty as well as likelihood. Consult users and affected groups. Compare with the current process and non-AI options: automation is not the only possible outcome.

A matrix can help prioritize, but does not turn uncertain judgments into facts. A serious harm must not disappear inside an average. Retain evidence, controls, residual risk and the accountable owner’s decision. Privacy and fundamental-rights assessments have their own applicability conditions and are not interchangeable documents.

Data governance and documentation

Document data provenance, permissions, representation, labeling, transformations, exclusions and limitations. Keep training and evaluation separate to avoid leakage. Dataset and model documentation can communicate intended uses, evaluations and restrictions; documentation does not replace testing.

Maintain traceability between data, code, configuration, model version and results. Define deletion, access and update procedures. For external services, examine contractual terms on training, retention, subprocessors and transfers. Availability alone is not a reason to put personal data or secrets into a tool.

Measure quality, fairness and limits

Evaluate representative cases, boundary situations and relevant groups. High overall accuracy can conceal concentrated errors. Report sample sizes and uncertainty. Choose metrics for the harm at issue: false rejection, false acceptance, prediction error or unsupported output. Fairness criteria can conflict.

For generative models, separate retrieval quality, faithfulness to sources, usefulness and safety. Use human evaluation with documented instructions and disagreements. A model judging another model may accelerate the work, but its judgments need validation. Plausible explanations and confidence scores guarantee neither truth nor causality.

Generative AI and agent security

A retrieved document or external page may contain malicious instructions. Treat that content as untrusted. Separate instructions and data, validate outputs, and limit tools, secrets and network access. Text filters are one layer, not a sufficient security boundary.

For agents, use minimal permissions, explicitly authorized actions, isolation and confirmation for sensitive operations. Add limits on cost, time and steps, protected logs and a tested operational stop. Test tool failures and supplier changes. Design recovery and rollback before enabling real-world actions.

Human oversight that works

Putting a person at the end does not guarantee control. Reviewers need time, training, independent information and authority to disagree. Design interfaces to expose uncertainty and support review rather than encourage automatic acceptance. Measure workload, disagreements and errors in the combined human-machine process.

Affected people need an understandable route to ask questions and complain, followed by review capable of providing remedy. Define who receives the complaint, how evidence is retained and the proposed internal response time. Investigate whether oversight detects unfamiliar failures rather than merely recording an approval.

Suppliers and the supply chain

Before procurement, request evidence on capabilities, limitations, security, data use and incident handling. Check whether you can record versions, assess changes, export information and leave the service. Commercial reputation is not a substitute for evidence.

Allocate responsibilities, change notifications, information access and incident cooperation contractually. Identify open components, licenses, dependencies and maintenance. Maintain a continuity plan for outage, withdrawal or changed terms. Outsourcing a model does not remove responsibility for how it is used.

Operation, incidents and retirement

Release begins another stage of control. Observe changes in inputs, outputs, actual use and complaints. Statistical drift is a signal to investigate, not automatic proof of harm. Define alerts, owners, contextual thresholds and periodic review; examine real outcomes when they become available.

For an incident: contain it, preserve access-controlled evidence, assess affected people, correct the failure and decide whether to resume. Review applicable notification obligations. Retirement requires removing access, updating dependencies, handling data and notifying users. Retain what is needed under a defined policy rather than logging conversations indefinitely.

Law, frameworks and standards

The EU AI Act creates obligations according to practices, systems and operator roles; consult the official implementation timeline alongside the current legal text. The original 2024 text has been amended. This guide does not assume that every system has the same obligations or that all rules start on one date.

The GDPR remains relevant when personal data is involved. ORGANIZATION-007 AI RMF offers a voluntary risk-management framework. ISO/IEC 42001 sets requirements for an AI management system; its scope does not certify that every output of every model is safe. Distinguish law, guidance, standards and technical evidence.

In ORGANIZATION-007 AI RMF 1.0, GOVERN establishes policies and accountability; MAP situates use and impacts; MEASURE gathers evidence; MANAGE prioritizes and treats risks. These functions are revisited across the lifecycle. The generative AI profile complements the framework; it does not turn it into a legal compliance test.

What we know and its limits

We can build inventories, control versions and evaluate systems under specified conditions. We can measure particular errors, detect some disparities and reduce exposure through permissions and review. Evidence remains conditional on the data, population and test setting: passing a benchmark does not demonstrate universal safety.

Documentation supports audit but does not itself reveal untested behavior. Accidents and specification failures show how optimizing a measure can miss the real objective. A useful policy acknowledges uncertainty, defines limits and allows a use to be suspended even when a commercial metric improves.

What the field investigates and what we want to know

Open directions include evaluations that predict real harm, oversight of more capable systems, interpretability, contextual fairness and agent control. Incentives, public participation, audit effectiveness and international coordination also matter. The following table proposes questions and reproducible studies; it does not claim that the author has conducted them.

To advance, state a falsifiable hypothesis, define measures before seeing results and compare an alternative. Record conditions, versions, samples, exclusions and negative results. Publish limitations and support replication when privacy and security permit. More measurement cannot by itself decide which distribution of benefits and harms is acceptable.

How to start: a proposed 90-day plan

Days 1–30: appoint a sponsor and owners, inventory uses, address urgent harms and select a pilot. Define limits and interim data and tool policies. Deliverables: inventory, ownership, prioritization criteria and a list of suspended or restricted uses.

Days 31–60: assess the pilot, document data, test controls, consult affected people and rehearse incidents. Days 61–90: independent review, recorded decision, limited deployment where justified and monitoring. This is an adaptable proposal, not a legal deadline or a promise of compliance in three months.

Governance that learns

Governance does not end at approval. Revisit whether the use still delivers value, whether newly affected people have emerged and whether controls still work. Treat each material change as a new decision: more autonomy, a different population, a new data source or a new version can change the risk.

The final question is not only “Can we do it?” It is also “Should we do it this way, who can object, and what would make us stop?” An organization learns when an objection or incident changes its design, resources or rules, and when it can demonstrate evidence of that improvement.