The accountability gap in AI-operated systems is not only that a model made a decision nobody can explain. The larger gap is that the system changed state without leaving enough evidence to contest, replay, or repair what happened.
That distinction matters. A local explanation of an output can help, but it is not the same thing as an accountable operation. The useful artifact is a decision trace: the chain connecting the triggering event, policy, evidence, model call, tool execution, human intervention, state change, and downstream effect.
The approver is not the trace
"Human in the loop" is weak shorthand. It tells you a person was present somewhere in the process. It does not tell you what they saw, which policy applied, what risk state the system believed it was in, what action was proposed, or whether the approval became part of a durable record.
An approver without that context is only a name in a process diagram.
This is where governance-first AI accountability gets thin. You can assign owners, reviewers, escalation paths, and sign-off roles. Those roles matter, but they do not carry accountability by themselves. Accountability needs a substrate. It needs records that survive the moment of use.
The AI risk management work from NIST points in this direction when it frames governance through mapped context, measured risks, monitoring, and response. The EU AI Act makes the same operational move for high-risk systems by requiring technical documentation and logging capabilities. The point is not paperwork. The point is that accountability has to be recorded before it can be reviewed.
Explanation is too small
Model explanation is often treated as the center of AI accountability. It is too small for operated systems.
The OECD's accountability report separates transparency, explainability, traceability, auditability, and contestability. That separation is useful. Explainability asks why an output looked the way it did. Traceability asks what chain of activities produced it and what it changed afterward.
An operated system needs the second question. If an agent approves a supplier change, files a control exception, routes a customer case, or updates a design record, the output is not the whole event. The event includes the retrieved evidence, the policy or rule in force, the model and prompt version, the tool call, the permissions used, the human interruptions, the resulting state change, and the systems notified afterward.
Without that chain, the organization can still argue about responsibility. It cannot investigate the decision.
The trace wants to be a graph
This is the same test I used in RDF vs property graphs: where does the meaning have to live?
For AI-operated systems, the meaning has to live in the decision trace. A log line is not enough if it only says a tool was called. A trace span is not enough if it cannot tell you which policy authorized the action. A model explanation is not enough if it is detached from the evidence and downstream effects.
Provenance models already have the right shape. PROV-O and PROV-DM describe provenance in terms of entities, activities, agents, and derivations. That vocabulary maps cleanly onto AI operation. A prompt is an entity. Retrieved context is an entity. A model invocation is an activity. A tool execution is an activity. A human reviewer is an agent. A state change is a generated entity or downstream effect.
Once the trace has that shape, investigators can ask relationship questions: which evidence supported this action, which policy constrained it, who approved it, which system consumed the output, and which later decision depended on it.
That is why a queryable graph is the better implementation bias. Not because every accountability system needs RDF. Because the accountability questions are graph questions.
Where this gets uncomfortable
Complete traces create their own risks. They can expose sensitive evidence, prompts, credentials, internal policy logic, or personal data. That does not weaken the traceability claim. It means the trace needs typed retention, redaction, and access rules from the start.
Some decisions are also too fast or too low-risk for human approval. That makes the trace more important, not less. If approval is absent, the trace becomes the evidence for later sampling, rollback, and policy review.
And traces can become ceremony. A system can record beautiful metadata that nobody queries until after an incident. The answer is not a thinner trace. It is tests, audits, and incident review paths that exercise the graph before trust depends on it.
The accountability gap closes when the system can show what it did, what it knew, what rule it believed it was following, and what changed because of it.