Misalignment Disclosures Are a Fleet Evidence Problem: AI Agent Incident Evidence Before the Session Rotates
OpenAI disclosed six misalignment classes from its own runs. Build AI agent incident evidence for your fleet: tool trail, approvals, cost, diff and environment.
Go deeper. Build your own.
OpenAI’s second misalignment report describes a model that wrote instructions into its own task summaries to hide mistakes from the user. The main sample completed on May 30, 2026. The behavior was discovered on July 9. Forty days, by our arithmetic, inside the lab that owns every log (OpenAI).
If an agent in your fleet did the same thing last night on a vendor’s computer, the interesting number is not how long you would take to notice. It is whether the AI agent incident evidence would still exist when you did. Vendor session pages rotate, streams do not replay, and one of the three managed harnesses in this piece has no documented export at all.
So this is the drill: map each disclosed behavior class to the artifact that would show it in your fleet, learn what each vendor surface exposes and what the docs do not say, export before the session rotates, and prove it on a Tuesday by reconstructing last night from the pack alone. The vendor-hosted evidence runbook defined the five-artifact pack and its cadence; this piece is what you do with it once vendors start publishing what agents do wrong.
Sep 16, 2026: OpenAI discloses six misalignment classes in its own models
On September 16, 2026, OpenAI published a page titled “Our framework for reporting model misalignment” and described it this way: “We are sharing a new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI, along with six reports on unexpected or concerning model behavior we’ve observed in the last six months.” The intent is speed over completeness: “This new framework is intended to expedite publishing misalignment reports following observation, even when we haven’t fully explained or mitigated the behavior we’re reporting” (OpenAI).
The six reports, by their link text on the framework page:
- “Self-generated instructions in task summaries”
- “Instructions to conceal mistakes in task summaries”
- “Searching public repositories for exposed API keys, then fabricating information”
- “Uploading files to the internet in order to cite them”
- “Unsanctioned writes and communication through an internal software repository”
- “Unsanctioned file sharing between collaborating agents”
Two things about that list matter more than the list. First, five of the six reports explicitly come from OpenAI’s training and evaluation of internal or unreleased models, and the framework labels none of them a customer-production event. Second, the page offers operators nothing: no monitoring guidance, no logging recommendation, no sandbox advice. It is a disclosure practice for OpenAI, covering among other things “new ways for models to act without authorization, coordinate with other models, or evade oversight.” What follows is the operator’s analog, ours to write, because nobody published one.
Screenshot: OpenAI, “Our framework for reporting model misalignment” (Sep 16, 2026), captured Sep 19, 2026.
The same month, Anthropic’s misuse report described what agents look like when the operator is hostile: “At the far end, operations ran autonomously, with minimal human input or supervision: these included multi-agent frameworks conducting reconnaissance, exploitation, and theft against multiple victims, in parallel, for hours or days at a time.” The mechanics are the same as any coordinator you run: “The operators routinely ran ‘agent swarms,’ where a lead AI agent decomposed reconnaissance and post-exploitation work and dispatched it to many subagents running in parallel.” Its advice aimed at you is about keys and where you buy access: “Organizations should treat AI keys and agent integrations with the same level of seriousness as they do production credentials—because attackers treat them with the same level of seriousness, too” (Anthropic).
Screenshot: Anthropic, “Detecting and countering misuse of AI: September 2026” (September 2026), captured Sep 19, 2026.
Together the two reports give the problem its shape. One vendor says models in its own runs hid mistakes, uploaded files and passed messages through a repository. Another says attackers run swarms of subagents for days. Neither hands you a checklist. And six days before its framework went up, OpenAI had opened the Agents API public beta, the managed harness where a growing share of this evidence will be born on someone else’s disk.
A chatbot’s mistake is in the reply; an agent’s is in the trail
When a model only answered, the evidence of a bad answer was the answer. An agent that acts leaves its mistakes in tool calls, network egress, repository writes and messages to other agents, and the reply at the end can be a tidy summary of none of it. Report two is the cleanest example: “Its summary directed the next context not to mention the mismatch.” The misbehavior lived in a summary that looked fine.
That is why each class maps to a different artifact, and why a screenshot of a vendor dashboard proves nothing. The dashboard shows what the agent reported. The pack shows what it did.
The runbook: AI agent incident evidence before the session rotates
Four steps: a mapping, a reading of the docs, the plumbing, and the drill that tells you whether the first three worked.
1. Map each disclosed class to the artifact that would show it in your fleet
Take OpenAI’s six link texts as the row headers. For each, write down what would show it in your fleet and where that comes from. This is our mapping; adjust it to your harnesses, but do not skip a row.
| Disclosed class | What would show it in your fleet | Where it comes from |
|---|---|---|
| “Self-generated instructions in task summaries” | The compaction summary as text, beside the turn that produced it | Transcript or saved items; whether summaries persist as items, the docs do not say |
| “Instructions to conceal mistakes in task summaries” | The summary beside the diff it describes; a mismatch is the finding | Summary from the transcript; diff from the git host |
| “Searching public repositories for exposed API keys, then fabricating information” | Tool calls with full arguments: the query, the fetch, the key-shaped string, then output citing data with no source | Tool trail with arguments; egress log |
| “Uploading files to the internet in order to cite them” | An outbound write to a host not on the list | Sandbox network policy plus egress log; locally, your proxy |
| “Unsanctioned writes and communication through an internal software repository” | Registry or repository writes outside the task’s scope, by the agent’s principal | Registry and git-host audit logs, keyed by principal |
| “Unsanctioned file sharing between collaborating agents” | Inter-agent messages, with content | Coordinator message items, where exposed; locally, your own bus |
Two rows deserve a note. Class four is the only one a policy can prevent rather than record: hosted Agents API sandboxes accept network.access: restricted, described as “Allow only the hosts listed in allowed_domains” with “1–100 exact host names” (OpenAI Agents API docs). Turn it on, and an upload to an unlisted host becomes a refused connection rather than a discovery weeks later. Class six is the one you are least likely to have; the Agents API docs say “the stream does not provide a full conversation transcript” between agents and that “Coordination items can omit message content” (multi-agent docs). If your subagents talk, the record of what they said may not exist anywhere you can reach.
2. Know what each vendor surface exposes, and what the docs do not say
Illustrative. An operator reading of the docs as fetched Sep 19, 2026, not a vendor specification. Gray means the docs do not say.
OpenAI Agents API. The evidence model is items and turns, not a transcript. “Events show live progress. Items are the saved messages and tool calls, including completed responses.” You list a session’s items with GET /v1/agents/sessions/{id}/items?order=asc&limit=100, list its turns, and retrieve a turn “to inspect its status, timestamps, usage, and error.” Attribution is documented: “Given a command item and its session ID, retrieve the command’s turn to identify the agent that ran it. The turn’s subagent_id is null for the main agent” (events docs).
The gaps are documented too: “Streams do not replay missed events,” and “Saved items let you recover completed work, but not every intermediate event you missed.” Webhooks flag a session needing action, but “The webhook does not include those details.” Files under /workspace/outputs “are published as immutable artifacts when a turn completes; those copies remain downloadable after the sandbox expires,” while the sandbox can be deleted after an hour without keep-alives.
What happens to artifacts on a cancelled turn, the docs do not say. Nor does the API support zero data retention, which the retention-versus-memory piece takes apart.
Claude Code Projects. Projects, redesigned September 17, run threads as cloud sessions under a coordinator that “sees what threads report back, not every step they take.” The thread transcript is readable in the Overview pane, described as “Read what Claude did, step by step,” and a Usage tab shows “token use by thread and by model.” That is the whole surface. “A project belongs to one user. You can’t share a project or its threads with another user, and thread transcripts don’t have the share option other cloud sessions have. There are no organization-level controls for projects during the beta” (Claude Code docs).
No export API, no CLI, no webhook. The one durable artifact is on GitHub, because each thread works on its own branch, and the docs warn that if a sandbox cannot resume “the thread continues from a fresh clone, so uncommitted changes can be lost.” The coordinator piece covers the product; for evidence, the operative fact is that the trail lives behind one person’s login.
Cursor Cloud Agents. The run stream carries tool_call, assistant, thinking and result events, is “scoped to the requested run and does not replay prior runs,” and is time-boxed: “Stream responses include the X-Cursor-Stream-Retention-Seconds header. After the retention window elapses, this endpoint may return 410 stream_expired” (Cursor Cloud Agents API). The docs give the header, not a number, so read it per response.
Get Agent Usage returns per-run inputTokens, outputTokens, cacheWriteTokens, cacheReadTokens and totalTokens. And Cursor Cloud MCP, a diagnostics server available during runs, can “fetch transcripts, diff metadata, environment details, run events, and setup logs,” with batch-fetch-details that “writes events.json per run; up to 50 runs per batch.” Team admins fetch across the team; non-admins see “Only your own runs and transcripts” (Cursor Cloud MCP). Approvals for a Project: the docs do not say.
Local harness. Whatever runs on your own machine writes its transcript to your disk, under your retention, with your names on the approvals. It is the only column with no gray cell, which is the strongest argument for keeping consequential work there when you can.
3. Export before the session rotates: two collectors, one job
The matrix forces two collectors, because no single pull gets everything.
A live consumer because streams do not replay; a post-run pull because the consumer will die once. Both feed one job.
The live consumer attaches when the session starts and writes raw events as they arrive. It exists because of one sentence: “Streams do not replay missed events.” A minute of disconnect is a minute gone from the stream, and the post-run pull recovers only what was saved as an item. On Cursor the same consumer reads the retention header on its first response and schedules a second pull inside that window.
The post-run pull runs on session close and again in a nightly sweep. For the Agents API it is the documented list-and-retrieve calls; for Cursor it is the usage endpoint plus a Cloud MCP batch; for Claude Code Projects it is a human, because there is nothing to call. The shape, with placeholder field names:
# illustrative: post-run pull for one Agents API session; field names are placeholders
H=(-H "Authorization: Bearer $OPENAI_API_KEY" -H "OpenAI-Beta: agents=v1")
B="https://api.openai.com/v1/agents/sessions/$SID"
d="$HOME/evidence/$(date -u +%Y/%m)/openai/$SID"; mkdir -p "$d/raw"
curl -sf "${H[@]}" "$B/items?order=asc&limit=100" > "$d/raw/items.json"
jq -c '.data[] | select(.type | test("call|command"))' "$d/raw/items.json" > "$d/trail.jsonl"
(cd "$d" && sha256sum trail.jsonl raw/items.json > manifest.sha256)For Claude Code Projects the pull is a procedure, not a script: open the Overview, read the thread, screenshot it with the thread id and clock in frame, file it under raw/, and copy the Usage tab’s per-thread tokens by hand. Then make the git host carry the weight: tell the project, in its instructions, to commit and push work in progress at every milestone, so the diff exists somewhere with an API. That is the honest cost of consequential work on a surface with no export.
The approval gate is the third source and it is yours. None of the three vendors documents an approval record with a human’s name on it; the Agents API hands your application an action-required webhook and expects it to answer. Route every gated action through your own gate so the name is in your file, the same discipline the approval-queue playbook argues for on other grounds.
4. The Tuesday drill: prove last night from the pack alone
Pick three sessions from last night, one per vendor if you run three. Close every vendor tab. Answer each question from the archive alone.
| Question about last night | Passes when |
|---|---|
| Which agent, main or subagent, ran the command that touched the production config? | trail.jsonl has the call with arguments, plus a turn record with subagent_id |
| Who approved it, when, with what scope? | approvals.jsonl has a row with a human name, not user |
| What did the session cost, in tokens and sandbox minutes? | usage.json joins to the session id with both meters |
| Does the final summary match the diff it describes? | final.diff and the summary sit side by side, compared |
| Did anything leave the sandbox to a host not on the list? | env.json records the network policy; the egress log is in raw/ |
| Did two agents exchange messages, and about what? | Inter-agent items with content, or a note that this vendor does not expose them |
Six questions, six classes. A row that fails is the drill working. Write the gap down and decide whether to close it with a policy (class four), a collector (class one), or by moving that work to a surface that can answer (class six). Thirty minutes, weekly, and the replay playbook turns the passing packs into something you can step through.
Four ways the incident evidence pack comes back short
| Failure | The signal | First move |
|---|---|---|
| The live consumer died mid-run | Sequence gaps in raw events; items exist for turns the stream never showed | Alert on disconnects; record the gap in the manifest |
| Stream expired before the pull | 410 stream_expired from Cursor |
Read the retention header at first byte; schedule the pull inside it |
| Summary and diff disagree | The drill’s fourth question fails | A class-two finding; pull the full item list, not the summary |
| Artifacts vanished after a cancel | /workspace/outputs copies missing for a cancelled turn |
The docs do not say; export before cancelling, log the gap |
Keeping agent evidence durable is an operating-layer job
Nothing in a model, a harness or a coordinator makes the record more durable, because durability was never their job. It is an operating-layer job, and it sits beside the other ones: the abort bus that stops three coordinators with one human, the allowlist that decides which agent may hold a gated model, and a fleet command center that can produce the trail for a given PR rather than a status page.
OpenAI’s framework exists so that disclosures happen fast, “even when we haven’t fully explained or mitigated the behavior.” Copy the posture. Export fast, before you understand what you are looking at, because the alternative is understanding it perfectly forty days later with nothing left to look at.
FAQ: AI agent incident evidence on managed harnesses
What evidence should I keep when an AI agent misbehaves?
Five artifacts per session: the tool trail with full arguments and attribution, approvals with a human name and scope, cost with both token and sandbox meters, the final diff anchored to the merge commit, and an environment snapshot including the network policy. Export them before the vendor session rotates, then hash them together.
Can I export a Claude Code Projects thread transcript?
Not through an API or CLI as of September 19, 2026. The docs say thread transcripts do not have the share option other cloud sessions have and that there are no organization-level controls during the beta. The transcript is readable in the Overview pane; the durable artifact is the branch and pull request on GitHub.
Sources
- OpenAI: “Our framework for reporting model misalignment” (Sep 16, 2026)
- OpenAI: “Instructions to conceal mistakes in task summaries” (report updated Sep 16, 2026)
- Anthropic: “Detecting and countering misuse of AI: September 2026”
- OpenAI Agents API docs: Overview (data residency and ZDR)
- OpenAI Agents API docs: Events and items
- OpenAI Agents API docs: Multi-agent
- OpenAI Agents API docs: OpenAI-hosted environments
- Claude Code docs: “Let Claude coordinate ongoing work with Projects”
- Cursor docs: Cloud Agents API endpoints
- Cursor docs: Cloud Agent capabilities, Cursor Cloud MCP
