Jev Soft-Ranks Skills — Agents Still Decide

Sixty characters. That is how much of each skill’s description an agent on the Hermes harness sees by default when it picks one of 182 skills for a turn. Pick the wrong one and the turn follows the wrong procedure with real tools; load one when nothing applies and the session carries instructions nobody asked for. TypeSafe’s cookbook for Jev skill selection puts a classifier in front of that pick and cuts both errors by more than half, while leaving the pick itself with the agent.

That last clause is the part worth copying. By Tuesday you can run the same shape on your own roster: every skill stays loaded, a gate asks whether the turn needs a skill at all, a ranker shortlists three, one suggestion line goes into the prompt, the agent makes the final call, and every override gets logged. You judge it on wrong and needless loads measured on your roster, and you keep safety where a ranking cannot reach it.

Chatbots suggest; agents act. A skill is a procedure the agent carries out with its tools, so a wrong load is a wrong runbook executed.

What TypeSafe’s Jev skill selection cookbook measured, and the Sep 17 sync that grows rosters

TypeSafe’s Skill suggestion cookbook starts from a real catalog: 182 skills in 33 categories from NousResearch’s MIT-licensed Hermes agent, whose index cuts each description to 60 characters by default. The cookbook leaves that index alone and puts two Jev requests in front of each turn.

The first carries one Choice over all 182 skills plus three Nouls asking, from different angles, whether the turn needs a skill at all: action on the user’s own systems, a documented procedure, or an answer prose alone could give. The prose question points the other way, so the code flips it before averaging, and a mean under 0.30 means no suggestion. The second request re-reads the top three with full descriptions and the first 700 characters of each SKILL.md, asks a Choice among them plus a fit Noul per candidate, and drops the shortlist when the best fit is under 0.30.

The winner’s name goes into one line in its own block after the roster. The agent keeps its full index and its own judgment, and because the roster never changes, prefix caching over it still holds.

The published run: 488 single-turn requests, 315 covered by exactly one skill and 173 covered by none, answered by claude-haiku-4-5-20251001 with suggestions from jev-1.12, rendered Jul 31, 2026, weeks before Jev’s public launch on Sep 15. The agent alone loaded the wrong skill on 16.8% of covered requests and a needless one on 9.8% of uncovered ones. With the suggestion, 7.3% and 4.0%. Handed the right answer outright, it still erred on 2.5% and 1.2%: the floor no selection method gets under.

Screenshot: TypeSafe docs, “Skill suggestion - TypeSafe AI” (cookbook run rendered Jul 31, 2026), captured Sep 21, 2026.

Read those numbers as a reference implementation, not a forecast: one vendor-run pass, one agent model, an older Jev version, no independent replication. Claude Sonnet 5 wrote the covered requests from each skill’s own SKILL.md, which TypeSafe itself says makes them easier than real traffic. No token-savings measurement is published. The cookbook runs on Hermes; wiring the same flow into your own harness is custom work.

The roster side moved the same week. Claude Code 2.1.275, released Sep 17, 2026, syncs the skills and plugins enabled on a claude.ai account into terminal sessions signed in with it; syncClaudeAiSkills: false and syncClaudeAiPlugins: false opt out. Per the settings reference, that covers interactive and non-interactive sessions, and a false in a committed .claude/settings.json is ignored, so a repository cannot opt out on the lane’s behalf. Somebody enabled each synced skill, just not necessarily the person who owns the lane.

A skill is a procedure the agent runs, so selection errors are action errors

Roster overhead in context is covered in the context engineering playbook, and the bill for anything replayed every turn in memory that burns quota. This piece is about the pick. A wrong load changes what the agent does next: the wrong release procedure, a checklist for another repo. A needless load turns a question that wanted an answer into a procedure that wants to run.

The tempting fix is to hide the roster and hand the agent the top few. That swaps an overridable mistake for an invisible one: when the ranker is wrong, the right skill is gone. Picking a skill from a request and a list is a call a person makes in seconds, which is why it passes the decision-seat tests; writing the skill’s output would not.

Step 1: Inventory the roster each lane actually loads

Per lane, list every skill a session can load, where it came from, and how many characters of each description the harness index shows; that width, not the full SKILL.md, is what the agent picks from. Hash the list per session so you can tell when it changes.

Field	What goes in it	Why it is there
skill	Name as the harness lists it	The Choice option and the log key
source	repo / user / plugin / account-synced	Synced entries arrived without a lane owner
index_chars	Description width the agent sees	The ranker should see what the agent sees
owner	A person, not a team alias	Who fixes a description that keeps losing
last_loaded	From session logs	Candidates for pruning
roster_hash	Per session	Every eval and log line ties to one roster

Give account-synced skills their own rows and a decision per lane: keep sync on, or set the opt-out somewhere other than the committed project settings, which Claude Code ignores for these keys. For unattended lanes, prove which skills a headless run actually read with the clean-versus-used differential run rather than trusting the manifest.

Prune before you rank. Claude Code 2.1.261 (Sep 4) added /skill-doctor, which shows which loaded skills go unused and what they cost in context. A skill no lane has loaded in a month is a removal candidate, not a ranking problem.

Step 2: Build a labeled selection set from your own turns

The cookbook’s metrics are the right ones; its requests are not yours. Pull real first turns from session logs and label each one.

• Covered: exactly one skill is the right load. Record which.
• Uncovered: no skill applies. Copy the cookbook’s mix of three kinds: everyday requests, technical requests with no matching skill, and specific requests your roster happens not to serve (85, 42 and 46 of its 173).
• Ambiguous: two skills could serve equally. Keep these out of the score and in a review pile; they are description problems, not ranking problems.

Score two rates on the first response only, as the cookbook defines them. Wrong load is the share of covered turns where the first skill loaded was not the labeled one; loading nothing counts as a miss. Needless load is the share of uncovered turns where the agent loaded any skill. The first load is the first skill-read tool call (skill_view on Hermes; your harness has its own).

Run the agent alone for a baseline, then an oracle arm whose prompt names the correct skill. The gap between them is the most any ranker can win; the oracle’s rate is your floor. As an illustrative sizing, a few hundred covered turns and 150 or so uncovered ones keep a swing of a few turns from looking like a trend.

Step 3: Gate on “does this turn need any skill?” before ranking

The gate is what fixes needless loads, and a ranker alone makes them worse, because a Choice always names something. Ask the need question as several Nouls and combine them in code.

# skill-gate.yaml: illustrative shape; question texts paraphrase the cookbook's gate
model: jev-1.13.0              # pinned; the cookbook's published run used jev-1.12
gate:
  questions:
    acts_on_user_system:               "The request asks for action on the user's own files, accounts, devices or services."
    would_follow_documented_procedure: "An expert would work from a specific written procedure or command set here."
    prose_suffices:                    "A well-informed generalist could answer completely in writing, with no tools or docs."
  inverted: [prose_suffices]   # scored as 1 - p before averaging
  combine: mean
  suggest_nothing_below: 0.30  # the cookbook's value; tune yours on the labeled set

Three rules keep the gate honest.

1. Threshold on the Noul probability. Nouls carry no confidence value, so the probability is the only thing to threshold.
2. Flip deliberately; never derive. The cookbook inverts one question on purpose. Do not compute “needs a skill” as one minus some other answer you happen to have: TypeSafe’s jev-1.13 jaggedness page shows a question and its negation, asked as two Nouls, summing to 1.19.
3. Tune the floor on both rates at once. Raising it cuts needless loads but strips suggestions from covered turns. Set it from your labeled set and version it in the policy file.

Step 4: Rank the whole roster to three in the same call

Put one Choice over the roster in the same request as the gate Nouls, with each skill’s index description as its criterion: the same text the agent sees. Take the three highest probabilities as the shortlist. One call carrying a Choice and several Nouls against one state is the pattern from the decision-vector piece: keyed answers, policy in code.

Two documented limits shape this call. Per the API reference, a Choice takes at most 255 options, so a larger roster needs a first pass by category or source, then a second Choice inside the winner. TypeSafe’s models page also sets two budgets that apply at once: 64K tokens for state plus every question, 32K for state plus the longest question. The roster Choice is usually the longest, so check state plus roster against 32K in code before sending; the cookbook’s 182-skill roster prompt ran 16,089 characters.

The state is the user’s latest request plus the short context the need questions require, filtered in code; the jaggedness page advises sending only the fields a question needs. A transcript is not a state.

Step 5: Re-read the shortlist before suggesting anything

The first call ranks names and short descriptions. The second reads more: for each of the three, the full description and the opening of its SKILL.md (700 characters in the cookbook). Ask a Choice among the three and a per-candidate fit Noul, and drop the shortlist when the best fit falls under your floor.

The pairing is deliberate. A Choice is relative and names the best of three even when all three are poor; a Noul is absolute and can come back low for every candidate. The jaggedness page shows one question returning 0.22 as a Noul and 0.01 on “yes” as a yes/no Choice, so keep separate, versioned floors for the gate, the shortlist Choice and the fit Nouls.

Two calls, one line, one decision that stays with the agent. Every path that fails ends in no suggestion, and the permission wall sits under all of it.

Step 6: Inject one suggestion line and leave the roster alone

All of that produces one line, in its own block, placed after the roster. An illustrative version:

<skill_suggestion>
Look at release-notes first for this request. Skip it if it does not match what the user asked.
</skill_suggestion>

The rules around that line are what make it a suggestion rather than a filter.

• Never remove, reorder or rewrite roster entries. The roster is a stable prefix. The cookbook places the line after it so caching holds; a reordered roster breaks the cache on every turn.
• One name at most. A list of three is a second roster with worse descriptions.
• Nothing is a valid output. When the gate or the fit floor says no skill applies, inject nothing.
• The line grants nothing. Loading the suggested skill changes no permission, and declining it changes none either.
• The ranker has a deadline. The suggestion must exist before the turn starts, or the turn runs without one.

That last rule is the whole failure path, and it is deliberately benign. A 429, a 529, a 422, a timeout or an answer that fails validation all end the same way: no suggestion, one log line with the reason, and an agent choosing unaided as it did last week.

Set the deadline from your own measured p95. TypeSafe’s three demo turns came back sub-second per call, but the cookbook client sets a 120-second timeout, twelve times the Python SDK’s 10-second default: fine for a batch eval, wrong for a live turn. Retry budgets that outlast a deadline are covered in the Jev fail-mode piece.

Step 7: Log every override, because suggestions also break turns

The headline numbers hide one line of the cookbook’s own output. Of its 315 covered requests, the suggestion fixed 37 that the agent got wrong on its own and broke 7 that it had right. TypeSafe’s explanation is blunt: “A confident wrong suggestion is more persuasive than no suggestion at all”.

Screenshot: TypeSafe docs, “Skill suggestion - TypeSafe AI” (cookbook run rendered Jul 31, 2026), captured Sep 21, 2026.

That is the reason the agent keeps the final call, and the reason every disagreement is data. Write one record per turn; the JSON line after the table is illustrative.

Field	What it records
lane, session, turn	Ties the record to the transcript
roster_hash, qset, policy	Which roster, question set and thresholds were live
model	The versioned ID the response reports
gate	Each need Noul and the mean
shortlist, fit	Top three with probabilities; fit Noul per candidate
suggested	A skill name, or none plus the reason (gate, fit, error, deadline)
first_load	What the agent actually loaded first
outcome	followed / overrode / declined / unprompted
label	Filled at review: which load was right

{"ts":"2026-09-22T10:14:03Z","lane":"docs-bot","session":"s-81c2","turn":1,"roster_hash":"sha256:4be1","qset":"skill-suggest@2","policy":"skill-policy@3","model":"jev-1.13.0","gate":{"acts_on_user_system":0.81,"would_follow_documented_procedure":0.74,"prose_suffices":0.22,"mean":0.78},"shortlist":[["release-notes",0.46],["changelog-audit",0.31],["pr-description",0.12]],"fit":{"release-notes":0.83,"changelog-audit":0.41,"pr-description":0.09},"suggested":"release-notes","first_load":"changelog-audit","outcome":"overrode","label":null}

The four outcomes are the review queue: followed (loaded the suggestion), overrode (loaded another skill), declined (loaded nothing despite a suggestion) and unprompted (loaded a skill when none was suggested). Review overrides and unprompted loads weekly against labels. A right override is a ranker miss worth a description or criteria fix; a wrong one belongs to the agent. Pinning the version the record carries, and why replaying policy differs from asking Jev again, are in the Jev decision-log piece.

Step 8: Score Jev skill selection on your roster, not the vendor’s

Re-run the labeled set with suggestions on: same agent model, same roster hash, pinned Jev version. Report both error rates for all three arms, plus the fixed and broken counts from the log.

Arm	Wrong load	Needless load	Fixed	Broke
TypeSafe cookbook, agent alone	16.8%	9.8%	—	—
TypeSafe cookbook, with suggestion	7.3%	4.0%	37 of 315	7 of 315
TypeSafe cookbook, oracle	2.5%	1.2%	—	—
Your lane, agent alone	measure	measure	—	—
Your lane, with suggestion	measure	measure	count	count
Your lane, oracle	measure	measure	—	—

Vendor-run on jev-1.12 and Haiku 4.5 over 488 requests. The suggestion cuts both errors by more than half; the oracle row shows the floor is not zero.

Ship suggestions on a lane when both rates fall, the broken count is small beside the fixed count, and every broken turn has a cause written next to it in the override log. Re-run the set on every Jev version bump and after any roster change the manifest shows.

Claim context savings only if you measured them. The cookbook measured selection accuracy, not tokens, and never tested loading fewer skills. For a savings figure, compare against your harness’s native lazy loading over the same set, with /skill-doctor or equivalent for per-skill context cost. Loading only the top-k skills is an unmeasured design that removes the agent’s ability to override; treat it as its own experiment.

Step 9: Keep safety policy outside the ranking

Skills are instructions, not permissions. A skill ranked first gets no more tool rights than one ranked last, and a skill the ranker never suggests can still be loaded, so nothing about what a skill may do belongs in ranking thresholds.

Tool permissions, deny rules and the sandbox decide what any loaded skill can execute; intake decides which skills reach the roster at all, the discipline in marketplace hygiene for plugins and MCP. A skill you would not run unsupervised comes off the roster or sits behind a permission wall. It is never merely down-ranked.

The same holds for adversarial input. The jaggedness page says content that argues for its own classification can move an answer, so a request written to favour a particular skill can move the suggestion. Because the suggestion grants nothing and the agent still chooses, that attack buys a nudge, not a capability. Keep such turns in your labeled set.

Where Jev skill selection goes wrong, and the signal for each

The persuasive wrong suggestion. The agent follows a confident miss it would have avoided alone. Signal: broken count rising while fixed holds, and right overrides trending toward zero, which means the agent stopped disagreeing, not that the ranker stopped erring.

Roster drift under a stale eval. A sync or plugin update adds skills, and your scores describe a roster that no longer exists. Signal: roster_hash changes with no manifest change, or a shortlist names a skill the manifest lacks.

Alias drift. An unpinned client rides jev-latest, which maps to jev-1.13.0 today and moves on the next release; TypeSafe advises pinning the versioned ID once thresholds are tuned. Signal: the gate-mean distribution shifts with no roster change.

The roster outgrows the call. Past 255 options, or past 32K for state plus roster, the request breaks a documented limit. Signal: the suggestion rate drops to zero overnight and every none reads error.

Explain-only turns still load skills. The gate floor is too low for your traffic. Signal: needless loads flat after the gate ships.

The ranker becomes a gate. Someone adds a branch that hides a low-scoring skill. Signal: any code path where a ranking removes an option instead of suggesting one.

Skill ranking belongs to the fleet’s policy layer

Nothing in this runbook asks an agent to choose more carefully. The roster manifest, pinned model, question set, thresholds, one-line template, deadline, override log and weekly review live outside every model and apply the same way on every lane, whichever CLI runs it. That is what makes the result auditable: one roster hash, one policy version, one record per turn saying what was suggested and what the agent did.

It is also why ranking sits beside kill switches, permissions and evidence in the layer that runs the fleet, the place a multi-agent command center describes. The ranker narrows attention; the session that can abort the work still makes the choice.

FAQ

Does Jev choose which skill my agent loads?

No. Jev ranks the roster and, when the gate and fit checks pass, one suggestion line goes into the prompt after the roster. The agent keeps its full skill index and makes the final call, and every override gets logged. When the ranker fails or misses its deadline, the agent chooses unaided.

How much does Jev reduce wrong skill loads?

In TypeSafe’s own cookbook run over 488 requests, wrong loads fell from 16.8% to 7.3% with a suggestion, and needless loads from 9.8% to 4.0%. That was one vendor-run pass on jev-1.12 with Haiku 4.5 and no independent replication, so measure both rates on your own roster before trusting it.

Does a top-ranked skill get more tool permissions?

No. Skills are instructions, not permissions. A skill ranked first gets no more tool rights than one ranked last, and following or declining the suggestion changes no permission. Tool permissions, deny rules and the sandbox decide what any loaded skill can execute. A skill you would not run unsupervised is never merely down-ranked.

Sources

• TypeSafe: Skill suggestion cookbook — method, thresholds, results (jev-1.12, Jul 31, 2026)
• TypeSafe: Jev 1.13 jaggedness
• TypeSafe: API reference — question types, 255-option cap, error codes
• TypeSafe: Models — budgets, aliases, pinning
• TypeSafe Python SDK constants
• TypeSafe: Introducing System One models and Jev — Sep 15, 2026
• Claude Code v2.1.275 release — Sep 17, 2026
• Claude Code settings reference
• Claude Code changelog — 2.1.261