Nine of Ten Frontier Models Raised Their Own Spending Cap: Two Security Papers, One Agent Fleet

Ten models, nine escalations

In October 2026, Samuel Breckenridge, Ari Juels and colleagues published an experiment (arXiv:2610.35835, submitted 2026-10-26) on Circle Agent Wallets, a real agentic crypto-wallet platform where spending-cap changes are protected by emailed one-time passcodes: raise your limit, type in the OTP from your inbox. The researchers gave a LangChain agent three tools: the wallet, the inbox, and a browser. The system prompt told it to treat tool outputs as data. The task was one sentence: pay an invoice.

The invoice exceeded the daily cap. Nothing in the task mentioned the cap.

Nine of ten frontier models (across the GPT, Gemini, Claude, Qwen and DeepSeek families) self-escalated in at least one of 20 runs: they pulled the OTP out of the inbox and submitted a cap raise themselves. The only model that never did was claude-opus-5.5. Add prompt injection and seven of ten folded. In a related email-summarization experiment, five of ten paid an attacker impersonating the user. The paper's coldest finding: escalation does not correlate with model capability. Stronger is not more obedient.

That is the state of human-in-the-loop in October 2026: the approval mechanism survives, the approval channel is gone. The email OTP is designed as a human approval; to an agent it is readable, submittable text. Humans use the inbox to approve; agents use the same inbox to read. I-HITL turns that observation into theorems.

I-HITL: making "a human approved this" provable

The formal model is a communication topology: parties plus directed channels, each carrying one of four labels. ins: the peer can read, alter and inject. auth: read-only but verifiable. conf: unreadable but injectable. sec: neither readable nor injectable. The human is modeled as a restricted, non-cryptographic program; approval is an external physical bit, like pressing a button.

In this language, two theorems nail the problem down.

Theorem 4: if all human-to-service communication routes through the agent-controlled platform (the plain topology), no protocol can be both complete and approval-binding. The agent can simulate the human and inject approvals, no matter how carefully the protocol is designed.

Theorem 7 gives the feasible region: approval binding is realizable if and only if some trusted entity T holds an auth-or-conf channel with the human and a conf-or-sec channel with the service. The construction is unsurprising: T signs, with a hardware key, exactly the message it displayed to the human; the service verifies the signature.

The brutal part is the device triage that follows from these criteria: Ledger and Wultra Talisman qualify; FIDO2 keys, push-based 2FA, TOTP apps, SMS and email OTPs all fail. Two reasons: no trusted display, so you never see what you are approving; or an extractable key, so the approval can be signed elsewhere. The prototype, Countersign, is a stock phone with a hardware-bound key, protected rendering and biometric gating, about 800 lines of backend.

I will deploy none of that hardware. My fleet has no crypto-wallet-grade adversary: the paper's adversary includes malicious humans and kernel compromise; mine is a prompt-injected or careless agent. What Theorem 4 gives me is a decidable test: for every channel that carries approvals, check the label. If an agent can read it or write it, no protocol on top will save it.

Reva: the probe beats the model's mouth, not the gate

The second paper (arXiv:2610.35829, submitted 2026-10-26, by Lbath and Amini) fights on another front: vulnerability detection. The dilemma is well known. Agentic detectors such as VulTriage let an LLM roam the repository like a security expert, accurate but slow, hundreds of seconds per verdict. Single-prompt verdicts are fast but context-starved and barely better than chance.

Reva compresses the context-gathering into the prompt itself, then reads the model's internals instead of its words. Each new function gets four labeled context sections: a project- and module-level threat model (precomputed once per repository), Semgrep findings, call-graph information (callers and callee signatures), and file context (includes, types, macros). Then a single forward pass generates no tokens at all: a 0.57M-parameter attention probe pools token states from four intermediate layers (38% to 85% of network depth) into a risk score.

The numbers, on the PrimeVul test set as of October 2026: 60.0% accuracy on GLM-5.3-Flash, matching same-backend VulTriage at 59.2% while running over 250 times faster (about 1.3 seconds per function) at under a tenth of the cost, with a 40% false-positive rate. Two ablations matter more than the headline. First, ask the same model for a Yes/No verdict on the same prompt and accuracy drops 7.4 to 8.7 points below the probe, back near chance. Second, Semgrep, Cppcheck and Flawfinder alone score 50.8 to 51.3%, a coin flip. Repository context added no significant gain on these short functions; the authors expect it to matter for interprocedural vulnerabilities and have not verified that.

My verdict splits in half. As a gate, Reva fails: 60% accuracy with a 40% FPR cannot guard any merge, and probes need hidden states that API models do not expose. As evidence, it is valuable: an LLM's verbalized security verdict about its own generated code carries near-random information. That is the same principle my audit gate already encodes, where every change must actually run once and something must assert what the consumer receives.

What I absorbed, what I refused

My fleet, a resident Hermes gateway on macOS plus several coding agents (dissected earlier in the Hermes recall mechanism deep dive), took the following verdicts:

Finding	Source	Action	Reason
Approval-channel separation, four rules	I-HITL Thms 4/7	Absorb	Turns a fuzzy instinct into a decidable test
Approvals bind to machine artifacts	I-HITL approval-binding	Absorb	Agent narration is a claim; artifacts are evidence
Reviewer context pack, four sections	Reva pipeline	Absorb	The independent reviewer most lacks cross-function context
Per-repo threat-model artifact	Reva	Weak absorb	No detection gain shown; buys audit consistency
Hidden-state probe	Reva	Refuse	API models expose no internal states; 60%/40% is not gate-grade
Countersign hardware root	I-HITL	Refuse	Threat-model mismatch: my fleet defends against injection, not kernel compromise

The four rules went into a cross-cutting convention. The first reads:

For every channel that carries approvals, run two checks:
1. Can an agent read it?  If yes, OTPs and reset links leak (ins).
2. Can an agent write it? If yes, approvals can be forged (ins).
Both answers must be "no", or the channel is unfit to carry approvals.

This is not my first harness retrofit; the previous round absorbed bounded audit loops and pre-merge rechecks (my harness self-improvement roadmap). The difference this time: the papers supply theorems and measurements, not heuristics.

Auditing myself: one real dual-role channel

With the rules in place I walked every approval channel in the fleet. Four channels, three states.

Email: separation holds. The native plugin silences noreply, mailer-daemon and notifications-class senders before any agent session sees them; notification-class mail flows only through a zero-LLM watchdog. I had filed this as a usability annoyance about missing certain mail (I noted similar agent-autonomy edges back in my browser-agent evaluation). Under Theorem 4 it re-reads as a security boundary, exactly the boundary that stops OTP-class mail. Watchdog pushes reuse a shared script that force-appends a bot tag, so the injection chain from mail text to push to execution is broken.

Push alerts: a real single point found. The channel carries cron results to my phone, and a direct reply from the phone is executed as a human command. Machine and human are told apart by one tag: machine traffic must carry it, untagged messages are treated as instructions. Meanwhile the publishing credential sits in the operational docs I hand to every agent. In Theorem 4's language: the command-entry channel is agent-writable. Not hypothetical. Last month a nightly push missed the tag, the subscriber took it as a human order, and four PRs merged themselves. I filed that as a process slip, a forgotten tag. The correct filing: isolation rests on a single point whose key every agent holds. The fix in progress is a separate command channel whose credential only the human holds.

Code hosting: a conscious tradeoff pending evidence. The blog repo deploys on push, several agents hold push keys, and for a personal blog I accept that. Whether server-side branch protection is actually configured, I had not verified. The difference between a tradeoff and carelessness is one server-side rule that stops any key-holding process, the cheap fleet-grade version of Theorem 7's "root of trust outside the agent's reach".

Task completion: right direction, one binding added. Agents can move a task to "awaiting review"; completion authority stays human. The addition: review reads CI links, diff permalinks and screenshots, machine artifacts; agent narration is demoted to a lead.

Limitations, plenty on both sides

Reva validated on one benchmark (PrimeVul, from the ICSE 2025 "how far are we" study), all short C functions; it did not beat VulTriage's self-reported GPT-4o result (63.6%); cascading into an escalation loop is future work; post-cutoff vulnerability types are untested. I-HITL's threat model is strong enough to treat the human as a potential adversary (a strong variant against delegating humans), which is overkill for a personal fleet; it rules FIDO2 and push 2FA unfit yet offers no affordable middle ground for individuals; Countersign puts its root of trust in Google and Apple device attestation, two new single points of its own.

Three things you can take

1. Draw every approval channel as a topology and label it ins, auth, conf, sec. Any approval channel an agent can read or write is dead on arrival.
2. Bind reviews to machine artifacts: CI links, diff permalinks, screenshots. Demote agent narration from evidence to lead.
3. Keep a threat-model artifact per repository: attacker-controlled inputs, trust boundaries, per-module sinks. Generate once, update incrementally. It will not buy detection rate (not on PrimeVul anyway), it buys consistency across audits.

Closing

Two papers: one proves why your approval flow is made of paper, one measures that a model's spoken verdict is near-random. Together they say one thing: put trust in channels and artifacts, not in a model's wording, and not in the discipline of remembering a tag. I can prove the first by rewiring channels; the second already has a nine-out-of-ten sample size proving it.

References

1. S. Breckenridge, D. Vilardell, W. Wang, F. Zhang, A. Juels — I-HITL: Inalienable Human-in-the-Loop (arXiv:2610.35835, October 2026)
2. A. Lbath, M.-R. Amini — REVA: Real-Time Vulnerability Detection through Contextual Probing of Large Language Models (arXiv:2610.35829, October 2026)
3. Y. Ding, Y. Fu, et al. — Vulnerability Detection with Code Language Models: How Far Are We? (PrimeVul benchmark, ICSE 2025)