deer-flow PR #4833 Source Deep-Dive: Durable Task Notifications That Never Get Lost, Duplicated, or Noisy

You ask an Agent to start a job that will run for hours, close your laptop, and go to dinner. When you come back and open the browser: is the task still running? Does the Agent even remember it exists? And when the result lands, who tells you?

Those three questions are where every long-horizon Agent system dies. deer-flow's issue #4652 (filed 2026-08-03) states the sharpest pain plainly: task IDs travel inside conversation context, and after a few turns, context compaction compresses them away — the model starts fabricating task IDs that never existed. The workaround — relentless polling — triggers tool-loop problems of its own. Neither failure mode, hallucination nor storm, is fixable with better prompts.

PR #4833 (commit 5ffc2d3, author AnnaSuSu, 68 files, +4,293/−121, merged 2026-08-22) is the complete answer. It is step three of epic #4652 (support for the MCP Tasks extension, SEP-2663):

PR	Merged	Scale	Delivered
#4665	2026-08-08	+1,677/−10, 29 files	Durable task runtime foundation: persistence, polling, leases
#4690	2026-08-15	+3,218/−100, 47 files	First concrete driver: submit/status tools connected
#4833	2026-08-22	+4,293/−121, 68 files	Reliable notifications + session-isolated cancellation + chat UI panel

As of 2026-08-23, deer-flow sits at 80,572 Stars and 11,065 Forks. I wrote up its overall architecture in May; this piece drills one level down: what exactly happens to a "notification" inside DeerFlow 2.0.

Architecture Overview: Three Lease Loops and One Table

The big picture first. The whole mechanism introduces zero new dependencies (the PR description calls this out explicitly), no message queue, no Redis — every reliability guarantee is pushed down into the schema of a single PostgreSQL table:

Four actors: the mcp_tasks table (the single source of truth), three work queues inside McpTaskService.run_once() (polling / cancellation / notification), the Gateway's idempotent run launcher, and the chat page's background-task panel. All three queues share one skeleton: claim (FOR UPDATE SKIP LOCKED + lease) → execute → release or commit. If any step crashes, the lease expires and the row returns to claimable state — there is no in-memory state to recover. That is the entire secret of restart self-healing.

Storage Layer: The Outbox Doesn't Need Its Own Table

The textbook outbox pattern is usually a separate table plus a relay process. This design is more aggressive: the outbox grows directly on the task row. Migration 0013_mcp_task_notifications adds 15 columns (12 for notification, 3 for cancellation) and 3 indexes to mcp_tasks:

# persistence/mcp_tasks/model.py (excerpt)
event_fingerprint: Mapped[str | None]        # sha256(event snapshot)
event_version: Mapped[int]                   # latest observed event version
notified_version: Mapped[int]                # last successfully delivered version
dispatch_version / dispatch_attempt / dispatch_event
notification_run_id / notification_error / notification_attempt_count
next_notification_at / notification_lease_owner / notification_lease_expires_at

"Is there an undelivered notification?" collapses into a single indexed comparison: event_version > notified_version. A version pair is a latch.

New events are born through fingerprint deduplication. Every polled remote snapshot only bumps the version if the sha256 of the event snapshot actually changed:

# persistence/mcp_tasks/sql.py (excerpt)
def _record_event_if_changed(row, *, tracking_degraded, now) -> bool:
    event = _notification_event(row, tracking_degraded=tracking_degraded)
    if event is None:
        return False
    fingerprint = _event_fingerprint(event)
    if fingerprint == row.event_fingerprint:
        return False                      # unchanged: not one notification fires
    row.event_fingerprint = fingerprint
    row.event_version = int(row.event_version or 0) + 1
    if row.notification_status not in _INFLIGHT_NOTIFICATION_STATUSES:
        row.notification_status = "pending"
        row.next_notification_at = now     # a dead-lettered channel revives on new events
    return True

Note the last branch: if the notification already went to dead letter, a new event pulls it back to pending and through the full pipeline again — dead letter kills a snapshot, not the task's notification channel. That detail is the heart of the "never lost" semantics: a poison snapshot cannot burn down the notifications that come after it.

Delivery Layer: A Notification Is an Idempotent Agent Run

This is the most counterintuitive decision in the PR, and the one most worth chewing on. The "delivery action" is not pushing a WebSocket message or firing a toast — it injects a hidden user turn into the task's original conversation thread, starts a full Agent run, and lets the LLM relay the event to the user:

# gateway/services.py (excerpt)
def _mcp_task_notification_prompt(event: dict[str, Any]) -> str:
    payload = frame_untrusted_text(json.dumps(event, sort_keys=True, ...))
    instruction = (
        "A durable background MCP task has an update that requires the user's "
        "attention. Explain the update clearly and concisely. Do not expose or "
        "ask for a remote task ID. ..."
    )
    return f"{instruction}\n\n{payload}"

idempotency_key = f"mcp-task:{task_id}:{dispatch_version}:{dispatch_attempt}"
record = await start_run(body, thread_id, request,
                         idempotency_key=idempotency_key,
                         require_existing_thread=True)

Why is this worth doing? Because the real audience of a notification is not the browser tab — it is the context of that conversation. A system toast reaches whoever is looking at a screen right now; a run injected into the original thread means the Agent itself knows the task finished. The next time the user asks "did that report ever get done?", the Agent isn't querying a database — it genuinely knows. The destination of a notification moves from "screen" to "memory", which kills the task-ID hallucination from #4652 outright: the ID no longer needs to live in the model's head; it lives in the database.

The cost is obvious: every notification burns one LLM call, so idempotency is the line between life and death. There are two layers:

1. State machine layer: the dispatched state records notification_run_id; subsequent passes poll that run's terminal status to decide delivered vs. retry;
2. Runs table layer: the migration adds a unique index on runs.idempotency_key — a duplicate launch of the same (task_id, version, attempt) (say, the worker crashed before marking) is caught by the database and returns the same run.

The State Machine: A Snapshot's Journey to Delivery

notification_status has seven states, and every edge is a lease-guarded conditional update (WHERE lease_owner = me AND lease not expired). A crash just expires the lease back to claimable:

Three exits deserve their own words:

Thread busy (409) → drop old, deliver latest. Notification runs launch with multitask_strategy="reject". A busy thread means 409 → ConflictError → wipe the stored snapshot, return to pending, and rebuild from the latest state on the next claim. Crucially, this path does not count as a failure — a busy thread is not the event's fault. For status notifications this is the correct coalescing semantics: intermediate states have no delivery value; the newest one suffices.

Thread gone (404) → permanent dead letter. require_existing_thread=True makes notification runs refuse to resurrect a deleted conversation (LookupError → 404 → PermanentNotificationError), going straight to dead letter without wasting a single retry. The distinction shows real craftsmanship: retryable and non-retryable failures are separated at the exception-type level (errors.py defines PermanentNotificationError specifically).

Five failed attempts → dead letter. The backoff formula is min(poll_interval × 2^min(failures,16), max_backoff) with _MAX_NOTIFICATION_ATTEMPTS = 5. Bounded retries are the final gate against notification storms — without it, a persistently failing delivery target turns your retry queue into a DoS source.

The Sequence: Happy Path and Its Three Exits

Assembled into one timeline (with the three exception exits in place):

Cancellation: Session Isolation and Fencing

Cancellation runs on its own queue symmetric to notification (claim_cancel_requests), with two race-hardened details:

# request_cancel: a cancel request "fences" in-flight poll results
row.cancel_requested_at = requested_at
row.lease_owner = None          # in-flight poll lease is voided immediately
row.lease_expires_at = None
# a repeated request must preserve the existing cancel lease
# so it cannot trigger a concurrent remote cancellation

Users get two cancellation entrances: the stop button in the frontend panel, and two built-in Agent tools, list_background_tasks / cancel_background_task. The latter means you can literally type "kill that task from earlier" into the chat, and the Agent matches by natural-language task name — the remote MCP task ID is never exposed to either side.

Security Boundary: Trust No External String

Remote MCP server output is untrusted input, and the PR defends at three independent points:

1. Before entering model context: the event JSON is wrapped by frame_untrusted_text() inside untrusted-boundary markers; forged tags like <system> get HTML-escaped (this is DeerFlow's existing prompt-injection defense middleware, renamed into a public primitive for reuse by this PR);
2. Before entering model state: when the worker projects the task list into graph input, task_name passes through neutralize_untrusted_tags(), and only 4 whitelisted fields make it in, capped at 20 rows;
3. Before leaving the API: _public_task() builds an outbound shape of exactly 7 whitelisted fields — the tests deliberately plant a "must-not-leak" remote_task_id and driver_data.secret, asserting they appear in no output.

Whitelisting is the only correct direction: a blacklist can never enumerate everything a JSON blob can hide.

Frontend: One Adaptive-Polling Panel

The chat page gains a Sheet panel (active/recent sections, expandable details, one-click cancel) backed by react-query:

// frontend/src/core/background-tasks/hooks.ts (excerpt)
refetchInterval: (query) =>
  query.state.data?.some(isActiveBackgroundTask) ? 3000 : 15000,
refetchIntervalInBackground: false,

3-second refresh while tasks are active, dropping to 15 seconds otherwise, and nothing while the tab is hidden. Combined with the Agent-side notification runs putting results into the conversation stream, users don't miss terminal states even if they never open the panel. The validation volume in the PR description deserves a mention too: backend make test passed 11,653 tests; the frontend ran 994 tests plus a production-build E2E.

This Design vs. The Conventional Approaches

Dimension	WebSocket/SSE push	Client polling only	deer-flow: row outbox + Agent run
After server restart	Connections break, events lost	Not lost but laggy	Lease expiry auto-resumes delivery
Duplicate delivery	In-memory dedup, weak	Naturally idempotent	Version pair + idempotency key, double-locked
Who receives	The browser tab	The browser tab	The conversation context itself (the Agent knows)
Poison messages	Retry forever or drop	No retries	Dead letter after 5; new events still deliver
Intermediate-state coalescing	None	None	409 auto-drops old for latest
Cost per notification	One push	N empty polls	One LLM call
External dependencies	Needs a connection layer	None	None (pure PG)

My call: if your notification audience is "the human watching the screen", push plus reconnect-replay is enough; if the audience is "the Agent that keeps the conversation going", this design is the right one. It is expensive — one LLM call per notification — but what it buys back is the unification of notification and context. That trade will only get more mainstream in Agent products.

Design Scorecard

Dimension	Score	Reasoning
Crash safety	9/10	Leases end-to-end + idempotency keys, no in-memory state; −1 for polling-granularity run-status confirmation
Semantic correctness	9/10	Version-pair latching, fingerprint dedup, dead-letter-doesn't-kill-channel, 409 coalescing — all four details land
Security	9/10	Three independent whitelist/escape points + leak-test assertions; boundaries well thought out
Performance/cost	6/10	One LLM call per notification; task projection occupies every run's context (capped at 20)
Portability	8/10	Zero new dependencies, pure-PG schema, liftable into any PG-backed system; but "LLM as renderer" assumes an Agent product shape

Reservations

Honestly, a few things I would not take at face value:

1. input_required is display-only. When a task enters "needs user input", the notification tells you what it's asking — but this MVP cannot send the answer back (the PR explicitly defers it to future driver capability). Half the interaction loop is missing.
2. Dead letter is silent for users. After 5 failures a notification enters dead_letter, discoverable only by opening the panel. For high-value events like "result completed", an admin-visible alert path would be worth adding.
3. Delivery latency is bounded by poll granularity. In dispatched state, run terminal status is only confirmed on the next run_once pass, so latency = poll interval. Irrelevant for minutes-scale tasks; needs tuning for second-sensitive scenarios.
4. Every run carries the task projection. background_tasks (capped at 20) enters every run's graph input as permanently paid context — threads with few tasks still pay. On-demand injection would be cheaper.

Where This Sits in the Trend

This is not an isolated artifact — it is the concrete instantiation of the "reliability infrastructure" layer from harness engineering: as Agent tool calls evolve from synchronous RPC into minutes-and-hours persistent tasks, the harness must grow a task substrate — submit, recover, notify, cancel — none of which can rely on model conscientiousness. One level up, the autonomous orchestration that loop engineering argues for only dares to let Agents run long loops because this durable substrate exists: the loop may break; the state will not be lost. With MCP's SEP-2663 Tasks extension making "task" a protocol-level concept, whoever's harness handles persistent tasks most invisibly has the most usable long-horizon Agent.

Conclusion: What to Steal

If you maintain any product with minutes-scale workflows (video generation, batch analytics, scheduled jobs), four things in this PR are worth copying wholesale, in priority order:

1. The two version numbers (event_version / notified_version) — "is there an undelivered event?" answered by one column comparison, cheaper than any outbox table;
2. Fingerprint deduplication — if nothing changed, nothing notifies; all perceived "non-noisiness" rests on this;
3. Bounded retries + dead-letter-doesn't-kill-the-channel — the 5-attempt cap prevents storms; new-event revival prevents loss;
4. Retryable and non-retryable failures get different exception types — 404 goes straight to dead letter, 409 doesn't count as failure. That classification is worth more than any retry parameter.

As for "notification = Agent run": first ask who your notification audience is. Human? Use push. Agent? This is the most complete public implementation to date.

References

1. AnnaSuSu — feat(mcp): complete durable task notifications and chat UI (PR #4833) (merged 2026-08-22, commit 5ffc2d3)
2. deer-flow maintainers — [feat] Add support for the MCP Tasks extension protocol SEP-2663 (Issue #4652) (filed 2026-08-03)
3. — feat(mcp): add durable task runtime foundation (PR #4665) (merged 2026-08-08)
4. — feat(mcp): add ordinary durable task driver (PR #4690) (merged 2026-08-15)
5. bytedance — deer-flow repository (as of 2026-08-23: 80,572 Stars / 11,065 Forks)
6. modelcontextprotocol — Tasks Extension (SEP-2663) proposal
7. This site — DeerFlow 2.0 Deep Dive (2026-05-07)