One Brain for Three AI Agents: Building a Shared Knowledge Layer with TencentDB Agent Memory

Anyone working with multiple AI agents hits the same wall eventually: Hermes knows things Claude Code (CC) doesn't. Architecture decisions you explained to Codex last week are gone this week. Every agent keeps its own memory files, its own skill directory, its own disconnected context. You become the human relay between three agents — re-explaining background, re-pasting documents, re-answering "what did we decide last time."

Starting 2026-08-24, I self-hosted TencentDB Agent Memory (often abbreviated TDAI Agent Memory; TDAI hereafter) as a shared memory stack so all three agents talk to one memory service. The next morning (08-25), in a single session, I wired in all three pieces of the knowledge layer: an Obsidian vault wiki, a Hermes skill-library wiki, and code graphs for ten repositories. This post is the complete build log — the data, the pitfalls, the fixes, and an honest review of both the good and the bad.

What TDAI Is: Four Assets, One Hub

TDAI is a team-level agent memory hub open-sourced by Tencent Cloud's database team (as of 2026-08-25: ~24,300 GitHub stars / 2,240 forks, MIT license, repository created April 2026). It turns agent "experience" into four governed asset types:

Asset	Stores	Traditional analog
Chat Memory	Conversations, facts, scenes, persona	Long-term memory
Skill	Operating manuals, procedural knowledge	Procedural memory
LLM-Wiki	LLM-rewritten docs and knowledge	Declarative knowledge
Code-Graph	Structured graphs of code repositories	Spatial/structural memory

The architecture is four services: MemoryCore (memory kernel), MemoryKnowledge (knowledge subsystem), MemoryProxy (gateway), and MemoryPanel (admin panel). Integration is refreshingly restrained — no protocol changes, no plugins; point the agent's base URL at the Proxy and you're done. The official adapter list includes DeepSeek Harness, Claude Code, Codex, CodeBuddy, WorkBuddy, Hermes, and OpenClaw. My setup: Hermes uses native memory tools; CC and Codex use their official adapters — all sharing one team memory.

One detail worth noting: the README acknowledgments state that its Skill asset management reuses part of the Skill-related code from Hermes Agent, and the LLM-Wiki layer design is directly inspired by Karpathy's LLM Wiki — documentation as "an LLM-maintained, incrementally growing knowledge artifact." The official PersonaMem benchmark reports an improvement from 48% to 76% (a 59% relative gain). That's a vendor-run benchmark — take it as directional, but the direction is right: memory lives or dies on recall, not storage.

Chat Memory internally is a four-layer structure: L0 raw conversation → L1 atomic facts → L2 scene blocks → L3 user persona, with automatic distillation from dialogue to profile. Wiki, Code-Graph, and Skill belong to the Knowledge subsystem — a separate data pathway from the four-layer memory. That isolation is exactly why importing a skill library later couldn't pollute the memory store.

Build Log 1: Full Vault Ingest — Switching Models Took the Failure Rate from 37% to 0%

The first piece was my Obsidian vault — a personal knowledge base of 418 source files covering project research, architecture decisions, meeting notes, and reading notes. The pipeline is one-way sync: the vault is the single source of truth, TDAI is a read-only searchable replica, sha256-based incremental checks, a cron job every 6 hours.

The first full ingest proved a point. On night one, with glm-4-flash as the extraction model, 12 of 32 files failed — a 37% failure rate, all protocol-compliance failures producing half-finished JSON of 500-1,000 tokens. After switching to glm-5.3 and re-running (checkpoint at 09:49 on 2026-08-25):

Metric	glm-4-flash	glm-5.3
Failure rate	12/32 (37%)	0/315 (0%)
Time per file	60-140s	~50s
Output density	500-1,000 tokens	3,000-5,500 tokens

Same pipeline, different model — the protocol-compliance problem vanished entirely. The takeaway for anyone running batch LLM extraction: the extraction model's quality sets the throughput floor of your memory pipeline, and a cheap model's structured-output failure costs will eat every cent it saved. After all 418 files finished, the pipeline automatically entered the merging/indexing phase to build the BM25 index; from then on, any connected agent could hit vault knowledge by asking.

Build Log 2: Skills Become a Wiki — Filling the "Procedural Memory" Gap

The second piece is the most interesting one. My Hermes skill library holds 204 SKILL.md files (762 .md files, 6.6MB in total) — operating manuals distilled from hundreds of tasks: how to deploy, how to audit, how to work around known traps. The problem: that procedural knowledge lives only on the Hermes side. CC and Codex can't see any of it.

The four-layer memory can't fix this: L0-L3 store facts (what happened, who you are), not procedures (how to do this thing). A skill library is exactly the carrier of procedural knowledge — so syncing 204 skills into a dedicated TDAI wiki ("hermes-skills") gives all three agents a shared procedural memory.

Two design decisions mattered:

A separate wiki, never mixed with the vault. My vault already contains skill-related material (skill audit reports, architecture analyses). If SKILL.md originals went into the same wiki as my own notes, BM25 retrieval would pit same-topic documents against each other — searching "SOLID" would return both the skill's source text and my audit report, and since wiki pages are LLM-rewritten, comparing two versions is actively misleading. Separate wikis, one query each — isolation for free.

SKILL.md only, no references/. At ~70s per file, a full 762-file ingest would take 14+ hours; syncing just 204 SKILL.md files takes ~4 hours and finishes overnight. references/ files are mostly one-off research documents. Start with the main files; if retrieval reveals gaps, backfill incrementally — classic "make it work, then make it complete."

The sync rule matches the vault: the local skill directory (git-mirrored) is the single source of truth, TDAI is read-only, never written back. One-way pipeline plus sha256 increments — no bidirectional sync conflicts possible.

Build Log 3: Ten Code Graphs — Turning Cold-Start Cost into a Save File

The third piece is Code-Graph. TDAI's framing is exactly right: most agents' first task is re-learning your project — a cost you've already paid once. Code-Graph turns that tuition into a save file: agents discover capabilities via /v3/tools/list and read relevant pages, source code, or impact paths via /v3/tools/call. Graph data only enters context when actually needed.

Graphs built on the morning of 2026-08-25 (all ready, query-verified):

Repository	Scale
TencentDB-Agent-Memory upstream	745 files / 13,804 nodes
task-vault	98 files / 1,555 nodes
desktop_portal	152 files / 1,481 nodes
shujietai	112 files / 1,640 nodes
cdut_stu_agents	127 files / 1,622 nodes
edu-agent	68 files / 878 nodes
hermes-voicedesk	31 files / 338 nodes
django_jwt_test_project	20 files / 102 nodes
tencentDDNS	8 files / 39 nodes

The live test says the most: exploring "错题本" (mistake notebook) in the edu-agent graph precisely hits removeMistake in the frontend MistakesView.vue and MistakeOut in the backend mistakes.py, complete with caller analysis. Any connected agent asked "how does the mistake notebook work" no longer needs to re-read the codebase — the same context-efficiency problem discussed in the Hermes recall mechanism deep-dive, solved for the code domain.

Two honestly recorded boundaries: my private repository (guancyxx/skills) can't be graphed — Code-Graph currently supports public HTTPS repositories only, and the official README explicitly says private repo and SSH credential support is "still being refined." That's an upstream feature boundary, not a deployment bug. And 14 third-party fork repos were left out — third-party code is not my memory asset.

At this point the knowledge layer is complete: vault wiki + skill wiki + code graphs, stacked on the existing four-layer memory. Three agents, one brain.

Pitfall Log: Three Traps You Only Find by Doing

This is the part the docs won't tell you.

Trap 1: A 413 misreported as "kernel service unavailable." Importing a large skill through the panel threw KERNEL_UNAVAILABLE. After much digging: the kernel was never down — the real cause was the skill package exceeding the kernel gateway's default 1MiB request-body limit, and the panel misreported the 413 as "kernel unavailable." Fix: inject MEMORY_MAX_BODY_BYTES=50MB into start-memory-core.sh and restart the container. Lesson: panel error messages are not trustworthy; check the kernel gateway logs first.

Trap 2: A hard 50,000-character body limit per file. That skill's SKILL.md body was ~78k characters, exceeding BODY_MAX = 50_000 — a hardcoded constant in the kernel, not configurable. The solution is splitting: the main file keeps the first 48,939 characters (compliant); the overflowing 29,931 characters become a resource file SKILL-PART2.md stored alongside, with a pointer added at the end of the main file. All 50 references/*.md (~294k characters) were imported as resources; total package 358KB, zero content loss. Verification meant reading content back: spot-checked resource files matched the local copies character for character — whether an import succeeded is judged by what you can read back, not by an API returning code:0.

Trap 3: Re-importing the same name does not overwrite. Re-importing a same-named skill through the panel makes the kernel create a duplicate rather than overwrite. The only fix is delete-old-then-import-new; "overwrite-style update" doesn't exist.

Verdict: What's Good, What's Not

After one morning plus one night of work, my judgment:

Credit where due:

• Restrained integration design. Point a base URL and it works — no protocol changes, no plugins. Much cleaner than a pile of MCP servers.
• Asset isolation done right. The Knowledge subsystem and four-layer memory don't pollute each other. My biggest worry when importing the skill library — retrieval contamination — was solved simply by separate wikis.
• Code-Graph's on-demand-context design is correct: the graph is a tool, not background noise injected every turn.
• For the multi-agent individual developer, this is one of the few open-source options that manages all memory forms in one hub.

Complaints:

• No Code-Graph support for private repositories — a hard limitation for anyone keeping code private (upstream is working on it).
• BODY_MAX hardcoded at 50k with no config; long documents must be split by hand. Primitive.
• Misleading panel error text (413 reported as kernel unavailable) multiplied debugging cost for no reason.
• Ingest at ~50-70s per file means first-round ingestion of libraries in the thousands of files must be planned in days.
• Same-name imports duplicate instead of overwriting — asset-management burden lands on the user.

The project is labeled Beta; half the traps above are feature boundaries and half are engineering maturity. I endorse the direction.

Conclusion

Three agents, four memory layers, three knowledge assets — all pointing at one brain. The most valuable output of this build isn't the tool itself but three reusable judgments:

1. Govern memory by form. Facts (L0-L3), procedures (Skill), knowledge (Wiki), and structure (Code-Graph) are four different things. Pool them together and retrieval pollution is guaranteed; separate assets and separate wikis is the answer.
2. The source of truth stays local; TDAI is read-only. Obsidian and the git-mirrored skill library are the only authoritative sources; the cloud side is a searchable replica. One-way sync keeps the whole system rebuildable — the same principle I held in Task Vault: durable state must come with rebuildability.
3. Don't cheap out on the extraction model. The 37%-to-0% failure-rate gap means everything glm-4-flash saved on tokens was lost to retries and babysitting.

If you're the human relay in a multi-agent workflow, this stack deserves a morning of your time. As for where the Agent Skills ecosystem heads and whether memory becomes the next standard component of agents — TDAI's 24k stars at least prove the demand is real and widespread.

References

1. TencentCloud — TencentDB-Agent-Memory GitHub repository (~24,300 stars as of 2026-08-25, MIT, Beta)
2. Tencent Cloud Developer Community — TencentDB Agent Memory is now open source (2026-05-14)
3. bowen-upenn — PersonaMem benchmark
4. Andrej Karpathy — LLM Wiki
5. Nous Research — Hermes Agent (parts of its Skill code reused by TDAI)
6. This site — Hermes Recall Mechanism Deep-Dive
7. This site — Task Vault: Managing AI Agent Tasks with an Obsidian Plugin
8. This site — Agent Skills Ecosystem Observations