[{"content":"Memory is a foundational component of LLM agents, but the industry is stuck on an isolated paradigm: an agent remembers only its own experience. The MemRec paper (ACL 2026) introduces the next milestone — collaborative memory, where agent memories are linked in a graph and exchange relational signals. I\u0026rsquo;ll break down MemRec to the level of equations and prompts, compare three production tools (Mem0, Letta, Zep), and show where research outpaces tooling.\n1. Why Isolated Memory Is a Dead End Why does an agent need memory at all? Context windows are growing — GPT-4o handles 128K tokens, Gemini a million. But a context window is not memory. It\u0026rsquo;s a workbench. Put every conversation ever seen, every user preference, every tool output on the workbench — and the workbench breaks. The LLM starts losing relevant facts in the noise, as shown in Lost in the Middle (Liu et al., 2024): performance degrades when the needed information sits in the middle of a long context.\nMemory solves a different problem — it makes the agent stateful (state persists between runs), allows accumulating experience and adapting. Without memory, an agent is a pure function: same inputs, same outputs. With memory, the agent evolves. This principle is well articulated in Generative Agents (Park et al., 2023), where a simulation of virtual town residents showed that the memory stream is precisely what makes agent behavior believable.\nThe evolution of agent memory has passed three milestones. Here\u0026rsquo;s the map:\ntimeline title Evolution of LLM Agent Memory Paradigms section No Memory 2023 : Vanilla LLM : Relies on inherent knowledgeevery run from scratch section Static Memory 2024 : iAgent, Chat-Rec : Retrieval from fixedstorage, no updates section Dynamic Memory 2025 : AgentCF, i2Agent : Agent iteratively updatesits understanding section Collaborative Memory 2026 : MemRec (ACL 2026) : Memory linked in a graphcollaborative signals Each milestone solved a concrete problem, but all three kept memory isolated. What does this mean in practice? Let\u0026rsquo;s look at recommender systems — where this problem is most clearly expressed, and where MemRec demonstrates a solution that generalizes to arbitrary agents.\nIn agentic recommender systems (RS), an agent stores a memory about user M_u — a narrative of past interactions, preferences. And memory about an item M_i — a semantic description. When the agent recommends a book to a user, it sees only that user\u0026rsquo;s M_u . But what if another user with similar tastes just rated a book our user hasn\u0026rsquo;t seen yet? The isolated agent doesn\u0026rsquo;t know — the collaborative signal (relational information from the user community) is cut off.\nThis isn\u0026rsquo;t specific to recommender systems. Imagine a team of agents working on code: one agent found a solution to a dependency problem, another encounters the same problem — but doesn\u0026rsquo;t know, because memory is isolated. Or a support agent: a user described a problem to agent A, then contacts agent B — and B starts from scratch. Memory isolation is a loss of collective intelligence.\n2. MemRec: Two Architectural Shifts The MemRec paper (Chen et al., ACL 2026) solves the isolation problem through two architectural shifts. But before unpacking them, we need to understand why the naive solution doesn\u0026rsquo;t work.\n2.1 Naive Brute-Force and Why It Fails The idea of \u0026ldquo;just give the agent access to all neighbors\u0026rsquo; memories\u0026rdquo; comes first. If a user interacted with books alongside a thousand other users — feed the agent all thousand memories. The problem is that this causes two catastrophes.\nCognitive overload. The LLM can\u0026rsquo;t distill signal from abundance. Lost in the Middle showed that as context grows, the LLM loses its ability to find relevant facts. Load the memories of dozens of neighbors into context — and the agent starts hallucinating, losing instruction adherence, confusing whose preference is whose.\nProhibitive collaborative updates. Collaborative memory must evolve. When a user interacts with an item, the knowledge should propagate to all connected neighbors. The naive approach requires a separate LLM call for each neighbor — updating one user\u0026rsquo;s memory means invoking the LLM dozens of times. In real-time serving, this is prohibitive.\nHere\u0026rsquo;s how naive and MemRec approach the same task:\ngraph LR subgraph \"Naive: single pass, monolithic\" RAW[\"Raw graph context1000 neighborsverbose memories\"] MONO[\"Single LLM(does everything)\"] RES1[\"Result:cognitive overload,hallucinations\"] RAW --\u003e MONO --\u003e RES1 end subgraph \"MemRec: two passes, decoupled\" GRAPH[\"Raw graphG=(V,E)\"] CUR[\"Curate(LLM rules filter)\"] SYN[\"Synthesize(LM_Mem distill)\"] FACETS[\"M_collab7 structured facets\"] REC[\"LLM_Rec(reasoning only)\"] RES2[\"Result:high-signal context,grounded rationales\"] GRAPH --\u003e CUR --\u003e SYN --\u003e FACETS --\u003e REC --\u003e RES2 end style MONO fill:#c62828,color:#fff style RES1 fill:#c62828,color:#fff style CUR fill:#2e7d32,color:#fff style SYN fill:#2e7d32,color:#fff style REC fill:#1565c0,color:#fff style RES2 fill:#2e7d32,color:#fff The naive approach feeds the entire raw context to one model — and hits an information bottleneck: the model can\u0026rsquo;t simultaneously ingest and reason. MemRec splits the task: curate (filter noise via domain-adaptive rules), synthesize (distill into structured facets), then reason on clean context. This is \u0026ldquo;Curate-then-Synthesize\u0026rdquo; — two compression passes guided by the Information Bottleneck principle.\nMemRec solves both problems through two architectural shifts.\n2.2 Shift #1: Decoupling Memory Management from Reasoning MemRec separates two functions that naive agents perform in a single model:\nLM_Mem (lightweight language model) — manages memory in the background: curates the graph, synthesizes context, updates neighbors. LLM_Rec (heavyweight large language model) — performs final reasoning: ranks candidates, generates justifications. Why not mix these in one heavy model? The Information Bottleneck (IB, a compression principle that preserves maximum task-relevant information while minimizing redundant signals; Tishby et al.) is at work. LM_Mem\u0026rsquo;s job is to compress raw graph context into a compact representation that preserves maximum useful signal for reasoning and discards noise. If the same model must both compress and reason, an information bottleneck arises: it can\u0026rsquo;t simultaneously absorb verbose context and perform complex ranking. The experiment confirms this: a naive monolithic agent (one model does everything) plateaus, while MemRec with decoupling delivers +34% relative H@1 gain on the Books dataset (see Section 5, RQ2).\nThis is an engineering principle that extends beyond recommender systems. Don\u0026rsquo;t make the expensive reasoning model do library work. Split it: a lightweight model gathers context and distills, a heavyweight model reasons. The cost effect is also significant — on expensive output tokens (which are 3-4x more costly than input), MemRec spends minimally: Stage-R and Stage-W are heavily input-biased (input is ~80% of total usage), which radically reduces effective cost.\n2.3 Shift #2: Collaborative Memory Graph MemRec doesn\u0026rsquo;t just give the agent access to neighbors\u0026rsquo; memories. It builds a memory graph G = (\\mathcal{V}, E) . Nodes \\mathcal{V} = \\mathcal{U} \\cup \\mathcal{I} are users and items, each node v stores an evolving semantic memory M_v . Edges E encode interactions and derived relations.\ngraph LR subgraph \"Isolated Memory\" U1[\"User AM_A\"] U2[\"User BM_B\"] I1[\"Item XM_X\"] I2[\"Item YM_Y\"] end subgraph \"Collaborative Memory Graph G=(V,E)\" UA[\"User AM_A\"] UB[\"User BM_B\"] UC[\"User CM_C\"] IX[\"Item XM_X\"] IY[\"Item YM_Y\"] IZ[\"Item ZM_Z\"] UA -.-\u003e|\"co-engagement\"| IX UB -.-\u003e|\"co-engagement\"| IX UA -.-\u003e|\"peer similarity\"| UB IX -.-\u003e|\"related\"| IY UC -.-\u003e|\"co-engagement\"| IY UB -.-\u003e|\"co-engagement\"| IZ end style UA fill:#1565c0,color:#fff style UB fill:#1565c0,color:#fff style UC fill:#1565c0,color:#fff style IX fill:#c62828,color:#fff style IY fill:#c62828,color:#fff style IZ fill:#c62828,color:#fff The difference is fundamental. The isolated paradigm is a set of disconnected narratives \\{M_u\\} \\cup \\{M_i\\} . The collaborative paradigm is a graph with high-order connectivity: a signal can travel from user to user through shared items, from item to item through co-engagement. For data-sparse users (little history), this is critical — the collaborative graph compensates for the deficit of personal history with signal from similar peers. And MemRec confirms: +91.4% relative H@1 gain for low-activity users over Vanilla LLM.\n3. Deep Dive into MemRec: The Three-Stage Pipeline Now let\u0026rsquo;s break down MemRec\u0026rsquo;s architecture in detail — equations, prompts, engineering trade-offs. This is the unpacking of the paper, as is.\nMemRec operates in three stages. Here\u0026rsquo;s the full pipeline, then each stage separately.\ngraph TB subgraph \"Background (LM_Mem, lightweight)\" G[\"Memory GraphG = (V, E)\"] C[\"Stage 1:Collaborative MemoryRetrieval\"] P[\"Stage 3:Async CollaborativePropagation\"] end subgraph \"Foreground (LLM_Rec, heavyweight)\" R[\"Stage 2:Grounded Reasoning\"] end G --\u003e|\"raw neighbormemories\"| C C --\u003e|\"M_collabdistilled facets\"| R R --\u003e|\"scores s_i,rationales r_i\"| OUT[\"Rankedrecommendations\"] R -.-\u003e|\"new interaction\"| P P -.-\u003e|\"ΔM updatesto neighbors\"| G style C fill:#2e7d32,color:#fff style P fill:#2e7d32,color:#fff style R fill:#c62828,color:#fff style G fill:#1565c0,color:#fff Green — LM_Mem working in the background, lightweight model. Red — LLM_Rec working in the foreground, heavyweight model, only when reasoning is needed. Blue — memory graph, persistent state. Note: propagation (Stage 3) is async, it doesn\u0026rsquo;t block foreground reasoning.\n3.1 Stage 1: Collaborative Memory Retrieval The goal of the first stage is to take the expansive memory graph and extract a concise collaborative memory M_\\text{collab} for the current task. The challenge: don\u0026rsquo;t overload the reasoning agent. The strategy is \u0026ldquo;Curate-then-Synthesize\u0026rdquo; (curate, then synthesize), two compression passes following the IB principle.\n3.1.1 LLM-as-Rule-Generator: Offline Curation The first pass is curate. The problem: how to select the top- k most relevant neighbors from the graph? Traditional approaches — rule-based heuristics (random walk, DeepWalk) or learned neural scorers (GNN attention). Both are bad for LLM agents: heuristics don\u0026rsquo;t adapt to domain semantics, learned scorers require expensive training.\nMemRec proposes a zero-shot paradigm: LLM-as-Rule-Generator. In the offline phase, LM_Mem analyzes domain statistics \\mathcal{D}_\\text{domain} and generates interpretable heuristic rules R_\\text{domain} :\nR_\\text{domain} \\leftarrow \\text{LM}_\\text{Mem}(\\mathcal{D}_\\text{domain} \\| P_\\text{meta}) \\quad \\text{(offline)} P_\\text{meta} is the meta-prompt that guides rule generation. Here it is (source: MemRec, Appendix F.1):\nMeta-Prompt Template for Rule Generation You are an expert AI engineer specializing in recommender systems and graph-based memory networks. Your task is to generate a set of domain-specific heuristic rules for a collaborative neighbor pruning algorithm. The goal is to select the top-k most relevant neighbors (users or items) from a candidate graph to build a compact, high-signal context for a downstream LLM recommender (MemRec). DOMAIN CONTEXT • Domain Name: {Domain Name} • Primary Interaction: {Primary Interaction with example} • Key Metadata: {Key Metadata} • Available Features: – edge_weight: {Domain-specific explanation} – recency_days: {Domain-specific explanation} – co_interaction_count: {Domain-specific explanation} – metadata_overlap_score: {Domain-specific explanation} – memory_similarity_score: {Domain-specific explanation} INSTRUCTIONS: 1. Based only on the domain context provided, generate 3-5 high-priority, interpretable ranking rules. 2. The rules should explain how to combine or prioritize the available features to find the best neighbors for this specific domain. 3. Be specific about thresholds and weights. 4. Consider domain-specific characteristics (e.g., books are content-driven with long-term preferences; movies are recency-critical). OUTPUT FORMAT: Rule 1: [Your rule here] Rule 2: [Your rule here] ... What matters: the rules are domain-adaptive. For Books (content-driven), LM_Mem generates a boost for metadata_overlap (\u0026gt;0.6 → 2.5x multiplier) — books are read by genre/author. For MovieTV (recency-critical) — strong recency decay (exp(-0.018 × recency_days)). For Yelp (category-driven) — categorical dominance (metadata_overlap \u0026gt; 0.7 → 3.5x). This is zero-shot: no training, just the LLM\u0026rsquo;s semantic understanding.\nAt inference time, the rules act as a high-speed filter — selecting top- k neighbors in milliseconds:\nN'_k(u) = \\text{Curate}(N(u), R_\\text{domain}, k) Quantitative validation: LLM curation reduces irrelevant item neighbors by 73.8% versus generic heuristic, while retaining 6.4% more user neighbors with low ID-overlap — because the LLM finds semantic similarity in memory narratives (two users who both enjoy \u0026ldquo;dystopian YA novels\u0026rdquo; without having clicked the same items).\n3.1.2 Collaborative Memory Synthesis: Distill into Facets The second pass is synthesize. The selected neighbors N'_k are still verbose. LM_Mem distills them into structured preference facets \\{F\\} — this is M_\\text{collab} :\nM_\\text{collab} = \\{F\\} \\leftarrow \\text{LM}_\\text{Mem}(\\text{Rep}(N'_k) \\| M_u^{t-1} \\| P_\\text{synth}) \\text{Rep}(N'_k) is a tiered representation: the target user u is represented by full memory M_u^{t-1} , neighbors by compact contextual representations (condensed signals, not verbose histories). P_\\text{synth} is the synthesis prompt (source: MemRec, Appendix F.3):\nStage-R Prompt: Collaborative Memory Synthesis You are an intelligent memory retrieval system for personalized recommendation. Your task is to analyze the user\u0026#39;s personal memory and collaborative memories from their neighbors to extract preference facets. Target User: User {user_id} User\u0026#39;s Personal Memory: User Memory Summary: {user_memory_summary} Collaborative Neighbor Memories: The following neighboring users and items provide collaborative signals: Collaborative Neighbors: {formatted_neighbor_list} Your Task: Analyze the user\u0026#39;s personal memory and the collaborative memories to identify {n_facets} distinct preference facets. For each preference facet, provide: 1. A concise natural language description of the preference 2. A confidence score between 0 and 1 3. A list of supporting neighbors (user IDs or item IDs) Output: JSON with \u0026#34;facets\u0026#34; array and \u0026#34;support_edges\u0026#34; array. The result — instead of \u0026ldquo;User prefers dystopian settings\u0026rdquo; (isolated), we get:\nTheme: Cyberpunk \u0026amp; Corporate Dystopia (Conf: 0.9)\nEvidence: User Neighbor (ID: 2057) shows deep interest in corporate control; Item Neighbor (\u0026lsquo;1984\u0026rsquo;) shares foundational dystopian themes.\nTheme: High-Stakes Survival (Conf: 0.75)\nEvidence: Item Neighbor (\u0026lsquo;Battle Royale\u0026rsquo;) exhibits strong survival elements matching recent interactions.\nEach facet is grounded in specific neighbors. This is the distilled high-signal context that goes to LLM_Rec.\n3.2 Stage 2: Grounded Reasoning The second stage is final reasoning. LLM_Rec receives the synthesized M_\\text{collab} , the user\u0026rsquo;s instruction \\mathcal{I}_u , and candidate memories C_\\text{info} :\n\\{s_i, r_i\\}_{i=1}^{N} \\leftarrow \\text{LLM}_\\text{Rec}(\\mathcal{I}_u \\| M_\\text{collab} \\| C_\\text{info} \\| P_\\text{rerank}) For each candidate, LLM_Rec generates a relevance score s_i (0-1) and a natural language rationale r_i . Grounding in M_\\text{collab} ensures the rationale is supported by community evidence, not fabricated. The rerank prompt (source: MemRec, Appendix F.4):\nStage-ReRank Prompt (MemRec Mode) You are an intelligent recommendation scoring system. Your task is to evaluate how well each candidate item matches the target user\u0026#39;s preferences based on their personal memory and collaborative signals. Target User: User {user_id} User\u0026#39;s Current Request: {instruction} User Preferences (Extracted from Collaborative Memories): {formatted_facets} Candidate Item Memories: {formatted_item_memories} Your Task: For each candidate item, provide a relevance score between 0 and 1: • 1.0 = Excellent match, highly aligned with user\u0026#39;s facets • 0.5 = Moderate match, partially relevant • 0.0 = Poor match, not aligned For each item, provide a brief rationale explaining your scoring. Output: JSON with \u0026#34;scores\u0026#34; array of {item_id, score, rationale}. And here\u0026rsquo;s the interesting part. LLM_Rec doesn\u0026rsquo;t see the raw graph. It sees only distilled facets with confidence scores and grounding evidence. Cognitive overload is impossible by construction — the context is already curated and synthesized at Stage 1.\n3.3 Stage 3: Async Collaborative Propagation The third stage is graph evolution. When user u interacts with item i_c , memory must update: the user learned something new, the item got a new signal, and — crucially — connected neighbors must receive propagation (knowledge spread).\nThe problem: the naive synchronous approach invokes the LLM for each neighbor separately — O(|N'_k|) calls per interaction, repeating user context in each. MemRec achieves O(1) call complexity.\nHow? Inspired by Label Propagation (Zhu \u0026amp; Ghahramani, 2002) — an algorithm where labels spread across the graph from node to node. MemRec propagates \u0026ldquo;semantic labels\u0026rdquo; (insights, conclusions) across the memory graph. But instead of separate calls for each node, MemRec conceptually decomposes the update:\nM_u^t, M_{i_c}^t \\leftarrow \\text{LM}_\\text{Mem}(M_\\text{collab} \\| M_u^{t-1} \\| M_{i_c}^{t-1} \\| P_\\text{update}) \\quad \\text{(self-reflection)} \\{\\Delta M_\\text{neigh}\\} \\leftarrow \\text{LM}_\\text{Mem}(M_\\text{collab} \\| M_u^{t-1} \\| M_{i_c}^{t-1} \\| N'_k(u) \\| P_\\text{update}) \\quad \\text{(neighbor propagation)} And executes both steps in a single batched async call with a unified prompt P_\\text{update} (source: MemRec, Appendix F.5):\nStage-W Prompt: Asynchronous Collaborative Propagation You are an intelligent memory management system for collaborative recommendation. Your task is to update the personal memories of the user, the clicked item, and relevant collaborative neighbors based on this new interaction. Interaction Context: User {user_id} has just interacted with (clicked) Item {item_id}. User Preferences (Extracted from Collaborative Memories): {formatted_facets} Current Personal Memory of User {user_id}: {current_user_memory} Current Memory of Item {item_id}: {current_item_memory} Collaborative Neighbors Available for Memory Propagation: {n_neighbors} neighbors: {formatted_neighbors} Your Task — Generate UPDATED memories for: 1. The current user (synthesize current memory + facets + clicked item) 2. The clicked item (describe what it is and who might enjoy it) 3. Selected neighbors (collaborative propagation is key!) * Select neighbors RELEVANT to this interaction * Update their memories to reflect new insights * This helps the system learn collaboratively! Output: JSON with \u0026#34;user_memory\u0026#34;, \u0026#34;item_memory\u0026#34;, \u0026#34;neighbor_updates\u0026#34;. A single LM_Mem call updates the user, item, and selected neighbors simultaneously. Async — doesn\u0026rsquo;t block foreground reasoning. O(1) call complexity. This resolves the prohibitive updates bottleneck.\ngraph TB subgraph \"Naive: O(|N_k'|) calls\" N1[\"User updatecall 1\"] N2[\"Item updatecall 2\"] N3[\"Neighbor 1call 3\"] N4[\"Neighbor 2call 4\"] N5[\"Neighbor kcall k+2\"] N1 --\u003e N2 --\u003e N3 --\u003e N4 --\u003e N5 end subgraph \"MemRec: O(1) batched async\" B[\"Single LM_Mem callP_update\"] B --\u003e|\"user_memory\"| UU[\"User M_u^t\"] B --\u003e|\"item_memory\"| II[\"Item M_i^t\"] B --\u003e|\"neighbor_updates\"| NN[\"ΔM_neigh(selected neighbors)\"] end style B fill:#2e7d32,color:#fff Naive approach: linear number of calls, each repeating user context. MemRec: one batched call, everything in a single prompt. The latency and cost difference is orders of magnitude.\n4. MemRec Experiments: What the Numbers Show MemRec is evaluated on four benchmark datasets: Amazon Books, Amazon Goodreads, MovieTV, Yelp. Different interaction densities, different domain characteristics. Metrics — Hit Rate (H@K) and NDCG (N@K) for K ∈ {1, 3, 5}. Implementation: gpt-4o-mini for both LM_Mem and LLM_Rec, k=16 neighbors, N_f=7 facets.\n4.1 Main Results (RQ1) Full results table for all four datasets:\nModel Books H@1 Books H@5 Goodreads H@1 Goodreads H@5 MovieTV H@1 MovieTV H@5 Yelp H@1 Yelp H@5 LightGCN 0.1753 0.5703 0.2499 0.7903 0.3482 0.6883 0.3444 0.7546 SASRec 0.0914 0.4845 0.1324 0.5407 0.3399 0.6382 0.2305 0.5597 P5 0.2192 0.5273 0.1569 0.5060 0.1696 0.5008 0.1444 0.5220 Vanilla LLM 0.3138 0.7270 0.2864 0.7390 0.4050 0.8603 0.1692 0.6861 iAgent (static) 0.3925 0.6905 0.2617 0.6591 0.4253 0.7420 0.3995 0.7300 RecBot (dynamic) 0.3984 0.6786 0.2705 0.6495 0.4367 0.7309 0.4007 0.7169 AgentCF (dynamic) 0.3457 0.7403 0.2951 0.7726 0.3906 0.7864 0.1925 0.6374 i2Agent (dynamic) 0.4453 0.7708 0.3099 0.7675 0.4912 0.8221 0.4205 0.7648 MemRec 0.5117 0.8007 0.3997 0.8052 0.5882 0.8817 0.4868 0.7908 Source: MemRec, Tables 2-3\nThe key takeaway is the paradigm hierarchy: Collaborative \u0026gt; Dynamic \u0026gt; Static \u0026gt; No Memory. Dynamic agents (AgentCF, i2Agent) consistently beat static ones (iAgent), which beat Vanilla LLM. But even the SOTA dynamic agent (i2Agent) falls significantly short of MemRec: on Goodreads H@1 +28.98% relative gain, on MovieTV +19.75%, on Yelp +15.77%.\n4.2 Cognitive Overload Validation (RQ2) This is the most engineering-interesting experiment. MemRec is compared with:\nVanilla LLM — no memory Naive Collaborative Agent — monolithic, processes uncurated context in one model MemRec — decoupled, curate-then-synthesize The Naive Agent plateaus: one model can\u0026rsquo;t simultaneously absorb verbose context and perform complex ranking. MemRec breaks the plateau through decoupling — LLM_Rec receives only high-signal distilled context. Result: +34% relative H@1 gain on Books over the Naive Agent. This directly confirms that architectural decoupling is not an optimization but a necessity.\n4.3 Ablation Studies (RQ4) What happens if we remove MemRec components? Ablation on Books:\nAblation H@1 Drop What it means MemRec (Full) 0.527 — baseline w/o Collab. Read 0.475 −9.9% Without collaborative retrieval — agent sees only personal history. Largest drop w/o LLM Curation 0.498 −5.5% Generic heuristics instead of domain-adaptive LLM rules. More noise w/o Collab. Write 0.505 −4.2% Without async propagation. Static graph still works, but loses evolving precision Collaborative retrieval is the most important component. LLM curation beats generic heuristics. Async propagation is critical for top-1 precision, though without it broad recall (H@5) remains high (0.814).\n4.4 Robustness (RQ5) MemRec is not just robust to data sparsity — it is most useful precisely for data-sparse users. The low-activity group gets +91.4% relative H@1 gain over Vanilla LLM. The collaborative graph compensates for the deficit of personal history.\nAt 30% noise injection (fake items in history), MemRec holds H@1 = 0.491 — resilience thanks to \u0026ldquo;Curate-then-Synthesize\u0026rdquo;: LLM curation filters irrelevant peers before they reach the reasoning agent.\n4.5 Pareto Frontier (RQ3) MemRec doesn\u0026rsquo;t just perform better — it establishes a new Pareto frontier between reasoning quality, online inference cost, and deployment flexibility. The key: heavy cognitive load of processing the collaborative graph is offloaded to async offline batches. Online inference sees only distilled M_\\text{collab} .\nConfigurations:\nConfiguration LLM_Rec / LM_Mem H@1 Latency Cost Notes Vanilla LLM 4o-mini / — 0.330 ~5.1s lowest No memory Standard 4o-mini / 4o-mini 0.524 ~16.5s low Base MemRec Cloud-OSS 4o-mini / OSS-120B 0.561 ~11.8s low Near-ceiling, open-weights LM_Mem Local-Qwen 4o-mini / Qwen-2.5-7B 0.470 ~34.0s low* Privacy-sensitive, on-premise Ceiling gpt-4o / 4o-mini 0.580 ~10.4s high Peak performance *Local deployment — zero API cost for memory maintenance.\nCloud-OSS is particularly interesting: open-weights LM_Mem (gpt-oss-120b) delivers near-ceiling results at low cost. This means memory management can be fully on-premise without quality loss — privacy-preserving deployment is real.\nThe token breakdown reveals an engineering insight: input tokens make up ~80% of total usage (input is 3-4x cheaper than output). Stage-R and Stage-W are heavily input-biased — they absorb verbose context (cheap) and produce condensed insights (expensive, but little). Per user: ~5,100 input + ~1,300 output = ~6,400 total tokens. Effective cost is radically lower than a naive total-token estimate would suggest.\n5. From Recommender Systems to Arbitrary Agents MemRec is a paper about recommender systems. But its architectural patterns are universal. Let\u0026rsquo;s make the transfer.\nExplicit concept mapping:\nRecommender Systems Arbitrary LLM Agents User u Agent a with personal memory M_a Item i Task, artifact, document t with memory M_t User-item interaction Agent-task execution, agent reads document Co-engagement (users share items) Shared context (agents worked on same task) Peer users Peer agents (team members) Collaborative filtering signal Organizational knowledge transfer Two concrete scenarios where collaborative memory applies to arbitrary agents.\nMulti-agent shared memory. A team of agents works on a project. Agent A solves a dependency conflict problem in Go modules. Agent B encounters the same problem a week later. Without collaborative memory — B starts from scratch, spends the same hours. With a collaborative memory graph — the interaction A→\u0026ldquo;dependency conflict solution\u0026rdquo; propagates to agent B through a shared edge (both work with Go modules). This isn\u0026rsquo;t a shared filesystem, it\u0026rsquo;s semantic propagation: B receives a distilled insight from A, grounded in evidence.\nOrganizational memory. A support agent has processed 100 SSO integration tickets. The memory graph links these tickets through common themes. A new support agent arrives — and instead of reading 100 tickets, gets synthesized facets: \u0026ldquo;SSO integration pitfalls: certificate chain (conf 0.9), redirect URI mismatch (conf 0.85), token refresh race (conf 0.7).\u0026rdquo; This is M_\\text{collab} , distilled from the organizational graph.\nSounds like science fiction? MemRec shows it works on recommender systems. The transfer to arbitrary agents is an engineering task, not a research breakthrough. The architecture is the same: LM_Mem curates the graph, LLM_Rec reasons, async propagation updates.\n6. The Landscape: Three Paradigms of Production Memory While research was paving the way to collaborative memory, industry was building tools. Three paradigms, each with an arxiv paper and production footprint. I chose these three specifically because they are fundamentally different — not feature-list competitors, but different models of memory.\ngraph TB subgraph \"Mem0: Managed Memory Layer\" M0[\"Vector store+ hybrid retrieval\"] M0 --\u003e|\"semantic + BM25+ entity\"| M0R[\"Fused results\"] end subgraph \"Letta: OS-Tiered Memory\" L0[\"Core memory(in-context, RAM)\"] L1[\"Archival memory(external, disk)\"] L0 \u003c--\u003e|\"page viafunction calls\"| L1 end subgraph \"Zep: Temporal Knowledge Graph\" Z0[\"Graphiti enginetime-aware facts\"] Z0 --\u003e|\"graph traversal+ temporal\"| Z0R[\"Context withhistory\"] end style M0 fill:#e65100,color:#fff style L0 fill:#4a148c,color:#fff style L1 fill:#4a148c,color:#fff style Z0 fill:#004d40,color:#fff 6.1 Mem0: Managed Memory Layer Mem0 (Chhikara et al., 2025) is a universal memory layer for AI agents. 59.6k GitHub stars, YC S24. The ideology: memory as a managed service — the agent doesn\u0026rsquo;t think about storage, Mem0 extracts, consolidates, retrieves.\nArchitecturally — multi-level memory with layers:\nLayer Lifetime What it stores Conversation one turn In-flight messages, tool outputs Session minutes-hours Current task context User weeks-forever Personalization, preferences Organizational global Shared FAQs, policies Retrieval is multi-signal (after the April 2026 update): semantic search + BM25 keyword + entity matching, all fused in parallel. This delivers benchmarks: LoCoMo 91.6 (+20 over previous algorithm), LongMemEval 94.8 (+27), BEAM 1M 64.1, BEAM 10M 48.6.\nThe new algorithm (April 2026, migration guide) is single-pass ADD-only: one LLM call for extraction, no UPDATE/DELETE. Memories accumulate, nothing is overwritten. Agent-generated facts are first-class citizens: if an agent confirmed an action, the information is stored with equal weight. Entity linking — entities are extracted, embedded, and linked across memories for retrieval boosting.\nMinimal SDK snippet:\nfrom mem0 import Memory memory = Memory() # Add memory memory.add( [\u0026#34;I\u0026#39;m Alex, I prefer YAML and dark theme.\u0026#34;], user_id=\u0026#34;alex\u0026#34; ) # Retrieval — hybrid: semantic + BM25 + entity results = memory.search( \u0026#34;What are Alex\u0026#39;s preferences?\u0026#34;, filters={\u0026#34;user_id\u0026#34;: \u0026#34;alex\u0026#34;}, top_k=3 ) 6.2 Letta (MemGPT): OS-Inspired Tiered Memory MemGPT (Packer et al., UC Berkeley, 2023) is the seminal paper that framed LLMs as operating systems. The idea: virtual context management, inspired by hierarchical memory in traditional OSes. Core memory (in-context, like RAM) + archival memory (external storage, like disk). The agent manages data movement between tiers via function calls.\nLetta is the production framework built on MemGPT. White-box, model-agnostic. The core abstraction is memory blocks: structured sections of the context window that persist across interactions.\ngraph TB subgraph \"Context Window (Limited)\" SYS[\"System Prompt\"] BLOCKS[\"Memory Blocks(Core Memory = RAM)\"] CONV[\"Conversation(working memory)\"] end subgraph \"External Storage\" ARCH[\"Archival Memory(= Disk)unlimited\"] end LLM[\"LLM\"] SYS --\u003e LLM BLOCKS --\u003e LLM CONV --\u003e LLM LLM --\u003e|\"core_memory_appendcore_memory_replace\"| BLOCKS LLM \u003c--\u003e|\"archival_memory_searcharchival_memory_insert\"| ARCH style BLOCKS fill:#4a148c,color:#fff style ARCH fill:#311b92,color:#fff style LLM fill:#1565c0,color:#fff Core memory (memory blocks) is always in context, like RAM. Archive is external storage, the agent \u0026ldquo;pages\u0026rdquo; data via function calls (archival_memory_search, archival_memory_insert). If a block overflows (chars_limit), the agent itself decides what to evict to archival. Here\u0026rsquo;s what the LLM sees:\n\u0026lt;memory_blocks\u0026gt; \u0026lt;persona\u0026gt; \u0026lt;description\u0026gt;The persona block: Stores details about your current persona, guiding how you behave and respond.\u0026lt;/description\u0026gt; \u0026lt;metadata\u0026gt;- chars_current=128 - chars_limit=5000\u0026lt;/metadata\u0026gt; \u0026lt;value\u0026gt;I am a helpful assistant named Sam. I enjoy helping users solve problems.\u0026lt;/value\u0026gt; \u0026lt;/persona\u0026gt; \u0026lt;human\u0026gt; \u0026lt;description\u0026gt;The human block: Stores key details about the person you are conversing with.\u0026lt;/description\u0026gt; \u0026lt;value\u0026gt;The user\u0026#39;s name is Alice. She is a software engineer who prefers concise answers.\u0026lt;/value\u0026gt; \u0026lt;/human\u0026gt; \u0026lt;/memory_blocks\u0026gt; Key properties of memory blocks:\nAgent-managed — the agent autonomously organizes information by block labels Always visible — blocks are in context always, no retrieval needed Shareable — multiple agents can read the same block (shared memory, multi-agent coordination) Read-only option — policies as read-only blocks, the agent can\u0026rsquo;t modify them Benchmark: DMR (Deep Memory Retrieval) 93.4% — the baseline that Zep surpassed.\nMinimal snippet:\nfrom letta import Letta client = Letta() agent = client.agents.create( name=\u0026#34;memory_agent\u0026#34;, memory_blocks=[ {\u0026#34;label\u0026#34;: \u0026#34;human\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;User prefers Go and concise answers.\u0026#34;}, {\u0026#34;label\u0026#34;: \u0026#34;persona\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;I am a senior engineering assistant.\u0026#34;}, ], # archival memory — external, agent pages via function calls ) 6.3 Zep: Temporal Knowledge Graph Zep (Rasmussen et al., 2025) is a memory layer service built on Graphiti. Graphiti is an open-source framework for temporal knowledge graphs (context graphs with a temporal axis). Unlike static RAG, Graphiti makes real-time incremental updates: relationships and facts evolve with the data, without batch recomputation.\nThe architecture: dynamic synthesis — simultaneously processes unstructured conversational data and structured business data while maintaining historical relationships. A fact in the graph has time validity: when it became true, when it stopped being true. A query can reason over how facts changed over time, not just what is true now.\ngraph LR subgraph \"Input\" CONV[\"Conversational data(unstructured)\"] BIZ[\"Business data(structured)\"] end subgraph \"Graphiti Engine\" ENT[\"EntityExtraction\"] EDGE[\"EdgeCreation\"] TEMP[\"TemporalAnnotation\"] GRAPH[\"TemporalKnowledge Graph\"] end subgraph \"Output\" FACTS[\"Facts withtime validity\"] Q[\"Query:'what changedsince X?'\"] end CONV --\u003e ENT BIZ --\u003e ENT ENT --\u003e EDGE --\u003e TEMP --\u003e GRAPH GRAPH --\u003e FACTS FACTS --\u003e Q style GRAPH fill:#004d40,color:#fff style TEMP fill:#00695c,color:#fff Each fact in the graph is annotated with temporal markers: when it became true (valid_from), when it stopped (valid_to). The query \u0026ldquo;what was the user doing in March?\u0026rdquo; traverses the graph by temporal edges, not just semantic similarity. This is a qualitatively different retrieval — not \u0026ldquo;find similar\u0026rdquo; but \u0026ldquo;find what was true at moment X.\u0026rdquo;\nBenchmarks:\nDMR: 94.8% vs MemGPT 93.4% — Zep outperforms on the benchmark the MemGPT team established as their primary metric LongMemEval: +18.5% accuracy, −90% latency vs baseline — especially strong in cross-session synthesis and long-term context maintenance Minimal snippet:\nfrom zep_cloud import Zep client = Zep(api_key=\u0026#34;...\u0026#34;) # Add an episode — Graphiti syncs the graph client.memory.add( session_id=\u0026#34;session-1\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;I switched from PostgreSQL to SQLite for project X.\u0026#34;}] ) # Retrieval — graph traversal + temporal reasoning context = client.memory.get(session_id=\u0026#34;session-1\u0026#34;) # context contains facts with temporal validity 6.4 Emerging: A-MEM and LangMem Mem0, Letta, Zep — three production tools. But research doesn\u0026rsquo;t stand still.\nA-MEM (arXiv:2502.12110, NeurIPS 2025) is an agentic memory system following Zettelkasten principles. Memory organizes itself: dynamic indexing and linking create interconnected knowledge networks. This is a research direction with no production footprint yet, but the idea — memory as a self-organizing system — resonates with collaborative propagation in MemRec.\nRecMem (arXiv:2605.16045, ACL 2026 Findings) rethinks not WHAT to store, but WHEN to consolidate. Existing systems eagerly invoke LLMs for every interaction — expensive. RecMem accumulates interactions in a subconscious layer (cheap embeddings), and invokes LLMs for episodic and semantic memory extraction only when sustained recurrence of semantically similar interactions is observed — an analogue of a phase transition. Result: up to 87% reduction in token cost while exceeding the accuracy of three SOTA memory systems. This is a direct response to the same cost challenge as MemRec, but through density-triggered consolidation rather than async propagation. For a visual walkthrough of the architecture, see this YouTube video: \u0026ldquo;Phase Transitions in AI Agent Memory: REC Memory Architecture\u0026rdquo;.\nLangMem (langchain-ai/langmem) is SDK-level primitives for LangGraph. Semantic, episodic, procedural memory as functional building blocks. Not a standalone system, but a library for those building their own memory architecture on top of the LangGraph storage layer.\n7. TOP-3 Comparison: Nine Axes Now let\u0026rsquo;s compare Mem0, Letta, and Zep systematically — across nine axes. Each axis is an architectural decision, not a feature.\nAxis Mem0 Letta (MemGPT) Zep 1. Memory model Vector store + entity graph OS-tiered (core=RAM, archival=disk) Temporal knowledge graph (Graphiti) 2. Architecture Managed layer (service/SDK) In-process agent (framework) Service (Graphiti engine) 3. Retrieval Hybrid: semantic + BM25 + entity (parallel fused) Hierarchical paging (function calls) Graph traversal + temporal reasoning 4. Temporal awareness Added Apr 2026 (time-aware retrieval) None First-class (facts with time validity) 5. Update cost Single-pass ADD-only (1 LLM call) Agent self-edits via function calls Real-time incremental (no batch) 6. Collaborative memory No (entity linking — intra-agent) Partial (shared blocks — multi-agent) No (graph per-session) 7. Benchmarks LoCoMo 91.6, LongMemEval 94.8, BEAM 1M 64.1 DMR 93.4 DMR 94.8, LongMemEval +18.5% 8. Production Self-host (Docker) / Cloud / CLI Self-host (framework) / Cloud Self-host / Cloud 9. Ergonomics SDK (Python/TS), CLI, agent skills SDK (Python/TS), white-box, model-agnostic SDK (Python/TS/Go) 7.1 Benchmarks Cross-Table Benchmarks are a sore subject. LoCoMo, LongMemEval, DMR, BEAM — different benchmarks measuring different things. Direct comparison is valid only on shared benchmarks.\nBenchmark What it measures Mem0 Letta Zep LoCoMo Long conversational memory (single/multi-hop, temporal, open-domain) 91.6 — — LongMemEval Cross-session synthesis, long-term context 94.8 — +18.5% acc, −90% latency DMR Deep Memory Retrieval — 93.4% 94.8% BEAM Million-token scale memory 64.1 (1M), 48.6 (10M) — — Shared benchmarks: LongMemEval for Mem0 and Zep (direct comparison possible); DMR for Letta and Zep (Zep outperforms 94.8 vs 93.4). LoCoMo and BEAM are Mem0-only. A global claim \u0026ldquo;X is better than Y\u0026rdquo; is impossible — only per-benchmark statements.\n7.2 Spotlight: Collaborative Memory — The Gap Axis 6 is the most important for this article. None of Mem0, Letta, or Zep implement collaborative memory in the sense MemRec defines it (memory graph with cross-agent/cross-entity relational signal transfer).\nTo be precise:\nMem0 entity linking is intra-agent: entities are linked within one user\u0026rsquo;s memory, not across agents. This is retrieval boosting, not collaborative signal transfer. Letta shared blocks enable multi-agent coordination: several agents read one read-only block. This is shared state, but not collaborative propagation — no graph evolution. Zep temporal graph is per-session: the graph is built for one session/user, no cross-agent graph with propagation. MemRec shows collaborative memory delivers +14-29% H@1. This isn\u0026rsquo;t a marginal optimization — it\u0026rsquo;s a qualitative leap, and no production tool makes it. More on this in Section 8.\n8. Research vs Tools: Where the Gap Is Collaborative memory isn\u0026rsquo;t the only gap. The research frontier (MemRec) outpaces tooling (TOP-3) in four areas. Let\u0026rsquo;s break down each: what research claims, what tools do, what to adopt now.\ngraph TB subgraph \"Research Frontier (MemRec)\" R1[\"Collaborative memorygraph G=(V,E)\"] R2[\"Architectural decouplingLM_Mem / LLM_Rec\"] R3[\"Temporal-awarefacts + history\"] R4[\"Curate-then-SynthesizeIB-grounded distillation\"] end subgraph \"Tooling Reality\" T1[\"Mem0: intra-agententity linking\"] T2[\"Letta: in-processno decoupling\"] T3[\"Zep: first-classMem0: added 2026Letta: none\"] T4[\"Mem0: consolidationLetta: agent-managedZep: graph synthesis\"] end R1 -.-\u003e|\"GAP\"| T1 R2 -.-\u003e|\"PARTIAL\"| T2 R3 -.-\u003e|\"Zep: COVERED\"| T3 R4 -.-\u003e|\"AD HOC\"| T4 style R1 fill:#c62828,color:#fff style T1 fill:#c62828,color:#fff style R2 fill:#f9a825,color:#000 style T2 fill:#f9a825,color:#000 Gap 1: Collaborative Memory — The Largest Research claim: MemRec shows +14-29% H@1 from collaborative signal. A memory graph G=(V,E) with high-order connectivity transfers relational signals between agents/items (Chen et al., 2026).\nTool status: None of the TOP-3 implement collaborative memory. Mem0 entity linking is intra-agent. Letta shared blocks are shared state without propagation. Zep temporal graph is per-session.\nAdopt now: If you need collaborative signal — custom build. MemRec\u0026rsquo;s architecture is portable: LM_Mem curates the graph, LLM_Rec reasons, async propagation updates. No magic, just engineering. Or wait for research propagation to reach tooling — given the pace (MemRec ACL 2026, A-MEM NeurIPS 2025), 12-18 months.\nGap 2: Architectural Decoupling — Partial Research claim: Decoupling memory management (LM_Mem) from reasoning (LLM_Rec) resolves cognitive overload (+34% H@1 over naive monolithic). IB-grounded \u0026ldquo;Curate-then-Synthesize\u0026rdquo; — two compression passes.\nTool status: Partial. Mem0 separation — extraction/retrieval are separate from the reasoning LLM, but not as a first-class architectural principle with IB framing. Letta — in-process, the agent self-edits memory blocks, no decoupling. Zep — Graphiti engine is separate from reasoning, but without explicit distillation stages.\nAdopt now: Mem0 is closest. If cost/latency trade-off matters — Mem0 separation delivers part of the benefit. Full decoupling as in MemRec — custom.\nGap 3: Temporal Awareness — Zep First-Class Research claim: Facts evolve. A query should reason over how facts changed, not just what is true now.\nTool status: Zep — first-class (Graphiti temporal graph, facts with time validity). Mem0 — added in April 2026 (time-aware retrieval, but not first-class like Zep). Letta — no temporal awareness.\nAdopt now: Zep. If temporal reasoning is critical (enterprise use cases, cross-session synthesis) — Zep is the only first-class option. LongMemEval +18.5% accuracy / −90% latency — direct confirmation.\nGap 4: Distillation — Ad Hoc Research claim: \u0026ldquo;Curate-then-Synthesize\u0026rdquo; — IB-grounded, domain-adaptive LLM curation rules, structured facets with confidence and grounding evidence. A unified approach.\nTool status: Ad hoc. Mem0 — consolidation (single-pass extraction, but no structured facets). Letta — agent-managed (the agent decides what goes in core memory). Zep — graph synthesis (Graphiti builds the graph, but not in facet format). No unified distillation approach.\nAdopt now: Depends on use case. Mem0 consolidation — for simple personalization. Zep graph synthesis — for complex relational reasoning. Structured facets as in MemRec — custom.\n9. Engineering Decision Framework Which tool to choose? Depends on four use-case characteristics.\ngraph TB START[\"Use case needsagent memory?\"] Q1{\"Temporal reasoningcritical?\"} Q2{\"Scale \u003e 1Mmemories?\"} Q3{\"Multi-agentcoordination?\"} Q4{\"Collaborativememory needed?\"} ZEP[\"Zep(temporal graph)\"] MEM0[\"Mem0(managed layer)\"] LETTA[\"Letta(OS-tiered)\"] CUSTOM[\"Custom buildMemRec-style architectureor wait for tooling\"] START --\u003e Q1 Q1 --\u003e|\"Yes\"| ZEP Q1 --\u003e|\"No\"| Q2 Q2 --\u003e|\"Yes\"| MEM0 Q2 --\u003e|\"No\"| Q3 Q3 --\u003e|\"Yes\"| LETTA Q3 --\u003e|\"No\"| Q4 Q4 --\u003e|\"Yes\"| CUSTOM Q4 --\u003e|\"No\"| MEM0 style ZEP fill:#004d40,color:#fff style MEM0 fill:#e65100,color:#fff style LETTA fill:#4a148c,color:#fff style CUSTOM fill:#c62828,color:#fff Brief summary by branch:\nUse case characteristic Recommendation Why Temporal reasoning critical Zep First-class temporal graph, DMR 94.8%, LongMemEval +18.5% Scale \u0026gt; 1M memories, personalization Mem0 BEAM 1M 64.1, managed layer, hybrid retrieval, 59.6k stars production-hardened Multi-agent coordination Letta Shared memory blocks, OS-tiered, agent-managed Collaborative memory needed Custom build None of the TOP-3 does collaborative. MemRec architecture is portable. Simple personalization, fast start Mem0 Best ergonomics, CLI, agent skills, cloud + self-host An important caveat: the decision tree is simplified. The real choice depends on deployment model (on-premise vs cloud), latency requirements, compliance. Mem0\u0026rsquo;s Cloud-OSS config shows that open-weights LM_Mem delivers near-ceiling results — privacy-preserving deployment is real.\n10. Open Problems and Where Research Is Heading The article is wrapping up, but the topic isn\u0026rsquo;t. Several open problems will define the next 12-18 months.\nCollaborative memory across agents. MemRec demonstrated collaborative memory within one system (recommender). The transfer to multi-agent systems is an open problem. How to propagate insights between agents with different specializations? How to avoid noise during propagation? A-MEM (NeurIPS 2025) points a direction — self-organizing memory via Zettelkasten — but there are no production tools yet.\nMemory consolidation / dreaming. Anthropic and OpenAI in 2026 shipped background memory consolidation — so-called \u0026ldquo;Dreaming.\u0026rdquo; The idea: the agent \u0026ldquo;sleeps\u0026rdquo; between sessions and consolidates memory — compression, deduplication, extracting patterns. This resonates with MemRec\u0026rsquo;s async propagation, but at a different scale: not propagation between graph nodes, but consolidation of the agent\u0026rsquo;s entire memory. The mechanics of Dreaming (Anthropic Auto Dream, OpenAI Dreaming V3 with 82.8% factual recall) are covered in detail in the AI Agent Design Patterns series, Part 6 — I won\u0026rsquo;t duplicate that here.\nEvaluation gaps. There is no unified benchmark for collaborative memory. LoCoMo, LongMemEval, DMR — single-agent benchmarks. MemRec uses RS-specific metrics (H@K, N@K). How to evaluate collaborative memory for arbitrary agents? This is a research problem — without a benchmark there\u0026rsquo;s no comparison, without comparison there\u0026rsquo;s no progress.\nPrivacy-preserving federated memory. MemRec notes in its limitations: future work is federated memory updates. If the collaborative graph is distributed across organizations (each with its own NDA) — how to transfer insights without transferring raw data? Federated learning for memory graphs is an open area.\nConclusion LLM agent memory has traveled from complete absence to dynamic isolated memory. MemRec (ACL 2026) points to the next milestone — collaborative memory, where agent memories are linked in a graph and exchange relational signals. Architecturally this means: decouple memory management from reasoning, build a memory graph, curate-then-synthesize for distillation, async propagation for evolution.\nThree production tools — Mem0, Letta, Zep — represent three different paradigms of isolated memory: managed layer, OS-tiered, temporal graph. All three are fit-for-purpose for single-agent use cases. But none implements collaborative memory, which MemRec has proven as the next step (+14-29% H@1).\nResearch outpaces tooling. This is normal — the research frontier moves faster than production adoption. For engineers today: choose from the TOP-3 based on use-case characteristics, but remember that collaborative memory is the next wave, and MemRec shows how to build it.\nReferences:\nMemRec: Collaborative Memory-Augmented Agentic Recommender System — Chen et al., ACL 2026 Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory — Chhikara et al., 2025 MemGPT: Towards LLMs as Operating Systems — Packer et al., 2023 Zep: A Temporal Knowledge Graph Architecture for Agent Memory — Rasmussen et al., 2025 A-MEM: Agentic Memory for LLM Agents — Xu et al., NeurIPS 2025 RecMem: Recurrence-based Memory Consolidation for Efficient and Effective Long-Running LLM Agents — Dai et al., ACL 2026 Findings Lost in the Middle — Liu et al., 2024 Generative Agents — Park et al., 2023 ","permalink":"https://triumphpc.github.io/blog/posts/agent-memory-collaborative-deep-dive/","summary":"A deep dive into LLM agent memory architecture based on the MemRec paper (ACL 2026): collaborative memory, decoupling memory management from reasoning, Information Bottleneck. Comparison of the TOP-3 production tools — Mem0, Letta (MemGPT), and Zep — across nine axes. Highlighting the gap: research outpaces tooling.","title":"Agent Memory: From Isolated Context to Collaborative Memory"},{"content":"When you write mu.Lock() in Go, you probably think \u0026ldquo;lock.\u0026rdquo; When you write atomic.AddInt64(\u0026amp;x, 1), you think \u0026ldquo;atomic add.\u0026rdquo; Two different tools, two different mental models, two different sections of the standard library documentation. But here is the strange thing: they are the same thing. More precisely, they are one stack of abstractions, and each layer is built on top of the one below.\nIn this article I will dig through that stack, from the bottom to the top and back. We will start with a broken counter++, climb down through CPU instructions, climb back up through CAS (compare-and-swap), atomic operations, futex, and finally arrive at sync.Mutex — and discover that it is mostly an optimization layered on top of atomic CAS, plus a fairness mechanism invented to fix a real production bug from 2015.\nThe Stack Before diving in, here is the picture I want you to keep in your head. Five layers, each one built on the previous:\ngraph TB L5[\"Level 5: When to use what\nBenchmarks, cheatsheet, pitfalls\"] L4[\"Level 4: sync.Mutex\nstate, sema, spinning, starvation\"] L3[\"Level 3: Futex\nuserspace fast path + kernel slow path\"] L2[\"Level 2: Atomic operations\nCAS, Load, Store, Add — sync/atomic\"] L1[\"Level 1: Hardware\nLOCK CMPXCHG (x86), LDADD/CAS (ARM LSE)\"] L0[\"Level 0: The Problem\ndata race, x++ is 3 instructions\"] L0 --\u003e L1 --\u003e L2 --\u003e L3 --\u003e L4 --\u003e L5 style L0 fill:#c62828,color:#fff style L1 fill:#ad1457,color:#fff style L2 fill:#6a1b9a,color:#fff style L3 fill:#4527a0,color:#fff style L4 fill:#283593,color:#fff style L5 fill:#1565c0,color:#fff The claim of this article is that you cannot really understand Level 4 (sync.Mutex) without understanding Level 2 (atomic), and you cannot understand Level 2 without at least a glimpse of Level 1 (hardware). Conversely, the only reason to care about Level 1 is that it produces the practical recommendations of Level 5.\nLet\u0026rsquo;s start at the bottom.\n1. The Problem: Why x++ Breaks Here is the simplest possible data race. A thousand goroutines increment a shared counter:\nvar counter int64 var wg sync.WaitGroup func main() { for i := 0; i \u0026lt; 1000; i++ { wg.Add(1) go func() { defer wg.Done() counter++ }() } wg.Wait() fmt.Println(counter) } Run it. The answer is not 1000. It is something like 987, or 993, or 971. Sometimes it is exactly 1000, and that is worse — it means the bug is hiding.\nWhy? Because counter++ is not one operation. It looks like one operation in Go source, but the compiler translates it to three. Here is the actual ARM64 assembly Go produces for counter++:\nMOVD main.counter(SB), R0 ; load counter from memory into register R0 ADD $1, R0, R0 ; add 1 to R0 MOVD R0, main.counter(SB) ; store R0 back to memory Three instructions: load, modify, store. Each is atomic on its own, but the sequence is not. Between the load and the store, another goroutine can sneak in.\nHere is the timeline when two goroutines collide:\nsequenceDiagram participant G1 as Goroutine 1 participant Mem as Memory (counter=5) participant G2 as Goroutine 2 G1-\u003e\u003eMem: MOVD counter → R0 (R0=5) Note over G1: about to ADD G2-\u003e\u003eMem: MOVD counter → R0 (R0=5) G1-\u003e\u003eG1: ADD → R0=6 G1-\u003e\u003eMem: MOVD R0 → counter (counter=6) G2-\u003e\u003eG2: ADD → R0=6 G2-\u003e\u003eMem: MOVD R0 → counter (counter=6) Note over Mem: One increment lost Both goroutines read 5, both compute 6, both store 6. We did two increments, the counter went up by one. Multiply by a thousand goroutines and you get 987.\nThis is a data race, and Go\u0026rsquo;s memory model (go.dev/ref/mem) defines it formally. A read-write data race on a memory location x consists of a read-like operation r and a write-like operation w on x, at least one of which is non-synchronizing, such that neither happens before the other. Read that twice: the definition is about ordering, not timing. Two operations on the same location, with no synchronization between them, is a race — even if in practice they almost never overlap.\n1.1. DRF-SC: the promise and the price The Go memory model makes one big promise, called DRF-SC: data-race-free programs execute in a sequentially consistent manner. If your program has no data races, you can reason about it as if all goroutines were multiplexed onto a single processor, taking turns. You do not have to worry about compiler reordering, store buffers, cache effects, or any of the other hardware-level weirdness that plagues C++ programmers.\nThe price is that you must actually eliminate the data races. Go\u0026rsquo;s race detector (go test -race, built on ThreadSanitizer) will catch them at runtime, but you have to write race-free code in the first place. And to do that, you need synchronization.\nThe formal model behind DRF-SC is the same one used by C++, Java, JavaScript, Rust, and Swift. It comes from Hans-J. Boehm and Sarita V. Adve\u0026rsquo;s paper \u0026ldquo;Foundations of the C++ Concurrency Memory Model\u0026rdquo; (PLDI 2008). Go aligned with this model in the June 2022 memory model revision, which shipped with Go 1.19.\n1.2. Happens-before: the relation that makes everything work The formal definition of data race depends on a relation called happens-before. Informally, an operation A happens-before an operation B if you can prove A must have completed before B started. The relation is the transitive closure of two things:\nSequenced-before: within a single goroutine, statements execute in source order. Statement 5 is sequenced before statement 6. Synchronized-before: across goroutines, certain operations create synchronization edges. ch \u0026lt;- x happens-before the corresponding \u0026lt;-ch. mu.Unlock() happens-before the next mu.Lock(). An atomic store happens-before the atomic load that observes the stored value. If A happens-before B, then anything A wrote to memory is visible to B. If neither happens-before the other, you have a race.\nThis is why counter++ is broken: nothing in the program establishes a happens-before relation between the increments of different goroutines. They are unordered, so the read in goroutine A can overlap the write in goroutine B.\nTo fix the race, we need to add synchronization. There are two main paths:\nMake the increment itself atomic — a single operation that cannot be interrupted. Wrap the increment in a mutex, so only one goroutine can execute it at a time. These are Levels 2 and 4 in our stack. Let\u0026rsquo;s take them in order.\n2. The First Path: Atomic Operations The cleanest fix for counter++ is to make it atomic:\nvar counter int64 // ... atomic.AddInt64(\u0026amp;counter, 1) atomic.AddInt64 performs the load, add, and store as a single indivisible operation. No other goroutine can observe an intermediate state, no other goroutine can sneak in between the load and the store, because there is no \u0026ldquo;between.\u0026rdquo;\nHow is that possible? Because underneath, atomic.AddInt64 compiles to a single CPU instruction.\n2.1. CAS: the universal primitive The fundamental building block of all atomic operations is CAS — Compare-And-Swap. Its signature, in pseudocode:\nCAS(addr, expected, new) → bool if *addr == expected: *addr = new return true else: return false The whole thing is atomic: the comparison and the store happen as one indivisible step. If *addr equals expected, the new value is written and CAS returns true. Otherwise, nothing is written and CAS returns false.\nCAS is special. In 1991, Maurice Herlihy published a paper called \u0026ldquo;Wait-Free Synchronization\u0026rdquo; in ACM Transactions on Programming Languages and Systems. He classified synchronization primitives by their consensus number — the maximum number of threads for which the primitive can solve the consensus problem (all threads agree on a single value). The hierarchy is brutal:\nPrimitive Consensus number Read/write registers 1 Test-and-set 2 Fetch-and-add 2 Queue, stack 2 CAS ∞ CAS has consensus number ∞. That makes it a universal primitive: any wait-free or lock-free data structure can be built on top of CAS. The others cannot. Test-and-set can solve consensus for 2 threads, fetch-and-add for 2, but only CAS scales to arbitrarily many. This is why every modern architecture provides CAS in hardware, and why Go builds everything else on top of it.\n2.2. CAS-loop: how Add is actually implemented You might wonder: if CAS is the universal primitive, how do you build Add out of it? You cannot CAS-and-add in one shot, because CAS only writes a specific new value, not \u0026ldquo;old plus one.\u0026rdquo; The answer is a loop:\nfunc addInt64(addr *int64, delta int64) int64 { for { old := atomic.LoadInt64(addr) new := old + delta if atomic.CompareAndSwapInt64(addr, old, new) { return new } // CAS failed — someone else wrote between our load and our CAS. // Loop, re-read the current value, try again. } } This is called a CAS-loop or compare-and-swap retry loop. It loads the current value, computes the new value, attempts to CAS. If the CAS fails, that means another goroutine modified the memory between our load and our CAS, so we retry with the new current value.\nflowchart TD Start([Add delta to addr]) --\u003e Load[\"old = *addr\"] Load --\u003e Compute[\"new = old + delta\"] Compute --\u003e CAS{\"CAS(addr, old, new)\"} CAS --\u003e|success| Done([done]) CAS --\u003e|failure — someone wrote| Load style Done fill:#2e7d32,color:#fff style CAS fill:#1565c0,color:#fff This is what makes CAS universal. Out of CAS, you can build Add, Subtract, Exchange, Min, Max, any operation you want — all of them become atomic by retrying until CAS succeeds.\nThere is a price. Under heavy contention (many goroutines all trying to update the same address), CAS-loops can waste CPU. Every failed CAS is a wasted attempt, and the loop spins until it wins. We will come back to this in section 4.\n2.3. Hardware: LOCK CMPXCHG and ARM LSE When you write atomic.CompareAndSwapInt64, the Go compiler emits a single CPU instruction. On x86, that instruction is LOCK CMPXCHG:\nlock cmpxchg [addr], new The LOCK prefix is important. In early x86 CPUs, LOCK literally asserted the LOCK# hardware pin, which froze the entire memory bus for the duration of the instruction. That was catastrophically expensive. In modern x86 CPUs (Pentium Pro and later), LOCK is much smarter: if the target address fits in a single cache line, the CPU uses cache line locking — it invalidates that one cache line on all other cores and performs the operation without touching the bus. Only if the operation spans two cache lines (which should not happen with proper alignment) does it fall back to the old bus lock.\nOn ARM64, things evolved in two stages. Pre-LSE (Large System Extensions, pre-ARMv8.1), CAS was emulated with a pair of instructions called LDXR/STXR (exclusive load and exclusive store). The load tagged the cache line for monitoring; if no other core wrote to it before the store, the store succeeded. Otherwise it failed and you retried — exactly the CAS-loop, but in hardware.\nARMv8.1 added LSE (Large System Extensions), which introduced single-instruction atomics: CAS, LDADD, LDCLR, STSET, and friends. These are faster because there is no in-CPU retry loop — the operation completes in one go. Apple Silicon (M1 and later) and AWS Graviton 3 and 4 all support LSE, which is why atomic-heavy code on these chips is so much faster than the pre-LSE era.\nThe key takeaway: every atomic.X call in Go is one CPU instruction. There is no function call, no loop (unless CAS fails), no syscall. Just one instruction. This is what makes atomics fast.\n2.4. Go\u0026rsquo;s sync/atomic package The sync/atomic package offers two styles of API.\nFunctions (the original form, present since Go 1):\nvar x int64 atomic.LoadInt64(\u0026amp;x) atomic.StoreInt64(\u0026amp;x, 42) atomic.AddInt64(\u0026amp;x, 1) atomic.SwapInt64(\u0026amp;x, 99) atomic.CompareAndSwapInt64(\u0026amp;x, expected, new) Plus equivalents for int32, uint32, uint64, uintptr, unsafe.Pointer, and the special-case atomic.Value for arbitrary types.\nTypes (introduced in Go 1.19, August 2022):\nvar x atomic.Int64 x.Load() x.Store(42) x.Add(1) x.Swap(99) x.CompareAndSwap(expected, new) var p atomic.Pointer[Config] // generic, type-safe var f atomic.Bool var u atomic.Uint64 var flags atomic.Uint32 The types are newer and nicer. They wrap the same underlying CPU instructions, but with three advantages:\nNo more \u0026amp;x — you call methods on the value directly, which makes ownership clearer. No alignment headaches — see the next section. Type safety for pointers — atomic.Pointer[T] is generic, no more unsafe.Pointer casting. The Go 1.19 release notes (go.dev/doc/go1.19) describe these types as a direct consequence of the memory model revision: \u0026ldquo;Along with the memory model update, Go 1.19 introduces new types in the sync/atomic package that make it easier to use atomic values, such as atomic.Int64 and atomic.Pointer[T].\u0026rdquo; Before 1.19, on 32-bit platforms (think GOARCH=386, GOARCH=arm), an int64 had to be 8-byte aligned for atomic operations to work — otherwise atomic.LoadInt64 would panic at runtime. You had to manually arrange struct fields or add padding. The new atomic.Int64 type guarantees alignment automatically, which kills a whole class of bugs.\n2.5. Memory ordering: Go only has one mode This is one of the biggest differences between Go and C++/Rust. C++11 introduced six memory orderings: memory_order_relaxed, memory_order_consume, memory_order_acquire, memory_order_release, memory_order_acq_rel, memory_order_seq_cst. Rust inherits the same six. The weaker orderings (relaxed, acquire, release) allow the compiler and CPU to reorder operations more aggressively, which can buy performance at the cost of more careful reasoning.\nGo has one. Sequentially consistent, always. Every atomic operation in Go is a full memory barrier. There is no atomic.LoadAcquire, no atomic.StoreRelease. As Russ Cox wrote when revising the memory model for Go 1.19: Go deliberately does not provide the relaxed orderings, because they are too easy to misuse, and the performance gain is usually small.\nThis is the same trade-off as in Level 1: simpler model, less performance, fewer bugs. If you are coming from C++ and looking for acquire/release in Go, stop. Use sync/atomic as-is, accept the full barrier cost, and your code will be correct by default.\nOne subtle but important consequence: atomic operations synchronize not just the atomic variable, but everything that happened before the atomic write in the same goroutine. This is the foundation of the publication pattern:\ntype Config struct { Hosts []string TTL time.Duration // ... many fields } var configPtr atomic.Pointer[Config] // Writer goroutine (applies config update) func updateConfig(c *Config) { // All writes to c.Hosts, c.TTL, etc. happen-before this Store. configPtr.Store(c) } // Reader goroutine (millions of calls) func getConfig() *Config { // Load synchronizes with the corresponding Store. // The returned *Config is fully initialized — no partial reads. return configPtr.Load() } The fields of Config are not atomic. They are plain []string, plain time.Duration. But because the writer fully constructs c before calling Store, and the reader\u0026rsquo;s Load synchronizes with that Store, the reader sees a fully initialized struct. No locks, no copying on the read path, no contention. This is the canonical way to do read-heavy configuration in Go.\n3. The Second Path: Mutex, Built on Top of Atomic Now we can finally explain sync.Mutex. The short version: it is mostly an optimization layered on top of atomic CAS, plus a fairness mechanism. Let\u0026rsquo;s see exactly how.\n3.1. The Mutex struct Here is the actual definition from Go 1.24 (source):\ntype Mutex struct { state int32 sema uint32 } Two fields. Eight bytes total. state is a packed bitfield carrying four pieces of information. sema is a semaphore that the runtime uses to park and wake goroutines.\nThe state field is laid out as follows:\n┌───────────────────────────────────────────────────────────────┐ │ state (int32) │ ├───────┬────────┬─────────────┬─────────────────────────────────┤ │ bit 0 │ bit 1 │ bit 2 │ bits 3 .. 31 │ │ │ │ │ │ │Locked │ Woken │ Starving │ Waiter count (29 bits, max ~536M)│ └───────┴────────┴─────────────┴─────────────────────────────────┘ 1 2 4 8, 16, 32, ... The constants in the source:\nconst ( mutexLocked = 1 \u0026lt;\u0026lt; iota // 1 mutexWoken // 2 mutexStarving // 4 mutexWaiterShift = iota // 3 starvationThresholdNs = 1e6 // 1 millisecond ) Four things encoded in 32 bits:\nLocked (bit 0): is the mutex currently held? 1 = yes. Woken (bit 1): has a waiter been woken up and is now trying to acquire? This is an optimization hint to Unlock — it knows not to wake another waiter because one is already awake. Starving (bit 2): is the mutex in starvation mode? See section 3.6. Waiter count (bits 3-31): how many goroutines are currently parked waiting for this mutex. 29 bits, so up to about 536 million waiters. You will never hit that. The sema field has no internal structure. It is an opaque token that the runtime\u0026rsquo;s semaphore code uses to manage a queue of parked goroutines.\n3.2. Fast path: one CAS, no syscall The hot path through Lock() is short:\nfunc (m *Mutex) Lock() { // Fast path: grab unlocked mutex. if atomic.CompareAndSwapInt32(\u0026amp;m.state, 0, mutexLocked) { // ... race detector bookkeeping ... return } // Slow path (outlined so that the fast path can be inlined) m.lockSlow() } That is it. One CAS. If the mutex is unlocked (state == 0), CAS it to mutexLocked (state == 1) and return. If the CAS fails, fall through to lockSlow.\nThe fast path is inlined. You can verify this yourself:\n$ go build -gcflags=\u0026#34;-m\u0026#34; main.go ./main.go:13:12: inlining call to sync.(*Mutex).Lock ./main.go:15:14: inlining call to sync.(*Mutex).Unlock Inlining matters here. It means that in the uncontended case, calling mu.Lock() does not even incur a function call overhead — the CAS instruction is planted directly at the call site. This is why uncontended mu.Lock() is roughly 20-60 nanoseconds: it is essentially one CAS instruction plus a few cycles of bookkeeping.\nIf you remember nothing else from this article, remember this: an uncontended mutex lock is one CAS instruction. That is the same instruction atomic.CompareAndSwapInt32 uses. The cost difference between uncontended mutex and atomic comes down to a few extra cycles of bookkeeping, not a qualitative difference.\n3.3. Slow path: spinning When the fast path CAS fails, the mutex is already locked, and we enter lockSlow. This is where things get interesting.\nThe first thing lockSlow tries is spinning: actively waiting in a tight CPU loop, hoping the current holder releases the lock soon. The rationale is simple — if the holder is going to release the lock in the next few hundred nanoseconds, it is much cheaper to spin and grab it immediately than to park the goroutine and wake it up later (a goroutine park/unpark cycle through the runtime costs on the order of hundreds of nanoseconds to microseconds).\nHere is the relevant slice of lockSlow:\nfunc (m *Mutex) lockSlow() { var waitStartTime int64 starving := false awoke := false iter := 0 old := m.state for { // Don\u0026#39;t spin in starvation mode, ownership is handed off to // waiters so we won\u0026#39;t be able to acquire the mutex anyway. if old\u0026amp;(mutexLocked|mutexStarving) == mutexLocked \u0026amp;\u0026amp; runtime_canSpin(iter) { // Active spinning makes sense. // Set the woken flag to inform Unlock that we are about to need it. if !awoke \u0026amp;\u0026amp; old\u0026amp;mutexWoken == 0 \u0026amp;\u0026amp; old\u0026gt;\u0026gt;mutexWaiterShift != 0 \u0026amp;\u0026amp; atomic.CompareAndSwapInt32(\u0026amp;m.state, old, old|mutexWoken) { awoke = true } runtime_doSpin() iter++ old = m.state continue } // ... try to acquire or queue ourselves ... } } The function runtime_doSpin ultimately calls runtime.procyield(cycles), which on ARM64 expands to this assembly:\nTEXT runtime·procyield(SB),NOSPLIT,$0-0 MOVWU cycles+0(FP), R0 again: YIELD SUBW $1, R0 CBNZ R0, again RET A tight loop that issues the YIELD instruction (a hint to the CPU that this is a spin-wait loop — the core can temporarily release execution resources to its hyperthreaded sibling, or reduce its own priority) and counts down from the cycle count. Go calls procyield(30), so 30 YIELDs per spin iteration.\nruntime_canSpin(iter) caps the spinning: at most 4 spin iterations, and only if GOMAXPROCS \u0026gt; 1 (spinning on a single-core machine is pure waste — there is nothing else to run, so you might as well park), and there is more than one runnable goroutine (so the runtime has something else to do if our spin fails). All told, the worst case is 4 spins × 30 YIELDs = 120 YIELDs before giving up. That is around a few hundred nanoseconds of wall-clock time — short enough that you barely notice, long enough that a holder finishing a small critical section is likely to release the lock in that window.\nAfter spinning, the goroutine builds a new state value (marking itself as a waiter if necessary, possibly marking itself starving if it has been waiting too long), CASes it into place, and if it still cannot acquire the lock, falls through to the slowest path of all: parking.\n3.4. The slowest path: futex and parking When spinning fails, the goroutine needs to actually sleep until the lock is released. This is where futex comes in.\nFutex — short for \u0026ldquo;fast userspace mutex\u0026rdquo; — is a Linux syscall (futex(2)) introduced in Linux 2.6 (2003) by Hubertus Franke, Matthew Kirkwood, Ingo Molnar, and Ulrich Drepper (yes, the same Drepper who wrote \u0026ldquo;What Every Programmer Should Know About Memory\u0026rdquo;). The original paper, \u0026ldquo;Fuss, Futexes and Furwocks: Fast Userlevel Locking in Linux\u0026rdquo; (OLS 2002), is the canonical reference.\nThe idea is brilliant in its simplicity. Most of the time, a lock is uncontended or only briefly contended. In those cases, you can acquire and release it entirely in userspace using CAS, with no kernel involvement. Only when you actually need to sleep (because the lock is held for a long time) do you call into the kernel. The kernel\u0026rsquo;s only job is to maintain a queue of waiters and wake them up at the right moment.\nThe two essential futex operations:\nFUTEX_WAIT(addr, expected): check that *addr == expected. If so, park the calling thread on the queue associated with addr and sleep. If not, return immediately. The check-and-park is atomic — there is no race where the value changes between the check and the sleep. FUTEX_WAKE(addr, n): wake up at most n threads parked on the queue associated with addr. Go\u0026rsquo;s runtime builds its own semaphore system on top of futex (on Linux) or the equivalent syscall on other operating systems (umtx on FreeBSD, futex on OpenBSD, etc.). The function runtime_SemacquireMutex(\u0026amp;m.sema, ...) ultimately parks the goroutine; runtime_Semrelease(\u0026amp;m.sema, ...) wakes one up.\ngraph LR subgraph US[\"Userspace\"] CAS[\"CAS(state, 0, locked)\"] Spin[\"Spin: procyield(30) × 4\"] Slow[\"lockSlow: build new state\"] end subgraph K[\"Kernel\"] Wait[\"FUTEX_WAIT(sema)\"] Queue[\"Waiter queue\"] Wake[\"FUTEX_WAKE(sema)\"] end Lock[\"mu.Lock()\"] --\u003e CAS CAS --\u003e|success| Done[\"done (no syscall)\"] CAS --\u003e|fail — locked| Spin Spin --\u003e|got it| Done Spin --\u003e|still locked| Slow Slow --\u003e Wait Wait --\u003e Queue Queue -.-\u003e|parked, sleeping| SleepZzz[\"...\"] Wake -.-\u003e|from another g's Unlock| Queue Queue --\u003e|woken| CAS style CAS fill:#2e7d32,color:#fff style Done fill:#2e7d32,color:#fff style Wait fill:#c62828,color:#fff style Wake fill:#c62828,color:#fff The cost difference between these paths is enormous. An uncontended CAS is one instruction, a few nanoseconds. A futex round-trip (park and wake) is two syscalls plus context switches, on the order of microseconds. That is a 1000x difference. This is why Mutex tries so hard to stay in userspace: spinning, optimistic CAS, all of it is designed to avoid hitting the kernel.\nThe GMP model: why parking a goroutine is not parking a thread There is a subtlety here that is easy to miss. Futex in Linux operates on kernel threads — FUTEX_WAIT puts the calling thread to sleep. But Go uses an M:N scheduler: many goroutines (M) are multiplexed onto fewer kernel threads (N, by default equal to the number of CPUs). Before looking at the parking mechanics, here are the components of the Go scheduler — the GMP model:\ngraph TB subgraph Gs[\"G — goroutines (thousands)\"] G1[\"G1 running\"] G2[\"G2 runnable\"] G3[\"G3 runnable\"] Gn[\"... Gn\"] end subgraph Ps[\"P — logical processors (GOMAXPROCS)\"] P1[\"P1\nlocal runq: [G2, G3]\"] P2[\"P2\nlocal runq: [G4, G5]\"] end subgraph Ms[\"M — kernel threads (count ≤ GOMAXPROCS + blocked)\"] M1[\"M1 ← P1\"] M2[\"M2 ← P2\"] end subgraph SYS[\"OS kernel\"] SCHED[\"Linux scheduler\nmanages M\"] FUTEX[\"futex syscall\nparks M\"] end G1 -.-\u003e|runs on| M1 G2 -.-\u003e|queued on| P1 G3 -.-\u003e|queued on| P1 M1 ===\u003e|bound to| P1 M2 ===\u003e|bound to| P2 M1 -.-\u003e|syscalls| SCHED M2 -.-\u003e|syscalls| SCHED SCHED --- FUTEX style G1 fill:#2e7d32,color:#fff style P1 fill:#1565c0,color:#fff style P2 fill:#1565c0,color:#fff style M1 fill:#c62828,color:#fff style M2 fill:#c62828,color:#fff style FUTEX fill:#6a1b9a,color:#fff Three letters:\nG (Goroutine) — a user-space goroutine. Lightweight, 2KB starting stack, grows on demand. Hundreds of thousands can coexist. M (Machine) — a kernel thread. Real, heavy, with its own multi-KB stack. Count is bounded — usually number of CPUs plus a few spares for threads blocked in syscalls. P (Processor) — a logical processor. Holds a local queue of runnable goroutines (runq) and the context to execute Go code. Count of P = GOMAXPROCS. For M to run Go code, it needs a P. When G1 on M1/P1 calls mu.Lock() and the mutex is already held, the runtime walks this decision tree:\nflowchart TD Start([G1 calls mu.Lock\nmutex already held]) --\u003e Mark[\"Runtime:\nG1 → _Gwaiting state\nremove from P1's runq\nadd to mutex sema queue\"] Mark --\u003e Check{\"Are there other\nrunnable G on P1?\"} Check --\u003e|yes — e.g. G2, G3| Switch[\"M1 switches to G2\n~200ns, no syscall\nP1 stays acquired\"] Check --\u003e|no| Global{\"Any G in\nglobal runq?\"} Global --\u003e|yes| Take[\"M1 takes G from globalq\nor steals from another P\"] Global --\u003e|no| ParkM[\"M1 releases P1\nanother M can grab it\nM1 calls FUTEX_WAIT\nkernel thread sleeps\"] Switch --\u003e Work[\"G2 executes\"] Take --\u003e Work ParkM --\u003e Kernel[\"... kernel keeps M1 parked\nin futex queue\"] Kernel -.-\u003e|\"somewhere else:\nGx releases the mutex\"| Wake Wake[\"runtime_Semrelease\nG1 → _Grunnable\nplaced on some P's runq\"] Wake --\u003e Rerun[\"G1 back in queue\nwaits for its M/P\nand resumes\"] style Start fill:#c62828,color:#fff style Switch fill:#2e7d32,color:#fff style Take fill:#2e7d32,color:#fff style ParkM fill:#ad1457,color:#fff style Wake fill:#1565c0,color:#fff The crucial check is the first one: \u0026ldquo;are there other runnable G on P1?\u0026rdquo;. In a typical Go program the answer is almost always yes — there are a few more goroutines in the local queue. So M1 does not sleep; it just switches to G2. This context switch happens entirely in user-space (hundreds of nanoseconds), without entering the kernel.\nOnly if the local queue is empty and the global queue is empty and nothing can be stolen from other Ps — only then M1 calls FUTEX_WAIT and sleeps at the kernel level. That is the optimization that separates Go from C++/Java: instead of parking a heavy kernel thread (with an OS context switch, on the order of microseconds), we park a lightweight goroutine and reuse the kernel thread for other work.\nPlatform What gets parked under contention Cost Scale C++ std::mutex Kernel thread (via futex) ~1-5 µs (syscall + ctx switch) ≤ thousands of threads Java synchronized Kernel thread (via JVM monitor) ~1-5 µs ≤ thousands of threads Go sync.Mutex Goroutine (in user-space) ~100-200 ns hundreds of thousands of goroutines This is why Go can have hundreds of thousands of goroutines waiting on mutexes without consuming hundreds of thousands of kernel threads — that would be a memory disaster (each kernel thread has at least a few KB of stack) and a scheduler disaster (the OS would have to traverse a huge thread queue).\nThe cost of \u0026ldquo;parking a goroutine\u0026rdquo; in Go is not the cost of a syscall, but the cost of a user-space context switch, on the order of a hundred nanoseconds. The FUTEX_WAIT syscall only happens when the kernel thread genuinely has no other work. This is another reason why mu.Lock() under contention in Go is often faster than the equivalent in C++/Java — we pay only for what we actually use.\nFor more on the Go scheduler, see Dmitry Vyukov\u0026rsquo;s \u0026ldquo;Scalable Go Scheduler Design Doc\u0026rdquo; (May 2012, still the basis of runtime/sched).\n3.5. Unlock: symmetric, with a handoff surprise Unlock looks symmetric to Lock, but with one twist:\nfunc (m *Mutex) Unlock() { // ... race detector bookkeeping ... // Fast path: drop lock bit. new := atomic.AddInt32(\u0026amp;m.state, -mutexLocked) if new != 0 { // Outlined slow path to allow inlining the fast path. m.unlockSlow(new) } } The fast path is atomic.AddInt32(\u0026amp;m.state, -mutexLocked) — one atomic subtract instruction, clearing the locked bit. If the resulting state is zero (no waiters, no flags), we are done — no syscall, no wake-up. This is why uncontended unlock, like uncontended lock, is just a few nanoseconds.\nIf there are waiters, unlockSlow runs, and its behavior depends on whether the mutex is in starvation mode:\nfunc (m *Mutex) unlockSlow(new int32) { if (new+mutexLocked)\u0026amp;mutexLocked == 0 { fatal(\u0026#34;sync: unlock of unlocked mutex\u0026#34;) } if new\u0026amp;mutexStarving == 0 { // Normal mode. old := new for { // If there are no waiters, or someone else already woke one, // or someone grabbed the lock, we don\u0026#39;t need to wake anyone. if old\u0026gt;\u0026gt;mutexWaiterShift == 0 || old\u0026amp;(mutexLocked|mutexWoken|mutexStarving) != 0 { return } // Grab the right to wake someone. new = (old - 1\u0026lt;\u0026lt;mutexWaiterShift) | mutexWoken if atomic.CompareAndSwapInt32(\u0026amp;m.state, old, new) { runtime_Semrelease(\u0026amp;m.sema, false, 2) return } old = m.state } } else { // Starving mode: hand off mutex ownership directly // to the next waiter, and yield our time slice so that // the next waiter can start to run immediately. // Note: mutexLocked is not set, the waiter will set it after wakeup. runtime_Semrelease(\u0026amp;m.sema, true, 2) } } Look at the last argument to runtime_Semrelease: false in normal mode, true in starvation mode. That boolean is the handoff flag. When true, the runtime directly transfers ownership of the mutex to the woken waiter — the waiter wakes up already owning the lock, with the locked bit pre-set. When false, the woken waiter has to compete for the lock like everyone else.\nThis single boolean is the entire point of starvation mode. To understand why it matters, we need to look at the bug it was invented to fix.\n3.6. Starvation mode: the bug behind issue #13086 On October 28, 2015, Russ Cox opened Go issue #13086 with the title \u0026ldquo;runtime: fall back to fair locks after repeated sleep-acquire failures.\u0026rdquo; The opening line was blunt: \u0026ldquo;Go\u0026rsquo;s locks make no guarantee of fairness.\u0026rdquo;\nThe bug report described a simple two-goroutine program. Goroutine 1 holds the lock almost all the time, releasing it for only 100 microseconds at a stretch. Goroutine 2 wants the lock only briefly, every 100 microseconds. The naive expectation is that goroutine 2 should get the lock once in a while, maybe within a second or two.\nThe reality, on Russ\u0026rsquo;s Linux workstation, was that goroutine 2 took 100 to 600 seconds to acquire the lock even once. Not milliseconds. Seconds. Minutes. Ten minutes for a single lock acquisition.\nRuss\u0026rsquo;s analysis identified the problem precisely. When goroutine 1 calls Unlock, it marks the lock unlocked and tells the runtime \u0026ldquo;wake up goroutine 2.\u0026rdquo; But goroutine 2 does not run immediately. Goroutine 1 keeps running, goes around its loop, calls Lock again — and because the lock is now unlocked and goroutine 1 is already on-CPU, goroutine 1 grabs it back. By the time goroutine 2 actually gets scheduled and tries to acquire, goroutine 1 has the lock again. The pattern repeats. Millions of times.\nRuss named the problem barging. The alternative, where Unlock keeps the lock locked and explicitly transfers ownership to the woken waiter, he called handoff. In Doug Lea\u0026rsquo;s earlier work on java.util.concurrent (AQS paper, 2003), barging was found to improve throughput. Lea\u0026rsquo;s measurements were on operating system threads with the Linux 2.4 NPTL scheduler, and his argument was that barging avoids bad OS scheduling decisions: if the OS is slow to schedule the woken thread, leaving the lock unlocked lets another thread do useful work in the meantime.\nBut Go does not use OS threads directly. It uses goroutines and its own user-space scheduler, where a goroutine switch costs tens of nanoseconds, not the microseconds of a thread context switch. The trade-off that justified barging in Java did not necessarily apply to Go.\nRuss proposed a hybrid: stay in barging mode (faster) most of the time, but fall back to handoff when severe unfairness is detected. The detection mechanism: a waiter that gets woken up but finds the lock unavailable (because some other goroutine barged in) increments a counter. After enough consecutive failures, the mutex switches to handoff mode.\ngraph LR subgraph Normal[\"Normal mode (default)\"] N1[\"Unlock: mark unlocked\"] --\u003e N2[\"Wake waiter W\"] N2 --\u003e N3[\"Continue running\"] N3 --\u003e N4[\"Another g calls Lock\"] N4 --\u003e N5[\"Barge! Acquire lock\"] N2 -.-\u003e|W scheduled too late| NLost[\"W wakes, finds lock taken\"] NLost --\u003e N1 end subgraph Starvation[\"Starvation mode (after 1ms wait)\"] S1[\"Unlock: keep locked\"] --\u003e S2[\"Hand off to next waiter W\"] S2 --\u003e S3[\"Yield time slice\"] S3 --\u003e S4[\"W wakes owning the lock\"] S4 --\u003e S5[\"W runs critical section\"] S5 --\u003e S1 end Normal -.-\u003e|\"waiter waited \u003e 1ms\"| Starvation Starvation -.-\u003e|\"last waiter OR waited \u003c 1ms\"| Normal style Normal fill:#1565c015 style Starvation fill:#c6282815 style N5 fill:#c62828,color:#fff style S4 fill:#2e7d32,color:#fff The implementation that landed in Go 1.9 (August 2017), authored by Dmitry Vyukov, was slightly different in detail but the same in spirit. Instead of counting failures, it tracks wall-clock wait time. If a goroutine has been waiting longer than starvationThresholdNs = 1e6 (1 millisecond) to acquire the lock, it sets the mutexStarving bit. In starvation mode:\nNew arrivals do not try to acquire the lock. They go straight to the back of the waiter queue. Unlock hands off ownership directly (the true we saw in runtime_Semrelease(\u0026amp;m.sema, true, 2)), and yields the time slice so the woken waiter runs immediately. Spinning is disabled — it would be useless since the lock is being handed off, not barged. The mutex exits starvation mode when either of two conditions is met:\nThe current waiter is the last one in the queue (waiter count would drop to 0 after this acquisition), or The current waiter has waited less than 1ms in this round. This is a self-correcting mechanism. When contention drops, the mutex naturally falls back to fast barging mode. When a pathological workload causes starvation, it switches to handoff mode just long enough to drain the queue fairly.\nThe impact, measured by Russ on his lockskew benchmark, was a 500000x speedup in the pathological case — from 100+ seconds per acquisition down to 162 microseconds. And on the common-case throughput benchmark (random number generation under heavy contention), performance actually improved slightly (1-12% faster), contradicting the Java-era wisdom that handoffs always hurt throughput. Goroutine scheduling is cheap enough that the handoff cost is invisible against the rest of the work.\nThis is why your Go code almost never starves: there is a 1ms threshold, sitting quietly inside sync.Mutex, defending you against a class of bugs that took real production systems down in 2015.\n3.7. The rules: don\u0026rsquo;t copy, don\u0026rsquo;t reenter Two operational rules that catch people out:\nNever copy a Mutex. This:\ntype Server struct { mu sync.Mutex // ... } func (s Server) Handle(req Request) { // BUG: receiver is by value s.mu.Lock() defer s.mu.Unlock() // ... } …compiles, but is a bug. The method receiver is a copy of Server, which means s.mu is a copy of the original mutex — a fresh, unlocked mutex with no connection to the original\u0026rsquo;s state or sema. Two goroutines calling Handle on the same Server will each lock their own private copy, providing zero mutual exclusion. go vet catches this:\n$ go vet ./server.go:12:20: Handle passes lock by value: Server contains sync.Mutex The fix: make the receiver a pointer (func (s *Server) Handle(...)), or embed *sync.Mutex instead of sync.Mutex.\nThe same applies to types that embed a Mutex — sync.WaitGroup, sync.RWMutex, anything containing one. The general rule: a value containing a Mutex must not be copied after first use.\nMutexes are not reentrant. Calling Lock twice from the same goroutine deadlocks:\nfunc (s *Server) Outer() { s.mu.Lock() defer s.mu.Unlock() s.Inner() // BUG: Inner calls s.mu.Lock again — deadlock } func (s *Server) Inner() { s.mu.Lock() defer s.mu.Unlock() // ... } The second Lock blocks forever waiting for the first Unlock, which will never come because the goroutine is stuck in Lock. There is no recursive mutex in Go. The language designers have repeatedly rejected proposals for one, on the grounds that reentrancy encourages sloppy thinking: if a function takes a lock it already holds, that is usually a sign that the locking discipline is unclear, and papering over it with a recursive mutex hides the real problem.\nThe fix is structural: split Inner into two functions, one that assumes the lock is held and one that takes it. Or document the locking discipline explicitly. Or restructure so the same goroutine never needs to acquire the same lock twice.\n4. When to Use What You now understand the stack. The remaining question is practical: given a piece of code, do you reach for atomic, or for mutex, or for something else? Let\u0026rsquo;s work through it.\n4.1. Public benchmarks: the numbers to remember Most of the published Go mutex-vs-atomic benchmarks cluster around the same numbers. From thecodinggopher\u0026rsquo;s benchmarks, goperf.dev, and others, on modern x86 hardware:\nOperation Uncontended Under contention atomic.AddInt64(\u0026amp;x, 1) 4-8 ns tens of ns to µs (CAS retries) atomic.LoadInt64(\u0026amp;x) 1-2 ns 1-2 ns (read-only) mu.Lock(); mu.Unlock() 20-60 ns hundreds of ns to µs mu.Lock(); work; mu.Unlock() (short cs) ~30 ns + work scales with contention CAS loop under heavy contention — microseconds of CPU burn The 5-10x gap between uncontended atomic and uncontended mutex shows up reliably. It comes from the bookkeeping we saw in Lock: even the fast path has to set the woken flag, check waiters, manage the state field, not just CAS the value.\nUnder contention, the picture inverts in a subtle way. A CAS loop under heavy contention can burn hundreds of nanoseconds to microseconds of CPU per operation, because every failed CAS is wasted work. A mutex, by parking the loser, lets the winner proceed without being repeatedly interrupted. For sustained contention, mutex is often faster than naive atomic.\n4.2. When atomic is right Use atomic when the operation is on a single memory location and you can express it as one of the atomic primitives.\nUse case Recommendation Counter (requests/sec, errors/sec) atomic.Int64.Add(1) Done/started flag atomic.Bool.Store(true) / .Load() Single-word state (idle/active/stopped) atomic.Int32 or atomic.Uint32 Pointer to immutable snapshot atomic.Pointer[T] Sequence number atomic.Uint64.Add(1) If your hot path is a counter being incremented from many goroutines, atomic is 5-10x faster than a mutex and just as correct. Do not put a mutex around a counter \u0026ldquo;for safety\u0026rdquo; — it is slower for no benefit.\nThe trickier case is when you want a more complex atomic operation, like \u0026ldquo;atomically update two counters together.\u0026rdquo; That is not directly expressible as a single atomic primitive. You have two options:\nPack both values into a single uint64 using bit shifts, and CAS the combined value. Use a mutex. Packing is faster but limits you to 64 bits total. Mutex is simpler and works for any size. Most of the time, mutex is the right answer here.\n4.3. When mutex is right Use mutex when the operation spans multiple memory locations, or when the operation needs to maintain an invariant across multiple fields.\nThe classic example: updating a map.\ntype Cache struct { mu sync.Mutex items map[string]*Entry size int } func (c *Cache) Put(k string, e *Entry) { c.mu.Lock() defer c.mu.Unlock() if _, ok := c.items[k]; !ok { c.size++ } c.items[k] = e } You cannot atomically update items and size together. They are separate fields. If you tried to use atomic for each, another goroutine could observe size updated but items not, or vice versa. The mutex guarantees that the entire Put operation is atomic with respect to other goroutines: they either see the state before, or the state after, never a mix.\nThis generalizes: any operation that needs to read-modify-write multiple fields while maintaining an invariant across them requires a mutex. Atomics do not help here, because they only synchronize a single word.\n4.4. atomic.Value vs RWMutex for read-heavy data For read-heavy data — configuration, feature flags, lookup tables — there are two common patterns:\nPattern A: atomic.Pointer[T] (copy-on-write).\nvar config atomic.Pointer[Config] func Get() *Config { return config.Load() } func Update(c *Config) { config.Store(c) } Reads are one atomic load — a few nanoseconds, no contention ever. Writes require building a complete new *Config and storing it atomically; the old pointer is garbage-collected once the last reader finishes with it.\nPattern B: sync.RWMutex.\nvar ( mu sync.RWMutex config *Config ) func Get() *Config { mu.RLock() defer mu.RUnlock() return config } func Update(c *Config) { mu.Lock() defer mu.Unlock() config = c } Reads take a read lock, which is cheaper than a write lock but not free (it still does an atomic add on the reader count, and writes have to wait for all readers to release).\nWhich is faster? It depends on the read/write ratio:\nMillions of reads per second, a few writes per minute: atomic wins decisively. Reads are 5-10x cheaper, and there is no write-side contention to worry about. Many writes per second, with the data being large: mutex may win, because building a complete new copy on every write is expensive. You need to mutate the data in place (not replace it wholesale): mutex, no question. atomic.Value/Pointer only works for wholesale replacement. For most configuration-style use cases, atomic.Pointer[T] is the right answer.\n4.5. False sharing: when independent atomics fight Here is a subtle performance bug. Suppose you have two counters that are updated independently from different goroutines:\ntype Stats struct { Requests int64 Errors int64 } var stats Stats // goroutine A: atomic.AddInt64(\u0026amp;stats.Requests, 1) // goroutine B: atomic.AddInt64(\u0026amp;stats.Errors, 1) Looks fine. The two counters are independent. They should scale linearly across cores.\nThey do not. On a modern CPU, Stats is 16 bytes, which fits comfortably in one cache line (typically 64 bytes). When goroutine A on core 0 does an atomic add to Requests, the CPU invalidates the entire cache line on all other cores. When goroutine B on core 1 does an atomic add to Errors, it has to reload the cache line, do the add, and invalidate it on core 0. The two goroutines ping-pong the cache line back and forth, even though they touch different bytes.\nThis is false sharing, and it can slow down parallel code by 5-10x.\nThe fix is padding: insert unused bytes between the hot fields so they land on different cache lines.\ntype Stats struct { Requests int64 _ [56]byte // pad to 64 bytes Errors int64 } Now Requests and Errors are guaranteed to be on different cache lines (assuming 64-byte lines, which is the case on essentially all modern x86 and ARM cores), and the two goroutines can update them independently without invalidating each other\u0026rsquo;s cache.\nThis is a Level 1 (hardware) concern, but it shows up in Level 5 (practice). Most Go code never needs to worry about it. But if you are writing a high-throughput metrics collector or a hot lock-free data structure, false sharing is one of the first things to look for.\n4.6. The double-checked locking trap Here is a classic concurrency bug, in a singleton initializer:\nvar ( instance *Config mu sync.Mutex ) func GetConfig() *Config { if instance == nil { // BUG: non-atomic read mu.Lock() defer mu.Unlock() if instance == nil { instance = loadConfig() // BUG: non-atomic write } } return instance } The \u0026ldquo;double-check\u0026rdquo; is the two instance == nil tests. The idea is to avoid taking the mutex on the common path (after initialization), while still being thread-safe during initialization.\nThis is broken in two ways:\nThe first instance == nil is a non-atomic read. The Go memory model does not guarantee it observes the write done under the mutex. The compiler and CPU are free to reorder, cache, or otherwise misbehave. Even if the read worked, instance = loadConfig() is a non-atomic write. Another goroutine doing the first check might observe a partially constructed *Config. The right answer in Go is sync.Once:\nvar ( instance *Config once sync.Once ) func GetConfig() *Config { once.Do(func() { instance = loadConfig() }) return instance } sync.Once uses atomic operations internally to make the fast path (after initialization) lock-free, and it correctly handles all the memory ordering issues. It is the canonical answer to \u0026ldquo;lazy initialization in a concurrent context.\u0026rdquo;\nIf for some reason you cannot use sync.Once, the correct hand-rolled version uses atomic load and store:\nvar instanceP atomic.Pointer[Config] func GetConfig() *Config { if c := instanceP.Load(); c != nil { return c } // ... take mutex, double-check with Load, build, Store ... } The atomic Load and Store provide the memory ordering guarantees that plain reads and writes do not.\n4.7. Cheatsheet Here is the one-page summary. When you are staring at a piece of code and asking \u0026ldquo;atomic or mutex?\u0026rdquo;, run through this table.\nScenario Recommendation Why Counter (requests++) atomic.Int64.Add(1) 5-10x faster than mutex, just as correct Boolean flag (stopped) atomic.Bool One-word state, no invariant to protect Single-word state machine atomic.Int32 or atomic.Uint32 Encode states as small integers Pointer to immutable snapshot atomic.Pointer[T] Lock-free reads, atomic replacement Update map + counter together sync.Mutex Multi-field invariant requires critical section Update slice in place sync.Mutex Slice header + backing array, multi-word Read-heavy config (rare updates) atomic.Pointer[T] or sync.RWMutex atomic.Pointer if you can replace wholesale; RWMutex if you mutate in place Per-goroutine state No synchronization If only one goroutine touches it, no race Lazy singleton init sync.Once Hand-rolled double-checked locking is wrong Hot path with two independent counters atomic.Int64 × 2 with padding False sharing if struct-packed If you remember nothing else from this article, remember this: single-word state — atomic. Multi-field invariant — mutex. Anything else — measure and think.\n5. Conclusion We have traveled the whole stack. From counter++ being three ARM64 instructions, through CAS as the universal primitive (Herlihy 1991), through LOCK CMPXCHG and ARM LSE, through Go\u0026rsquo;s sync/atomic package with its single sequentially-consistent memory ordering, through futex as the bridge between userspace and kernel, through sync.Mutex\u0026rsquo;s packed state field with its spinning and starvation mode, and finally to the practical recommendations.\nThe two main takeaways:\nMutex and atomic are one stack, not two tools. Mutex is built on atomic CAS, with a fairness layer on top. Uncontended mu.Lock() is one CAS instruction — barely more expensive than atomic.CompareAndSwapInt32. The cost difference shows up only under contention, and that is where mutex\u0026rsquo;s parking (instead of busy-spinning) actually wins. Starvation mode is not theoretical. It exists because real production code was hitting 100-second lock acquisition times in 2015 (issue #13086). The 1ms threshold inside sync.Mutex is silently defending your code against that pathology, every day. If you want to dig deeper, the references below are the canonical reading list. The Go source files (sync/mutex.go, runtime/sema.go) are readable, well-commented, and not as scary as they look. Russ Cox\u0026rsquo;s memory model revision notes and the issue #13086 discussion are the two pieces of historical context that explain why the code looks the way it does.\nThe next article in the Go Internals series will look at sync.Pool — which builds on sync/atomic and per-P local storage to give Go one of the cheapest object allocation strategies of any mainstream language. Stay tuned.\nReferences Go Memory Model reference — go.dev/ref/mem Go 1.19 release notes (atomic types, memory model revision) — go.dev/doc/go1.19 Russ Cox, \u0026ldquo;Go memory model\u0026rdquo; revision notes — research.swtch.com/gomm Go issue #13086 — \u0026ldquo;runtime: fall back to fair locks after repeated sleep-acquire failures\u0026rdquo; — github.com/golang/go/issues/13086 Hans-J. Boehm, Sarita V. Adve, \u0026ldquo;Foundations of the C++ Concurrency Memory Model,\u0026rdquo; PLDI 2008 — dl.acm.org/doi/10.1145/1375581.1375591 Maurice Herlihy, \u0026ldquo;Wait-Free Synchronization,\u0026rdquo; ACM TOPLAS 13(1), 1991 — cs.brown.edu/people/mph/Herlihy91/p124-herlihy.pdf Doug Lea, \u0026ldquo;The java.util.concurrent Synchronizer Framework\u0026rdquo; (AQS paper), 2003 — gee.cs.oswego.edu/dl/papers/aqs.pdf Hubertus Franke, Matthew Kirkwood, Ingo Molnar, Ulrich Drepper, \u0026ldquo;Fuss, Futexes and Furwocks: Fast Userlevel Locking in Linux,\u0026rdquo; OLS 2002 — kernel.org/doc/ols/2002/ols2002-pages-479-494.pdf Linux futex(2) man page — man7.org/linux/man-pages/man2/futex.2.html Ulrich Drepper, \u0026ldquo;What Every Programmer Should Know About Memory,\u0026rdquo; 2007 — people.freebsd.org/~lstewart/articles/cpumemory.pdf Go sync/mutex.go source (Go 1.24) — cs.opensource.google/go/go/+/refs/tags/go1.24.0:src/internal/sync/mutex.go VictoriaMetrics blog, \u0026ldquo;Go sync.Mutex: Normal and Starvation Mode\u0026rdquo; — victoriametrics.com/blog/go-sync-mutex goperf.dev, \u0026ldquo;Atomic Operations and Synchronization Primitives\u0026rdquo; — goperf.dev/01-common-patterns/atomic-ops \u0026ldquo;Mutex or Atomics? Choosing the Right Tool in Go\u0026rdquo; — thecodinggopher.substack.com/p/mutex-or-atomics-choosing-the-right ","permalink":"https://triumphpc.github.io/blog/posts/sync-mutex-atomic/","summary":"Mutex and atomic are not two independent tools — they are one stack of abstractions. I dig through the stack from data race to starvation mode: why x++ is three ARM64 instructions, how CAS became a universal primitive (Herlihy 1991), why Mutex is built on top of atomic CAS, and how Russ Cox\u0026rsquo;s barging-vs-handoff debate (issue #13086) gave Go its 1ms starvation threshold. With public benchmarks, the bit layout of Mutex state, and a cheatsheet for when to reach for which.","title":"Go Internals: sync.Mutex and sync/atomic — From Data Race to Starvation Mode"},{"content":"In Part 5 I surveyed approaches to agent memory: from OpenClaw\u0026rsquo;s file-based brain to mem0\u0026rsquo;s managed layer. We chose a hybrid — MemPalace-style memory layers, OpenClaw-style file structure. But a survey is one thing, and engineering mechanics is quite another. How exactly do files end up in context? Why are some always injected and others only on demand? How does the agent even know it needs to read something? And what happens when memory turns into a dumpster — who cleans it up?\nThis article is a deep dive into the mechanics. Five memory files, four context assembly strategies, TOOLS.md with two sources of responsibility, and — the most interesting part — Dreaming: background memory consolidation that Anthropic and OpenAI both shipped in 2026. I\u0026rsquo;ll show the engineering decisions made behind the scenes, why we chose this particular path, and when we\u0026rsquo;ll need to change it.\n1. Five Memory Files: Complete Catalog In the first part I described four memory layers and mentioned SOUL.md, ABOUT.md, TOOLS.md. But the real file model is richer. Five files, each with its own responsibility zone, its own way of entering context, and its own isolation scope.\n1.1. Exhaustive Table # File Purpose Scope When it enters context How Who writes 1 SOUL.md Agent identity and personality: values, communication style, persona agent Always (every request) Bootstrap injection into system prompt User via UI / Agent via memory tools 2 ABOUT.md Agent profile and description: role, specialization, tasks agent Always (every request) Bootstrap injection into system prompt User via UI / Agent via memory tools 3 TOOLS.md User notes on tools: quirks, examples, limitations agent Always (every request) Bootstrap injection (merged with auto-generated) User via UI / Agent via memory tools. Lazy lifecycle: file is not created by default 4 USER.md User profile: name, preferences, work context — for a specific agent agent_user On demand (agent decides) Through memory tools Agent via memory tools / User via chat 5 MEMORY.md Persistent memory: facts, decisions, dialog context for a specific agent+user pair agent_user On demand (agent decides) Through memory tools Agent via memory tools (auto-capture) The key distinction is injection vs on-demand. Three files (SOUL, ABOUT, TOOLS) are injected into the system prompt on every request. This is \u0026ldquo;hot\u0026rdquo; memory — the agent always sees its identity and tool descriptions. Two files (USER, MEMORY) are on-demand, through memory tools. The agent decides for itself when it needs user context.\ngraph TB subgraph \"Bootstrap Injection (~1-2K tokens, EVERY request)\" SOUL[\"SOUL.md\nidentity\"] ABOUT[\"ABOUT.md\nprofile\"] TOOLS[\"TOOLS.md\ntool notes\"] end subgraph \"On-demand (via memory tools, agent decides)\" USER[\"USER.md\nuser profile\"] MEMORY[\"MEMORY.md\nlong-term memory\"] end SOUL --\u003e SP[\"System Prompt\"] ABOUT --\u003e SP TOOLS --\u003e SP SP --\u003e LLM[\"LLM\"] LLM --\u003e|\"memory tools\"| USER LLM --\u003e|\"memory tools\"| MEMORY style SOUL fill:#6a1b9a,color:#fff style ABOUT fill:#6a1b9a,color:#fff style TOOLS fill:#6a1b9a,color:#fff style USER fill:#e65100,color:#fff style MEMORY fill:#e65100,color:#fff 1.2. SOUL.md — Identity SOUL.md is \u0026ldquo;who I am.\u0026rdquo; Values, personality, communication style. Limit — 2000 characters. Injected into every request as the ## Agent Identity section in the system prompt.\nWhy not more? Because every character in SOUL.md is paid for on every request. 2000 characters ≈ 500 tokens. If you increase it to 10K — the agent will spend 2.5K tokens just on its own identity before saying a single word. For a platform with hundreds of agents, that\u0026rsquo;s unacceptable.\nWho writes it? The user via UI — setting the personality at agent creation. Or the agent itself via memory tools — when the user asks \u0026ldquo;be more formal\u0026rdquo; or \u0026ldquo;use Russian by default.\u0026rdquo;\n1.3. ABOUT.md — Profile ABOUT.md is \u0026ldquo;what I do.\u0026rdquo; Role, specialization, typical tasks. Same limit — 2000 characters, same injection on every request.\nWhy a separate file instead of merging with SOUL? The identity/profile split isn\u0026rsquo;t accidental. Identity (SOUL) is a constant — it rarely changes. Profile (ABOUT) is more dynamic: \u0026ldquo;right now I\u0026rsquo;m helping with gRPC migration\u0026rdquo; → a month later → \u0026ldquo;helping set up monitoring.\u0026rdquo; Separate files let you update the profile without touching the core personality.\n1.4. USER.md — User Profile for a Specific Agent Now things get interesting. USER.md is stored in the agent_user scope — meaning each agent has its own profile for each user. Agent A knows that \u0026ldquo;Alice prefers YAML\u0026rdquo; — but Agent B in the same project doesn\u0026rsquo;t, until it reads it itself.\nWhat\u0026rsquo;s stored:\nCategory Examples Identification Name, timezone, language, role Preferences \u0026ldquo;Answer concisely\u0026rdquo;, \u0026ldquo;use YAML\u0026rdquo;, \u0026ldquo;table format\u0026rdquo; Work context \u0026ldquo;Works with Python, uses FastAPI\u0026rdquo;, \u0026ldquo;PM of project X\u0026rdquo; Tools \u0026ldquo;Primary calendar — Google\u0026rdquo;, \u0026ldquo;Todoist for tasks\u0026rdquo; Standing notes \u0026ldquo;Standup at 9:00 Mon-Fri\u0026rdquo;, \u0026ldquo;Avoid Friday meetings\u0026rdquo; Limit — 2000 characters. And here\u0026rsquo;s why USER.md is on-demand rather than injection: not all agents need a user profile. A coding assistant — sure, it matters that you prefer Go. But a text summarization agent — it doesn\u0026rsquo;t care about your timezone. Why spend 500 tokens on a profile the agent won\u0026rsquo;t use?\n1.5. MEMORY.md — Long-Term Memory MEMORY.md is \u0026ldquo;what I know.\u0026rdquo; Key decisions, client context, current tasks, conclusions from conversations. Scope agent_user — tied to the agent+user pair. Limit — 5000 characters (soft limit — when exceeded, the agent receives an instruction to compress).\nMEMORY.md implements the write-manage-read pattern I described in the first part:\nWrite (who saves and when):\nAgent — via memory tools, when it identifies an important fact Auto-capture (future): before compaction/session end — automatic flush of important facts Manage (compression, deduplication):\nAgent receives instruction: \u0026ldquo;Regularly compress MEMORY.md, remove outdated facts\u0026rdquo; If MEMORY.md \u0026gt; 5000 characters — instruction to the agent to compress Future: automatic deduplication via LLM Read (when it\u0026rsquo;s loaded):\nAgent reads via memory tools on demand Instruction in system prompt: \u0026ldquo;search memory before acting\u0026rdquo; (OpenClaw pattern) NOT injected into system prompt on boot (token economy) Why is MEMORY.md on-demand rather than injection? Because MEMORY.md grows. Today — 500 characters, a month later — 5000. If injected — every request will spend more and more tokens on memory. OpenClaw solves this with truncation (cuts off \u0026gt; 20K characters), but truncation means data loss. We prefer the agent to decide for itself what it needs from memory right now.\n1.6. Scope Summary ┌──────────────────────────────────────────────────────────┐ │ AGENT MEMORY LAYERS │ ├──────────────┬──────────────┬───────────────┬───────────┤ │ Scope │ Files │ Loading │ │ ├──────────────┼──────────────┼───────────────┤ │ │ Agent │ SOUL.md │ Always │ │ │ │ ABOUT.md │ Always │ Injection │ │ │ TOOLS.md │ Always │ │ ├──────────────┼──────────────┼───────────────┤ │ │ Agent+User │ USER.md │ On demand │ On-demand │ │ │ MEMORY.md │ On demand │ (tools) │ └──────────────┴──────────────┴───────────────┴───────────┘ Agent receives Prompt Hint in system prompt: \u0026#34;You have memory tools. Load USER.md and MEMORY.md when needed.\u0026#34; Two scopes, two isolation levels. agent — shared agent files, visible to all users. agent_user — personal memory, isolated per agent+user pair. No context leakage between users.\n2. Context Assembly Strategy: Why Hybrid v2 In the first part I said we chose a hybrid strategy — injection for hot context, tools for the rest. But I didn\u0026rsquo;t cover which alternatives we considered and rejected. And that\u0026rsquo;s perhaps the most important architectural decision in the entire memory system.\n2.1. Option 1: Full Injection (like OpenClaw) All memory files are read from storage and injected into the system prompt on every request.\ngraph LR S3[(Storage)] --\u003e|\"read all files\"| ASM[Context Assembler] ASM --\u003e SP[SystemPrompt] SP --\u003e LLM[LLM] subgraph \"Always injected (~5-10K tokens)\" SOUL[\"SOUL.md\"] ABOUT[\"ABOUT.md\"] USER[\"USER.md\"] MEMORY[\"MEMORY.md\"] TOOLS[\"TOOLS.md\"] end SOUL --\u003e ASM ABOUT --\u003e ASM USER --\u003e ASM MEMORY --\u003e ASM TOOLS --\u003e ASM Pros: simple implementation (~30-50 lines), predictable context, easy to debug.\nCons: 5-10K tokens per request, multiple storage reads, MEMORY.md grows → truncation → data loss, agent can\u0026rsquo;t update files through tools.\nOpenClaw does exactly this. And it works — for a coding assistant with one user. But on a platform with hundreds of agents where every request costs money, burning 10K tokens on boot is a luxury.\n2.2. Option 2: On-Demand (Tool-Based) Memory files are NOT injected. The agent receives memory tools and decides for itself what to read.\nPros: 0 tokens from memory on boot, agent can update files, scales without limit.\nCons: agent may \u0026ldquo;forget\u0026rdquo; to read context → poor responses. Every read = one react-loop iteration. Harder to debug — context depends on agent behavior.\nThis is the opposite extreme from Full Injection. Cheap at start, but unreliable. An agent without identity is like a person without a name — they can work, but they don\u0026rsquo;t know who they are.\n2.3. Option 3: Hybrid v1 — L0/L1 Boot + L2 Tools L0 (SOUL, ABOUT) — always injected. L1 (USER) — also injected. L2+ (MEMORY) — through tools.\nPros: ~2-3K on boot, predictable base, close to MemPalace pattern.\nCons: USER.md burns tokens even when the agent doesn\u0026rsquo;t need a user profile. On a platform with 50+ agent types, far from all need USER.md — a summarization agent, a translator, a code reviewer don\u0026rsquo;t interact with users personally.\n2.4. Option 3b: Hybrid v2 — Minimal Injection + Prompt Hint ← chosen The key insight from studying OpenClaw: even OpenClaw doesn\u0026rsquo;t inject MEMORY.md into the system prompt. If memory tools are available, the agent gets a hint \u0026ldquo;use memory_search\u0026rdquo; and loads MEMORY.md itself. USER.md in the Codex integration isn\u0026rsquo;t always injected either.\nHence — Hybrid v2: minimal injection (SOUL + ABOUT + TOOLS) + prompt hint directing the agent to memory tools for USER.md and MEMORY.md.\ngraph TD subgraph \"System Prompt Injection (~1-2K tokens)\" SOUL2[\"SOUL.md / ABOUT.md\nagent identity\"] TOOLS2[\"TOOLS.md\ntool notes\"] AUTO[\"Auto-generated tools\ntool descriptions\"] end subgraph \"On-demand via memory tools\" USER2[\"USER.md\nuser profile\"] MEMORY2[\"MEMORY.md\nlong-term memory\"] end SOUL2 --\u003e SP2[\"System Prompt\"] TOOLS2 --\u003e SP2 AUTO --\u003e SP2 SP2 --\u003e LLM2[\"LLM\"] LLM2 --\u003e|\"memory tools\"| USER2 LLM2 --\u003e|\"memory tools\"| MEMORY2 Prompt Hint (added to system prompt):\nYou have access to memory tools. Load USER.md (user profile) and MEMORY.md (long-term memory) when needed.\n2.5. Comparing the Four Options Criterion Full Injection On-Demand Hybrid v1 Hybrid v2 Boot tokens ~5-10K 0 ~2-3K ~1-2K Context predictability ✅ All files always ❌ Agent may forget ✅ L0/L1 always ⚠️ L1 via prompt hint Agent updates files ❌ API only ✅ Via tools ⚠️ L2+ via tools ✅ All via tools Storage reads on boot 5+ 0 3-4 2-3 Scalability ❌ MEMORY.md grows ✅ ⚠️ USER.md burns tokens ✅ Implementation complexity Simple Medium Complex Medium Why not Hybrid v1? Because USER.md in injection is tokens wasted when the agent doesn\u0026rsquo;t need a profile. On a platform with dozens of agent types (summarizer, translator, code analyst), far from every agent interacts with the user personally. Why load USER.md into every request?\nBottom line: Hybrid v2 — minimal token budget on boot, maximum flexibility, consistent with how OpenClaw actually works (doesn\u0026rsquo;t inject MEMORY.md, gives a hint instead).\n2.6. Prompt Hint — How the Agent Knows What to Read The agent won\u0026rsquo;t read USER.md or MEMORY.md on its own — it needs to be told. The Prompt Hint is an instruction added to the system prompt:\nYou have memory tools. Load USER.md (user profile) and MEMORY.md (long-term memory) when needed.\nThis is analogous to the OpenClaw approach: the agent gets a hint \u0026ldquo;use memory_search\u0026rdquo; and loads MEMORY.md itself. We added \u0026ldquo;when needed\u0026rdquo; — the agent decides whether it needs user context in the current request.\nRisk: the agent may \u0026ldquo;forget\u0026rdquo; to load USER.md → responses without personal context. Mitigation: clear Prompt Hint + examples in system prompt (\u0026ldquo;At the start of a session, load USER.md and MEMORY.md\u0026rdquo;). In practice — USER.md is small (~200-500 tokens), loading takes one react-loop iteration.\n3. TOOLS.md: Two Sources — Two Responsibilities TOOLS.md is the most non-trivial memory file. The system already has a programmatic mechanism for describing tools: ToolSet.RegisterInAgent() generates SystemPromptInjection with tool descriptions for the LLM. Why add another source?\n3.1. The Problem: Mechanics ≠ Practice The system automatically generates a description of each tool: name, parameters, types, brief description. This is enough for the LLM to understand how to call a tool. But the auto-generated description doesn\u0026rsquo;t contain practical experience — and that\u0026rsquo;s what determines when and why the agent will use the tool.\nWhat auto-generated provides (mechanics):\nTool names and parameter names Data types and parameter required/optional Brief description from ToolSettings What auto-generated DOESN\u0026rsquo;T provide (practice):\nWhen to use a specific tool versus another Quirks of working with specific data Typical usage patterns in the project context Limitations not obvious from the description (rate limits, response size) User preferences for working with tools The gap between \u0026ldquo;how to call\u0026rdquo; and \u0026ldquo;when to use\u0026rdquo; is the gap between API documentation and real-world experience with it. TOOLS.md fills this gap.\n3.2. Solution: Auto-generated + User Extensions The System Prompt (tools section) is formed from two sources:\n{auto-generated description from RegisterInAgent()} ← mechanics (in memory) --- ## Tool Notes {contents of TOOLS.md, if exists} ← practice (from storage) Two sources — two responsibilities:\nSource Where stored Who updates What it contains Auto-generated In memory (system prompt) Runtime on every request Mechanics: names, parameters, types, lists TOOLS.md In storage User (UI) / Agent (memory tools) Practice: tips, quirks, examples, warnings Key invariant: the auto-generated part exists only in memory and is never written to storage. TOOLS.md is stored persistently and is never automatically overwritten. The two sources don\u0026rsquo;t conflict.\n3.3. Examples Knowledge DB (RAG) Auto-generated (formed on every request):\nsearch_knowledge_db: Search documents in knowledge databases. Parameters: query (string), db_name (string, optional) Available knowledge databases: - \u0026#34;Product Docs\u0026#34; (product documentation) - \u0026#34;Internal Wiki\u0026#34; (internal processes) TOOLS.md (user notes):\n## Knowledge DB Notes - For Product Docs search, use exact product names, not descriptions - Internal Wiki is updated weekly — if you don\u0026#39;t find current info, ask the user to confirm - Recommended pattern: first search → then read_knowledge_db_document - Search result limit is 5 chunks, if you need more — refine the query 3.4. Why Not \u0026ldquo;Fully Auto\u0026rdquo; and Not \u0026ldquo;Fully Manual\u0026rdquo; Fully auto (an option we considered): only auto-generated description, no TOOLS.md. Pro — always current, no sync issues. Con — the user can\u0026rsquo;t add quirks, tips, usage context. The agent sees \u0026ldquo;search_knowledge_db(query, db_name)\u0026rdquo; but doesn\u0026rsquo;t know that \u0026ldquo;Product Docs needs exact names.\u0026rdquo;\nFully manual (like OpenClaw): user/agent writes TOOLS.md manually, the system generates nothing. Pro — full customization. Con — when a new skill or MCP tool is added, TOOLS.md becomes stale. The user must remember to update. And if they forget — the agent sees descriptions for three tools when five are connected.\nSplitting into auto-generated + user extensions solves both problems: mechanics are always current (generated from ToolSettings), practice is added manually (TOOLS.md). And when MCP arrives — the auto-generated part automatically includes MCP descriptions.\n3.5. Lazy Lifecycle Another nuance: TOOLS.md is not created by default. The file appears in storage only when the user or agent first writes content to it. An \u0026ldquo;empty\u0026rdquo; TOOLS.md is an absent file, not an empty string.\nThis same rule applies to all memory files. Missing file is not an error:\nScenario Behavior TOOLS.md doesn\u0026rsquo;t exist at injection ## Tool Notes section is not added SOUL.md / ABOUT.md don\u0026rsquo;t exist Corresponding sections are omitted Agent requests a non-existent file via memory tools Empty result, not an error Agent writes a new file via memory tools File is created Graceful degradation — a new agent without memory files doesn\u0026rsquo;t crash, it works with minimal context. The prompt hint is always added — even if files don\u0026rsquo;t exist yet, the agent knows the tools are there.\n4. Dreaming: Background Memory Consolidation And now — the most interesting part. Everything I\u0026rsquo;ve described above is an inline mechanism: the agent writes to MEMORY.md during a session, manages size through a \u0026ldquo;compress regularly\u0026rdquo; instruction. There\u0026rsquo;s no consolidation (deduplication, contradiction resolution, stale data removal).\nThe question: should we add a background consolidation process, triggered after sessions end? And if so — how? Anthropic and OpenAI have already answered \u0026ldquo;yes\u0026rdquo; and implemented it in 2026. Let\u0026rsquo;s look at how they do it.\n4.1. Anthropic: Auto Dream (Claude Code) A 4-phase background process, triggered between sessions by a separate sub-agent (solaius, 2026):\nPhase Action Result 1. Orient Reads current state of MEMORY.md and topic files Baseline memory map 2. Gather Signal Searches completed session logs for new facts, drift, contradictions. Grep, not full read (token economy) Signals for update 3. Consolidate Merges duplicates, resolves contradictions, normalizes time (\u0026ldquo;yesterday\u0026rdquo; → \u0026ldquo;2026-03-15\u0026rdquo;), removes stale entries Updated memory files 4. Prune \u0026amp; Index Rebuilds MEMORY.md as an index, updates references, enforces 200-line cap Clean index graph LR LOGS[\"Completed session\nlogs\"] --\u003e|\"Phase 1: Orient\"| MAP[\"Baseline memory\nmap\"] MAP --\u003e|\"Phase 2: Gather Signal\"| SIGNAL[\"Signals:\nnew facts, drift,\ncontradictions\"] SIGNAL --\u003e|\"Phase 3: Consolidate\"| UPDATED[\"Updated\nmemory files\"] UPDATED --\u003e|\"Phase 4: Prune \u0026 Index\"| CLEAN[\"Clean index\nMEMORY.md\"] style LOGS fill:#1565c0,color:#fff style CLEAN fill:#2e7d32,color:#fff Trigger: double gate — 24+ hours since last consolidation AND 5+ new sessions. Not \u0026ldquo;after every conversation\u0026rdquo; but \u0026ldquo;when enough material has accumulated.\u0026rdquo; Smart: frequent short sessions don\u0026rsquo;t trigger dreaming every minute.\nSafety: the sub-agent only writes to memory/, has no git/npm/MCP tools. It can\u0026rsquo;t break code, can\u0026rsquo;t reach the internet — only memory cleanup.\nResults (Harvey, legal AI): 6x increase in task completion rate (solaius, 2026). But Harvey is a legal AI where memory is critical. A realistic estimate for typical scenarios — 1.5–3x.\nCost: standard Claude token rates. 913 sessions consolidated in 8–9 minutes. At \u0026ldquo;once per day with 5+ sessions\u0026rdquo; frequency — pennies compared to main LLM calls.\n4.2. Anthropic: Dreams API (Managed Agents, Enterprise) Enterprise version of Auto Dream with additional capabilities (solaius, 2026):\nProperty Value Scale Up to 100 past sessions per dream Isolation Input store is not modified — a new output store is created Review gate Auto-apply or review before applying Pattern types Recurring mistakes, workflow convergence, shared preferences Versioning Each mutation creates an immutable memver_... with audit trail The review gate is the key difference from basic Auto Dream. Enterprise can\u0026rsquo;t afford \u0026ldquo;LLM silently rewrote memory.\u0026rdquo; Every change goes through review. Versioning is immutable — you can roll back to any previous version. This is critical for regulated workflows (healthcare, legal, finance).\n4.3. KAIROS (unreleased, internal Anthropic) And here\u0026rsquo;s a preview of the architecture for always-on agents (solaius, 2026). Append-only logs + nightly dreaming.\nLogs: logs/YYYY/MM/YYYY-MM-DD.md — append-only, never modified Nightly /dream distills logs into topic files and MEMORY.md Analogy with WAL (Write-Ahead Log) in databases: full audit trail + lean working memory Why does this matter? Because it solves the fundamental problem of current approaches: what if dreaming deletes something important? In KAIROS — it won\u0026rsquo;t, ever. Logs are append-only, they don\u0026rsquo;t change. /dream creates a new representation (MEMORY.md), but the original stays. Like WAL in PostgreSQL: even if the checkpoint crashes — the logs are there, you can recover.\nIt\u0026rsquo;s the same principle as in event sourcing: state is a function of event history. MEMORY.md is a projection, and logs are the event store. A projection can be recreated, an event store cannot.\n4.4. OpenAI: Dreaming V3 (June 2026) A background synthesis process that completely replaced manual \u0026ldquo;saved memories\u0026rdquo; (OpenAI, 2026):\nGeneration Mechanism Factual Recall 2024: Saved Memories Manual list, explicit \u0026ldquo;remember this\u0026rdquo; 41.5% 2025: Dreaming V0 Supplemental synthesis overlay on top of saved memories 67.9% 2026: Dreaming V3 Fully automatic synthesis, replaces saved memories 82.8% The evolution is impressive: from 41.5% to 82.8% in two years (OpenAI, 2026). But even more interesting — how Dreaming V3 works:\nReads the entire dialog history of the user Automatically synthesizes a profile: facts, preferences, instructions, implicit patterns Self-updating: \u0026ldquo;user is traveling to Singapore in July\u0026rdquo; → after the trip, automatically rewritten to \u0026ldquo;user traveled to Singapore in July 2026\u0026rdquo; Runs between conversations, not during Self-updating is the killer feature. In our approach, MEMORY.md contains \u0026ldquo;user is traveling to Singapore in July.\u0026rdquo; After July — that\u0026rsquo;s a stale fact. In Dreaming V3 — the fact automatically updates. No manual management.\nBut there are problems:\nThe platform silently rewrites records — no revision log (OpenAI, 2026; solaius, 2026). You can\u0026rsquo;t find out what was there before the rewrite. Audit trail is opaque — critical for regulated workflows. In healthcare, you\u0026rsquo;re obligated to know why the agent made a decision, and if memory silently changed — you won\u0026rsquo;t know. 5x compute reduction enabled expansion to the Free tier — but at the cost of quality for complex scenarios. 4.5. Comparison of Approaches Aspect Auto Dream (Anthropic) Dreams API (Enterprise) KAIROS (unreleased) Dreaming V3 (OpenAI) Our approach (inline) When capture After session After session Nightly After session During session What it analyzes Full logs Up to 100 sessions All logs Entire history Only current context Consolidation Auto: merge, contradictions, normalization Auto + review gate Nightly distillation Auto: synthesis + self-update Instruction \u0026ldquo;compress MEMORY.md\u0026rdquo; Trigger 24h + 5 sessions On demand Nightly Between conversations No trigger Audit — Immutable versions Append-only logs (WAL) No revisions Last-write-wins Cost Additional LLM call Additional LLM call Additional LLM call nightly Additional compute (5x reduced) No additional cost Update latency 24h+ On demand 1 night Between conversations Immediate 4.6. Why We\u0026rsquo;re NOT Doing Dreaming Yet Four reasons:\nThe basic mechanism is sufficient — inline write via memory tools + \u0026ldquo;compress\u0026rdquo; instruction covers 80% of the value. Memory works, facts are saved, agents recall context. The \u0026ldquo;memory turns into a dumpster\u0026rdquo; problem is theoretical — we haven\u0026rsquo;t seen it in production yet.\nInfrastructure — dreaming requires a worker, task queue (NATS stream or BullMQ), session completion triggers, consolidation prompt. None of this exists, and it\u0026rsquo;s not \u0026ldquo;a couple lines of code.\u0026rdquo;\nNo data — without accumulated experience with MEMORY.md in production, it\u0026rsquo;s unclear how critical the problem is. Maybe the \u0026ldquo;compress\u0026rdquo; instruction is sufficient. Maybe not. We need to look at metrics: MEMORY.md size, staleness frequency, duplicate count.\nCost and risk — the LLM might incorrectly merge different facts, delete important things as \u0026ldquo;stale.\u0026rdquo; At this stage, a review gate is needed (not auto-apply), and that\u0026rsquo;s a different UX.\nRecommended path:\nCurrent epic — implement the inline mechanism Observation — collect metrics on MEMORY.md in production Next epic — if metrics show degradation → implement Option B (batch consolidation) with review gate Future — if audit trail is needed → Option C (append-only + nightly distillation) Approximate cost of batch consolidation: ~5500 tokens per run (3000 for session log + 1000 for MEMORY.md + 500 for prompt + 1000 for output). At \u0026ldquo;once per 24h with 5+ sessions\u0026rdquo; frequency and a cheap model (OSS 20B) — ~$0.01–0.05 per agent/day. That\u0026rsquo;s 1–3% of the cost of serving the agent. Pennies — but only if the basic mechanism already works.\n5. Write-Manage-Read in Practice In the first part I described the write-manage-read loop (Du et al., 2026) as an abstract model. Let\u0026rsquo;s see what it looks like in our implementation.\n5.1. Write: Who, When, Through What File Who writes When Through what SOUL.md User (UI) / Agent At agent creation, on user request UI / memory tools ABOUT.md User (UI) / Agent At creation, on role change UI / memory tools TOOLS.md User (UI) / Agent When practical experience accumulates UI / memory tools USER.md Agent / User At first interaction, on preference update Memory tools / UI MEMORY.md Agent When an important fact is identified Memory tools 5.2. Manage: Compression, Deduplication, Staleness The current mechanism is an instruction to the agent:\nRegularly compress MEMORY.md. Remove outdated facts. Merge duplicates. If MEMORY.md \u0026gt; 5000 characters — compress it.\nThis isn\u0026rsquo;t automatic consolidation, but delegation to the agent. Does it work? In most cases — yes. An LLM is perfectly capable of compressing 5000 characters of facts down to 3000, removing duplicates and stale data. But not always correctly — it might delete something important or merge things that should stay separate. Hence the soft limit, not hard truncation.\nFuture: automatic deduplication via LLM (like Auto Dream Phase 3). But — after accumulating production metrics.\nTemporal validity: every memory file has a last_modified. If a fact hasn\u0026rsquo;t been updated longer than a configurable TTL — it\u0026rsquo;s marked stale. Simpler than MemPalace\u0026rsquo;s Knowledge Graph with valid_from/valid_to, but sufficient for a platform with hundreds of agents. Graph queries are Zep\u0026rsquo;s territory.\n5.3. Read: Injection vs On-Demand Mechanism Files When Tokens Bootstrap injection SOUL.md, ABOUT.md, TOOLS.md Every request ~1-2K Prompt Hint — Every request ~50 On-demand (memory tools) USER.md, MEMORY.md Agent\u0026rsquo;s decision ~0.5-5K Total on boot: ~1-2K tokens. On first request with USER.md + MEMORY.md load: ~2-4K. On each subsequent request: depends on the task — the agent might not access memory at all.\n6. Takeaways 6.1. Memory Files Summary File Scope Loading Char limit Who writes Key purpose SOUL.md agent Injection 2000 UI / agent Identity ABOUT.md agent Injection 2000 UI / agent Profile TOOLS.md agent Injection 2000 UI / agent Tool practice USER.md agent_user On-demand 2000 agent / UI User profile MEMORY.md agent_user On-demand 5000 (soft) agent only Long-term memory 6.2. Context Assembly Strategies Strategy Boot tokens Predictability Flexibility Scalability Full Injection ~5-10K ✅ ❌ ❌ On-Demand 0 ❌ ✅ ✅ Hybrid v1 ~2-3K ✅ ⚠️ ⚠️ Hybrid v2 ~1-2K ⚠️ ✅ ✅ 6.3. Dreaming: Not \u0026ldquo;If\u0026rdquo; but \u0026ldquo;When\u0026rdquo; Anthropic and OpenAI have already implemented background memory consolidation. The results are impressive: 82.8% factual recall at OpenAI, 6x task completion at Harvey (Anthropic). But both approaches have tradeoffs: OpenAI silently rewrites without an audit trail, Anthropic requires a separate sub-agent and double gate.\nOur path — inline mechanism first, then metrics, then dreaming. Not because we don\u0026rsquo;t believe in consolidation, but because we need to understand what exactly to consolidate. Without production data — it\u0026rsquo;s shooting in the dark.\nWhen we do implement it — most likely Option B (batch consolidation with review gate), like Anthropic. And if audit trail is needed — Option C (append-only logs + nightly distillation), like KAIROS. But that\u0026rsquo;s a whole different story.\nPrevious article in the series: Part 5: Agent Memory Management\nReferences OpenAI. \u0026ldquo;Dreaming: Better memory for a more helpful ChatGPT.\u0026rdquo; June 4, 2026. openai.com/index/chatgpt-memory-dreaming solaius (Red Hat Research). \u0026ldquo;Claude Memory \u0026amp; Dreaming Deep Dive.\u0026rdquo; June 2026. github.com/solaius/ai-asset-registry ","permalink":"https://triumphpc.github.io/blog/posts/ai-agent-design-patterns-6-memory-deep-dive/","summary":"Deep dive into agent memory mechanics: five memory files, four context assembly strategies, TOOLS.md as a dual-source responsibility, and Dreaming — background consolidation from Anthropic (Auto Dream, Dreams API, KAIROS) and OpenAI (Dreaming V3, 82.8% factual recall). Engineering decisions behind a production platform.","title":"AI Agent Design Patterns. Part 6: Agent Memory in Practice"},{"content":"1. Introduction In Part 1, I covered ReAct — an agent that thinks and acts. In Part 2 — Plan-and-Execute, where a planner builds an N-step plan. In Part 3 — Reflexion, where an agent learns from its mistakes. Three patterns, three solo agents.\nAnd here we hit a ceiling. One LLM agent can\u0026rsquo;t be an expert at everything. Well, it can — but poorly. One system prompt — one \u0026ldquo;role.\u0026rdquo; Writer ≠ Reviewer ≠ Researcher in the same head. Conflicting interests, hallucinations, loss of focus. And even Reflexion won\u0026rsquo;t help if the task requires fundamentally different expertise — an agent can\u0026rsquo;t \u0026ldquo;reflect\u0026rdquo; a skill it doesn\u0026rsquo;t have into existence.\nSo we need a team. Multiple agents, each with its own role, coordinating to solve a shared task. Sounds simple, but here\u0026rsquo;s where it gets interesting: how exactly do they coordinate? Who makes decisions? How do you pass context? How do you avoid infinite handoff loops?\nI\u0026rsquo;ll break down five multi-agent orchestration architectures — from a centralized supervisor to a parallel debate — with Mermaid diagrams, code in Eino (Go), and honest limitations. All examples use the same scenario: a code review pipeline with Researcher, Coder, and Reviewer — easier to compare that way.\n2. Supervisor Pattern Motivation The most common scenario: you have a task requiring different expertise, but you need someone to decide which agent to delegate what to. Without centralized coordination, agents will talk simultaneously, interrupt each other, or — even worse — pass the task around in circles.\nSupervisor is centralized coordination: one router agent receives the task, decides who to delegate to, collects results, and decides the next step.\nArchitecture graph TB USER[\"👤 User\"] --\u003e|\"task\"| SUP[\"🧑‍💼 Supervisordelegates, aggregates\"] SUP --\u003e|\"delegate: research\"| RES[\"🔍 Researchergathers context\"] SUP --\u003e|\"delegate: code\"| COD[\"💻 Coderwrites code\"] SUP --\u003e|\"delegate: review\"| REV[\"🔎 Reviewerreviews code\"] RES --\u003e|\"result\"| SUP COD --\u003e|\"result\"| SUP REV --\u003e|\"result\"| SUP SUP --\u003e|\"final answer\"| USER style SUP fill:#2e7d32,color:#fff style RES fill:#1565c0,color:#fff style COD fill:#e65100,color:#fff style REV fill:#6a1b9a,color:#fff The Supervisor sees the full picture: each sub-agent returns its result to the supervisor, not to the next agent. This eliminates chaos, but creates a bottleneck.\nImplementation in Eino Eino ADK provides a ready-made supervisor.New:\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;log\u0026#34; \u0026#34;github.com/cloudwego/eino/adk\u0026#34; \u0026#34;github.com/cloudwego/eino/adk/prebuilt/supervisor\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; ) func main() { ctx := context.Background() // Initialize ChatModel for all agents chatModel, err := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ APIKey: os.Getenv(\u0026#34;OPENAI_API_KEY\u0026#34;), Model: os.Getenv(\u0026#34;OPENAI_MODEL\u0026#34;), BaseURL: os.Getenv(\u0026#34;OPENAI_BASE_URL\u0026#34;), }) if err != nil { log.Fatal(err) } // Create sub-agents researcher, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;researcher\u0026#34;, Description: \u0026#34;Researches best practices and gathers context for coding tasks\u0026#34;, Instruction: \u0026#34;You are a research specialist. Find relevant information, API docs, and best practices.\u0026#34;, Model: chatModel, }) coder, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;coder\u0026#34;, Description: \u0026#34;Writes Go code based on research findings\u0026#34;, Instruction: \u0026#34;You are a Go developer. Write clean, idiomatic Go code.\u0026#34;, Model: chatModel, }) reviewer, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;reviewer\u0026#34;, Description: \u0026#34;Reviews code for bugs, style issues, and correctness\u0026#34;, Instruction: \u0026#34;You are a code reviewer. Check for bugs, edge cases, and style issues.\u0026#34;, Model: chatModel, }) // Supervisor agent (coordinator) coordinator, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;coordinator\u0026#34;, Description: \u0026#34;Coordinates research, coding, and review sub-agents\u0026#34;, Instruction: \u0026#34;You are a project coordinator. Delegate tasks to the right specialist.\u0026#34;, Model: chatModel, }) // Assemble the Supervisor pattern supervisorAgent, err := supervisor.New(ctx, \u0026amp;supervisor.Config{ Supervisor: coordinator, SubAgents: []adk.Agent{researcher, coder, reviewer}, }) if err != nil { log.Fatal(err) } // Run runner := adk.NewRunner(ctx, adk.RunnerConfig{ Agent: supervisorAgent, }) result, _ := runner.Query(ctx, \u0026#34;Implement a thread-safe LRU cache in Go\u0026#34;) _ = result } Key point: supervisor.Config takes a Supervisor (coordinator) and SubAgents (array of specialists). The Supervisor itself decides who to call and when.\nLimitations Bottleneck: all results pass through the supervisor → latency grows linearly as agent count increases Context explosion: the supervisor must hold results from all sub-agents in its context → tokens burn fast Single point of failure: if the supervisor delegates incorrectly, the entire process goes off track 3. Swarm / Handoff Pattern Motivation But what if you don\u0026rsquo;t need a supervisor? What if agents already know who to hand off to? This is decentralized coordination: an agent completes its part and transfers (handoff) to the next one.\nSounds tempting — no bottleneck, no single point of failure. But there\u0026rsquo;s a price: who decides when to hand off? The agent must understand when its work is done and who to pass the baton to.\nArchitecture graph LR RES[\"🔍 Researcher\"] --\u003e|\"handoff:context ready\"| COD[\"💻 Coder\"] COD --\u003e|\"handoff:code ready\"| REV[\"🔎 Reviewer\"] REV --\u003e|\"handoff:needs fix\"| COD REV --\u003e|\"done\"| USER[\"👤 User\"] style RES fill:#1565c0,color:#fff style COD fill:#e65100,color:#fff style REV fill:#6a1b9a,color:#fff Notice: the Reviewer can hand the task back to the Coder (found a bug → fix it). This is a cycle — unlike the Assembly Line, where the flow is strictly unidirectional.\nImplementation in Eino Eino provides host.NewMultiAgent — a Host pattern where a host agent routes to specialists:\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;log\u0026#34; \u0026#34;github.com/cloudwego/eino/flow/agent/multiagent/host\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; ) func main() { ctx := context.Background() chatModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ APIKey: os.Getenv(\u0026#34;OPENAI_API_KEY\u0026#34;), Model: os.Getenv(\u0026#34;OPENAI_MODEL\u0026#34;), BaseURL: os.Getenv(\u0026#34;OPENAI_BASE_URL\u0026#34;), }) // Host agent: router hostAgent := \u0026amp;host.Host{ ChatModel: chatModel, SystemPrompt: \u0026#34;You manage a code review pipeline. Route requests to the right specialist.\u0026#34;, } // Specialists (implement the host.Specialist interface) researcher := NewResearchSpecialist(chatModel) coder := NewCodeSpecialist(chatModel) reviewer := NewReviewSpecialist(chatModel) // Assemble multi-agent multiAgent, err := host.NewMultiAgent(ctx, \u0026amp;host.Config{ Host: hostAgent, Specialists: []host.Specialist{researcher, coder, reviewer}, }) if err != nil { log.Fatal(err) } // Run result, _ := multiAgent.Run(ctx, \u0026#34;Implement a thread-safe LRU cache in Go\u0026#34;) _ = result } Eino\u0026rsquo;s Host pattern is a hybrid: the host agent routes tasks, but specialists can be invoked by context, not just by direct order. This is closer to swarm than to a pure supervisor.\nLimitations Deadlock: Agent A → B → A → B → \u0026hellip; infinite handoff loop. No supervisor to break the cycle Routing quality: if an agent incorrectly assesses who to hand off to, the entire chain breaks Context isolation: each agent sees only its own context + handoff message. No global picture 4. Debate Pattern Motivation What if the task has no single correct answer? Or if it\u0026rsquo;s critical to test multiple hypotheses in parallel? Du et al. showed that multi-agent debate improves factual accuracy by +8% on GSM8K through debate between N agents (Du et al., 2023).\nThe idea: several agents independently solve the task, then critique each other\u0026rsquo;s solutions — and arrive at a consensus. Like Minsky\u0026rsquo;s society of minds, but in practice.\nArchitecture graph TB TASK[\"📋 Task\"] --\u003e A1[\"🤖 Agent A\"] TASK --\u003e A2[\"🤖 Agent B\"] TASK --\u003e A3[\"🤖 Agent C\"] A1 \u003c--\u003e|\"critique\"| A2 A2 \u003c--\u003e|\"critique\"| A3 A1 \u003c--\u003e|\"critique\"| A3 A1 --\u003e J[\"⚖️ Judge / Consensus\"] A2 --\u003e J A3 --\u003e J J --\u003e RESULT[\"✅ Final Answer\"] style A1 fill:#2e7d32,color:#fff style A2 fill:#1565c0,color:#fff style A3 fill:#e65100,color:#fff style J fill:#c62828,color:#fff Three agents solve the task in parallel, critique each other, and a Judge (or majority vote) selects the final answer. This is not a pipeline — it\u0026rsquo;s parallel consensus.\nImplementation in Eino There\u0026rsquo;s no ready-made Debate pattern in Eino. We build one using compose.Graph:\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/cloudwego/eino/compose\u0026#34; \u0026#34;github.com/cloudwego/eino/schema\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; ) func main() { ctx := context.Background() chatModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ APIKey: os.Getenv(\u0026#34;OPENAI_API_KEY\u0026#34;), Model: os.Getenv(\u0026#34;OPENAI_MODEL\u0026#34;), BaseURL: os.Getenv(\u0026#34;OPENAI_BASE_URL\u0026#34;), }) g := compose.NewGraph[string, string]() // Three independent agents — generate solutions in parallel g.AddGraphNode(\u0026#34;agent_a\u0026#34;, func(ctx context.Context, task string) (string, error) { msgs := []*schema.Message{ schema.SystemMessage(\u0026#34;You are a senior Go developer. Solve the task.\u0026#34;), schema.UserMessage(task), } resp, _ := chatModel.Generate(ctx, msgs) return resp.Content, nil }) g.AddGraphNode(\u0026#34;agent_b\u0026#34;, func(ctx context.Context, task string) (string, error) { msgs := []*schema.Message{ schema.SystemMessage(\u0026#34;You are a careful code reviewer. Solve the task with emphasis on correctness.\u0026#34;), schema.UserMessage(task), } resp, _ := chatModel.Generate(ctx, msgs) return resp.Content, nil }) g.AddGraphNode(\u0026#34;agent_c\u0026#34;, func(ctx context.Context, task string) (string, error) { msgs := []*schema.Message{ schema.SystemMessage(\u0026#34;You are a performance expert. Solve the task with emphasis on efficiency.\u0026#34;), schema.UserMessage(task), } resp, _ := chatModel.Generate(ctx, msgs) return resp.Content, nil }) // Judge: aggregates and picks the best answer g.AddGraphNode(\u0026#34;judge\u0026#34;, func(ctx context.Context, input string) (string, error) { // input contains results from all three agents msgs := []*schema.Message{ schema.SystemMessage(\u0026#34;You are a judge. Compare three solutions and pick the best one. Explain your choice.\u0026#34;), schema.UserMessage(input), } resp, _ := chatModel.Generate(ctx, msgs) return resp.Content, nil }) // Parallel generation → Judge g.AddEdge(compose.START, \u0026#34;agent_a\u0026#34;) g.AddEdge(compose.START, \u0026#34;agent_b\u0026#34;) g.AddEdge(compose.START, \u0026#34;agent_c\u0026#34;) g.AddEdge(\u0026#34;agent_a\u0026#34;, \u0026#34;judge\u0026#34;) g.AddEdge(\u0026#34;agent_b\u0026#34;, \u0026#34;judge\u0026#34;) g.AddEdge(\u0026#34;agent_c\u0026#34;, \u0026#34;judge\u0026#34;) g.AddEdge(\u0026#34;judge\u0026#34;, compose.END) r, _ := g.Compile(ctx) result, _ := r.Invoke(ctx, \u0026#34;Implement a thread-safe LRU cache in Go\u0026#34;) fmt.Println(result) } Key point: compose.START → three nodes in parallel → Judge aggregates. Graph automatically parallelizes nodes whose inputs are all ready.\nLimitations 3x cost: three LLM calls instead of one (minimum). Judge is the fourth Doesn\u0026rsquo;t work without a Judge: simple majority vote can converge on an error (collective hallucination) Latency: parallel is faster than sequential, but still at least max(agent_a, agent_b, agent_c) + judge 5. Assembly Line Pattern Motivation What if the task is strictly sequential? Research → Code → Review — each stage depends on the previous one. No parallelism, no debate. Just a pipeline.\nThis is exactly what ChatDev (Qian et al., 2023) and MetaGPT (Hong et al., 2023) implemented. MetaGPT added SOP: each agent receives a clear input/output format, which reduces hallucinations by 1.8x compared to ChatDev.\nArchitecture graph LR TASK[\"📋 Task\"] --\u003e RES[\"🔍 Researchergathers context\"] RES --\u003e|\"researchfindings\"| COD[\"💻 Coderwrites code\"] COD --\u003e|\"codedraft\"| REV[\"🔎 Reviewerreviews\"] REV --\u003e|\"approvedcode\"| DONE[\"✅ Result\"] style RES fill:#1565c0,color:#fff style COD fill:#e65100,color:#fff style REV fill:#6a1b9a,color:#fff Strict sequence. No cycles, no parallel branches. Each agent receives the previous one\u0026rsquo;s output — and only that.\nImplementation in Eino Assembly Line is a classic compose.Workflow (or Chain):\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/cloudwego/eino/compose\u0026#34; \u0026#34;github.com/cloudwego/eino/schema\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; ) func main() { ctx := context.Background() chatModel, _ := openai.NewChatModel(ctx, \u0026amp;openpi.ChatModelConfig{ APIKey: os.Getenv(\u0026#34;OPENAI_API_KEY\u0026#34;), Model: os.Getenv(\u0026#34;OPENAI_MODEL\u0026#34;), BaseURL: os.Getenv(\u0026#34;OPENAI_BASE_URL\u0026#34;), }) // Assembly Line via Chain chain := compose.NewChain[string, string]() // Stage 1: Research chain.AppendChatTemplate( prompt.FromMessages(schema.FString, schema.SystemMessage(\u0026#34;You are a research specialist. Gather context and best practices for the task. Output: structured research findings.\u0026#34;), schema.UserMessage(\u0026#34;{input}\u0026#34;), ), ).AppendChatModel(chatModel) // Stage 2: Code chain.AppendChatTemplate( prompt.FromMessages(schema.FString, schema.SystemMessage(\u0026#34;You are a Go developer. Write code based on the research findings. Output: complete Go source code.\u0026#34;), schema.UserMessage(\u0026#34;{input}\u0026#34;), ), ).AppendChatModel(chatModel) // Stage 3: Review chain.AppendChatTemplate( prompt.FromMessages(schema.FString, schema.SystemMessage(\u0026#34;You are a code reviewer. Review the code for bugs, edge cases, and style issues. Output: approved code or list of fixes.\u0026#34;), schema.UserMessage(\u0026#34;{input}\u0026#34;), ), ).AppendChatModel(chatModel) r, _ := chain.Compile(ctx) result, _ := r.Invoke(ctx, \u0026#34;Implement a thread-safe LRU cache in Go\u0026#34;) fmt.Println(result) } compose.Chain is a Workflow under the hood. Each ChatTemplate + ChatModel pair is one pipeline stage. The previous stage\u0026rsquo;s output is automatically fed as input to the next.\nLimitations Sequential = slow: no parallelism, each stage waits for the previous one Cascading errors: if Researcher provides bad context, Coder writes bad code, and Reviewer might not catch the root problem No cycles: if Reviewer finds a bug — the entire pipeline needs to restart (unlike Swarm, where handoff back is possible) 6. Hierarchical Pattern Motivation What if the task is so complex that one level of delegation isn\u0026rsquo;t enough? A project with 10 subtasks, each breaking down into 3-4 micro-tasks. A single supervisor would drown in context.\nHierarchical is multi-level decomposition. The CEO delegates to Team Leads, who delegate to specialists. Each level manages only its own scope.\nArchitecture graph TB USER[\"👤 User\"] --\u003e CEO[\"🎯 CEO Agentstrategy\"] CEO --\u003e|\"designresearch\"| TL1[\"📋 Team Lead: Research\"] CEO --\u003e|\"implement\"| TL2[\"📋 Team Lead: Dev\"] TL1 --\u003e R1[\"🔍 Researcher 1\"] TL1 --\u003e R2[\"🔍 Researcher 2\"] TL2 --\u003e D1[\"💻 Coder 1\"] TL2 --\u003e D2[\"💻 Coder 2\"] TL1 --\u003e CEO TL2 --\u003e CEO CEO --\u003e USER style CEO fill:#c62828,color:#fff style TL1 fill:#2e7d32,color:#fff style TL2 fill:#2e7d32,color:#fff style R1 fill:#1565c0,color:#fff style R2 fill:#1565c0,color:#fff style D1 fill:#e65100,color:#fff style D2 fill:#e65100,color:#fff The key difference from Supervisor: here we have two levels of delegation. The CEO doesn\u0026rsquo;t know about Researcher 1 and Coder 2 — it only works with Team Leads. This reduces context load at each level.\nImplementation in Eino Eino ADK provides deep.New — a DeepAgent with built-in task management:\npackage main import ( \u0026#34;context\u0026#34; \u0026#34;log\u0026#34; \u0026#34;github.com/cloudwego/eino/adk\u0026#34; \u0026#34;github.com/cloudwego/eino/adk/prebuilt/deep\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; ) func main() { ctx := context.Background() chatModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ APIKey: os.Getenv(\u0026#34;OPENAI_API_KEY\u0026#34;), Model: os.Getenv(\u0026#34;OPENAI_MODEL\u0026#34;), BaseURL: os.Getenv(\u0026#34;OPENAI_BASE_URL\u0026#34;), }) // Create sub-agents researcher, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;researcher\u0026#34;, Description: \u0026#34;Researches best practices and gathers context\u0026#34;, Instruction: \u0026#34;You are a research specialist. Find relevant information and API docs.\u0026#34;, Model: chatModel, }) coder, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;coder\u0026#34;, Description: \u0026#34;Writes Go code based on requirements\u0026#34;, Instruction: \u0026#34;You are a Go developer. Write clean, idiomatic Go code.\u0026#34;, Model: chatModel, }) reviewer, _ := adk.NewChatModelAgent(ctx, \u0026amp;adk.ChatModelAgentConfig{ Name: \u0026#34;reviewer\u0026#34;, Description: \u0026#34;Reviews code for correctness and quality\u0026#34;, Instruction: \u0026#34;You are a code reviewer. Check for bugs and style issues.\u0026#34;, Model: chatModel, }) // DeepAgent: automatically breaks down tasks into subtasks // and delegates them to sub-agents deepAgent, err := deep.New(ctx, \u0026amp;deep.Config{ Name: \u0026#34;project_manager\u0026#34;, Description: \u0026#34;Manages complex coding projects by breaking them into subtasks\u0026#34;, ChatModel: chatModel, Instruction: \u0026#34;You are a project manager. Break down tasks and delegate to specialists.\u0026#34;, SubAgents: []adk.Agent{researcher, coder, reviewer}, }) if err != nil { log.Fatal(err) } runner := adk.NewRunner(ctx, adk.RunnerConfig{ Agent: deepAgent, }) result, _ := runner.Query(ctx, \u0026#34;Implement a thread-safe LRU cache in Go with tests\u0026#34;) _ = result } DeepAgent uses the built-in write_todos tool for planning and the task tool for delegating to sub-agents. Context is isolated between the main agent and sub-agents — this prevents context pollution.\nLimitations Overhead: two levels of coordination = two levels of LLM calls. CEO + Team Lead before the task reaches the executor Debugging complexity: an error at a lower level can silently bubble up Over-engineering for simple tasks: if a task fits in a single supervisor — hierarchy is overkill 7. Pattern Comparison Pattern Centralization Parallelism Coordination complexity Token overhead When to use Supervisor High Low Low Medium Clear roles, need control over execution order Swarm / Handoff Low Medium High Medium Agents know their boundaries; flexible routing Debate Low (or Judge) High Medium High Ambiguous tasks, need hypothesis testing Assembly Line Low None Low Low Strict sequential stages, SOP Hierarchical High (multi-level) Medium High High Complex projects with subtask decomposition How to read the table: Centralization — how many decision points exist. Parallelism — can agents work simultaneously. Coordination complexity — how much effort goes into routing. Token overhead — how many extra tokens are spent on coordination.\n8. Honest Limitations Communication overhead Every exchange between agents = full context in the prompt. With 3 agents and 2 debate rounds — 3×2×context_length tokens. With long context, this can cost $0.50+ per request. Mitigation: summarize intermediate results before passing them on.\nCoordination errors The Supervisor can delegate incorrectly. An agent can hand off to the wrong specialist. The Judge can pick the worst answer. Li et al. showed that \u0026ldquo;more agents\u0026rdquo; improves results through sampling-and-voting, but this only works when the base model is already good enough (Li et al., 2024). If the model gives \u0026lt;50% accuracy, more agents won\u0026rsquo;t help.\nDiminishing returns More agents ≠ better results. Li et al. showed that sampling-and-voting scales with the number of agents, but with diminishing returns: each additional agent contributes less than the previous one. In practice, 3-5 agents is the optimum. Beyond 5-7, costs grow while gains are minimal.\nDeadlock / Infinite loop In Swarm/Handoff, Agent A passes to B, B to A, ad infinitum. In Debate, agents can\u0026rsquo;t agree. In Hierarchical, Team Leads keep bouncing the task to each other. Mitigation: WithMaxRunSteps in Eino Graph (as in Reflexion), timeout on handoff count, or circuit breaker when re-transfer to the same agent occurs.\nWhen one agent is better If the task:\nDoesn\u0026rsquo;t require different expertise Has a clear success criterion Fits in a single system prompt \u0026hellip;then multi-agent is unnecessary complexity. ReAct with the right prompt often works better than three agents coordinating through a supervisor. Don\u0026rsquo;t add agents for the sake of adding agents.\n9. Summary Five multi-agent orchestration patterns — not competing, but complementary:\nSupervisor — when you need control and predictability Swarm — when agents know their boundaries and can self-organize Debate — when hypothesis testing and factual accuracy matter Assembly Line — when stages are strictly sequential and SOP matters more than flexibility Hierarchical — when task complexity requires multi-level decomposition In Eino: three patterns out of the box (supervisor.New, host.NewMultiAgent, deep.New), two via compose.Graph. Choosing a pattern is choosing a trade-off between control and flexibility, parallelism and cost, simplicity and scalability.\nAnd the main point: adding agents doesn\u0026rsquo;t fix a bad prompt. Multi-agent is a tool for coordinating expertise, not a magic pill for hallucinations.\nNext in the series: Part 5: Agent Memory Management\n","permalink":"https://triumphpc.github.io/blog/posts/ai-agent-design-patterns-4-multi-agent/","summary":"Five multi-agent orchestration architectures — Supervisor, Swarm, Debate, Assembly Line, Hierarchical — with Mermaid diagrams, code examples in Eino (Go), and an honest look at limitations. When one agent isn\u0026rsquo;t enough, and when you\u0026rsquo;re better off staying solo.","title":"AI Agent Design Patterns. Part 4: Multi-Agent Patterns"},{"content":"1. The Problem: Agent Amnesia In Part 1, I covered the ReAct agent. In Part 3 — Reflexion, where the agent learns from mistakes through episodic memory. Four patterns, four sessions — and none solves a fundamental problem: the context window is not memory.\nPicture this: your debug assistant on Friday found that a middleware bug was a race condition on a shared cache. On Monday, you ask the same agent about a similar bug — and it starts from scratch. It doesn\u0026rsquo;t remember the conclusion, the context, or even that the cache exists. Every Monday is Groundhog Day.\nAnd this isn\u0026rsquo;t exotic. It\u0026rsquo;s the norm. LLMs are stateless by nature: process tokens, return result, forget. The context window — 128K, 200K, even 1M tokens — is always finite. Session ends, window clears, agent knows nothing about your project.\nBut humans don\u0026rsquo;t need to remember everything either — they store what matters and forget the rest. An agent doesn\u0026rsquo;t need infinite memory, it needs the right memory. What to save? How to compress? When to surface into context? These are the questions I\u0026rsquo;ll tackle.\n2. A Taxonomy of Agent Memory The systematic survey by Du et al. (Memory for Autonomous LLM Agents, 2026) formalizes agent memory as a write-manage-read loop:\ngraph LR W[\"✏️ WRITE\nWhat \u0026 when to save\"] --\u003e M[\"🔧 MANAGE\nCompression, deduplication,\nconflict resolution\"] M --\u003e R[\"📖 READ\nWhat \u0026 when to\nsurface into context\"] R --\u003e|\"new experience\"| W Three phases, three fundamentally different engineering decisions:\nWrite: not everything is worth saving. \u0026ldquo;User asked about weather\u0026rdquo; — noise. \u0026ldquo;User prefers YAML over JSON\u0026rdquo; — fact. Filtering is essential. Manage: facts go stale, duplicate, contradict each other. \u0026ldquo;Bob leads the ML team\u0026rdquo; and \u0026ldquo;Alice manages the ML team\u0026rdquo; — you need a resolution mechanism. Read: surface the right thing at the right time. Not the entire archive, just what\u0026rsquo;s relevant. And fit it within the token budget. Du identifies three dimensions for classification: temporal scope (short-term / long-term), representation (text / vectors / graphs), and control policy (fixed rules / learnable). Combinations yield five mechanism families — from context-resident compression to policy-learned management. But in practice, engineers choose from ready-made solutions. And there are surprisingly many.\n3. Approach 1: File-First Brain (OpenClaw) OpenClaw — a project from the team behind mem0 (58K ⭐ on GitHub, YC S24), a popular managed memory layer for AI agents. If mem0 externalizes memory into a service, OpenClaw goes to the opposite extreme — a radical idea: the filesystem = the agent\u0026rsquo;s brain.\nThe agent stores everything in markdown files:\nFile Purpose Size SOUL.md Identity: who I am, my values ~2K tokens AGENTS.md Procedures: how I work ~3K tokens MEMORY.md Curated long-term memory ~5K tokens memory/YYYY-MM-DD.md Raw daily logs No limit Every session starts with a boot sequence — the agent reads SOUL.md, AGENTS.md, MEMORY.md, and recent daily logs. This is its \u0026ldquo;morning coffee\u0026rdquo;: 4K–10K tokens just to start, before it says a single word.\ngraph TB START[\"🚀 New Session\"] --\u003e SOUL[\"📖 SOUL.md\nidentity\"] SOUL --\u003e AGENTS[\"📖 AGENTS.md\nprocedures\"] AGENTS --\u003e MEMORY[\"📖 MEMORY.md\ncurated facts\"] MEMORY --\u003e LOGS[\"📖 daily logs\nrecent entries\"] LOGS --\u003e READY[\"✅ Agent Ready\"] style START fill:#1565c0,color:#fff style READY fill:#2e7d32,color:#fff Why files, not a vector database? OpenClaw\u0026rsquo;s philosophical argument: RAG is for looking up information; an agent needs a brain. A vector DB is fragmented — semantic search returns chunks without context. Files are whole, natively readable by the agent, human-editable.\nBut there\u0026rsquo;s a cost. 10K tokens per boot is 10K tokens not used for the actual task. At 5M tokens/day (typical OpenClaw burn), this is a serious budget. And the bigger MEMORY.md grows, the more expensive each session becomes.\n4. Approach 2: Structured Compression (MemPalace) MemPalace solves OpenClaw\u0026rsquo;s main problem — token crushing. The idea: don\u0026rsquo;t load everything upfront; surface on demand.\nArchitecture is a Memory Palace hierarchy: Wing (project/person) → Room (sub-topic) → Hall (memory type: facts, events, discoveries, preferences, advice). At each level — a compressed description that LLM reads natively. Compression isn\u0026rsquo;t magic AAAK — it\u0026rsquo;s plain truncation: snippets are cut to 200–300 characters and grouped by room.\nFour memory layers (layers.py):\nLayer What it stores How it works Tokens L0 — Identity Who I am, my principles, key people Reads ~/.mempalace/identity.txt (user-written) ~100 L1 — Essential Story Top-15 important moments from the entire palace Scans up to 2000 drawers, ranks by importance/emotional_weight/weight, groups by room, truncates to 3200 chars ~500–800 L2 — On-Demand Filtered retrieval by wing/room Metadata filter in ChromaDB (not semantic!), up to N drawers ~200–500 per call L3 — Deep Search Full semantic search col.query(query_texts=...) across the entire palace with similarity ranking Unlimited Startup — only L0 + L1, ~600–900 tokens. L2 and L3 surface via MCP tools when the agent encounters a task requiring context.\ngraph TB BOOT[\"🚀 wake-up\n~600-900 tokens\"] --\u003e L0[\"L0: Identity\n~100 tokens\nidentity.txt file\"] BOOT --\u003e L1[\"L1: Essential Story\n~500-800 tokens\ntop-15 drawers by weight\"] L0 --\u003e L2[\"L2: On-Demand\n~200-500 tokens\nfilter by wing/room\"] L1 --\u003e L2 L2 --\u003e L3[\"L3: Deep Search\nunlimited\nsemantic search\"] L3 --\u003e WING[\"🏰 Wing\nproject / person\"] WING --\u003e ROOM[\"🏠 Room\nsub-topic\"] ROOM --\u003e HALL[\"🚪 Hall\nmemory type\"] style BOOT fill:#2e7d32,color:#fff style L2 fill:#e65100,color:#fff style L3 fill:#c62828,color:#fff Bonus: Knowledge Graph with temporal validity — facts have expiration dates. \u0026ldquo;Bob leads the ML team\u0026rdquo; is valid until a certain date. When Alice becomes the lead — the fact automatically expires. This solves the conflict resolution problem from the write-manage-read loop.\nResult: 96.6% R@5 on LongMemEval (long-term memory benchmark) at ~600–900 tokens per wake-up (L0 + L1). Versus 10K for OpenClaw. A 10–15x difference — MemPalace recalls better while spending an order of magnitude less context on boot. And this is without an LLM at search time: pure ChromaDB, pure semantic search, zero API calls.\n5. Approach 3: Managed Memory Layer The third approach — externalize memory into a separate service. The agent stores nothing itself; it requests context from an external layer.\nmem0 (GitHub, 58K ⭐) — the \u0026ldquo;universal memory layer.\u0026rdquo; A managed service (YC S24) that automatically extracts facts from conversations, deduplicates, and returns relevant context on request. New algorithm (April 2026): 94.8% on LongMemEval at 6.8K tokens. Pros: no need to think about write/manage — the service does it all. Cons: vendor lock-in, dependency on an external API.\nNow Zep is a different beast. It\u0026rsquo;s an open-source memory server, and worth a closer look because it solves a problem mem0 doesn\u0026rsquo;t: the structure of relationships between facts.\nPicture this: the agent knows \u0026ldquo;Bob works on the ML team\u0026rdquo; and \u0026ldquo;The ML team uses PyTorch.\u0026rdquo; A vector DB (like mem0\u0026rsquo;s) will find each fact separately — but won\u0026rsquo;t infer that Bob likely works with PyTorch. Zep adds a Knowledge Graph on top of vectors — powered by its Graphiti engine. Entities and relationships are extracted automatically from conversations, and namespace-based isolation separates different users\u0026rsquo; contexts.\nWhy are we considering it? Because in real projects, facts don\u0026rsquo;t live in a vacuum — they\u0026rsquo;re connected. \u0026ldquo;Client X switched to plan Y\u0026rdquo; and \u0026ldquo;Plan Y doesn\u0026rsquo;t support feature Z\u0026rdquo; — the agent should draw a conclusion, not just return both facts. A graph makes that possible.\nBut there\u0026rsquo;s a cost. Zep requires infrastructure: vector DB + graph + embedding service. Not one service, but a stack. If you don\u0026rsquo;t have DevOps capacity — deployment will hurt.\nLangMem (docs, by LangChain) — SDK with three memory types: semantic (facts), episodic (past experiences), procedural (evolving behavior). Procedural memory is unique: the agent updates its own prompt based on feedback, \u0026ldquo;learning\u0026rdquo; to behave better. Pros: native LangGraph integration, namespace isolation for privacy. Cons: tied to the LangChain ecosystem.\nWhat\u0026rsquo;s common? All three are middleware: they sit between the agent and the LLM, intercept context, and enrich it with relevant facts. Differences lie in storage (vectors vs. graph vs. prompt), management (auto vs. curated), and deployment (SaaS vs. self-hosted vs. embedded).\n6. How We Solve It Our team is tackling this exact problem — designing memory for AI agents on our platform. And the architecture we\u0026rsquo;ve arrived at looks suspiciously like a hybrid of OpenClaw and MemPalace.\nFour memory layers:\ngraph TB SESSION[\"🔄 Session Layer\ncurrent dialogue,\nauto-cleanup\"] AGENT[\"🤖 Agent Layer\nSOUL.md, ABOUT.md,\ncharacter \u0026 profile\"] USER[\"👤 User Layer\npreferences,\nuser context\"] PROJECT[\"📁 Project Layer\nAGENTS.md, TOOLS.md,\nworkspace + Virtual FS\"] SESSION --\u003e AGENT --\u003e USER --\u003e PROJECT style SESSION fill:#1565c0,color:#fff style AGENT fill:#6a1b9a,color:#fff style USER fill:#e65100,color:#fff style PROJECT fill:#2e7d32,color:#fff Runtime context assembly: on every request, the system builds context layer by layer — from session (cheapest, always in window) to project (most expensive, loaded on demand). Like MemPalace: cheap startup, detail on demand.\nThe agent\u0026rsquo;s file structure is directly inspired by OpenClaw:\nFile OpenClaw Analog Purpose SOUL.md SOUL.md Agent\u0026rsquo;s character ABOUT.md — Profile/description AGENTS.md AGENTS.md Agent creation instructions TOOLS.md — Auto-assembly from skills + MCP From MemPalace, we adopted layered loading: don\u0026rsquo;t load everything into context at once, surface layers as needed. And temporal validity — facts have expiration dates; stale ones don\u0026rsquo;t make it into context. But what does this mean in practice?\nLayered loading: how it works for us MemPalace has four layers (layers.py), each solving a specific problem:\nL0 — Identity (~100 tokens). Plain-text file ~/.mempalace/identity.txt, written by the user. \u0026ldquo;I am Atlas, a personal AI assistant for Alice. Traits: warm, direct. People: Alice (creator), Bob (Alice\u0026rsquo;s partner). Project: A journaling app.\u0026rdquo; This is a constant — doesn\u0026rsquo;t change between sessions, not computed, just read from disk.\nL1 — Essential Story (~500–800 tokens). Auto-generated from the palace: scans up to 2000 drawers (memory chunks), ranks by weight (importance, emotional_weight, weight), takes top-15, groups by room for readability, truncates to 3200 characters. This is \u0026ldquo;the most important things that happened\u0026rdquo; — not the full archive, but a digest. The algorithm isn\u0026rsquo;t perfect: e.g., a drawer with high emotional_weight might crowd out a more useful but less \u0026ldquo;emotional\u0026rdquo; fact. But for the boot task — giving the agent minimal context — it works.\nL2 — On-Demand (~200–500 tokens per call). Filtered retrieval: \u0026ldquo;give me everything from wing=driftwood, room=auth-migration.\u0026rdquo; This is a metadata filter in ChromaDB, not semantic search — simply WHERE wing = ? AND room = ?. Surfaces when a specific topic comes up in conversation.\nL3 — Deep Search (unlimited). Full semantic search: col.query(query_texts=[\u0026quot;why did we switch to GraphQL\u0026quot;]) across the entire palace. Returns results with similarity ranking. This is the heavy artillery — when L1 and L2 didn\u0026rsquo;t provide enough context.\nStartup — wake_up() → L0 + L1, ~600–900 tokens. L2 and L3 surface via MCP tools when the agent encounters a task requiring context.\nWe adapted this model, but with a key difference: MemPalace is an external service, while our layers are assembled inside the agent\u0026rsquo;s runtime. No need to call MCP for memory — context is already assembled by the time the LLM starts generating.\nWhat the assembly looks like on each request:\nLayer What\u0026rsquo;s loaded When Tokens Session Current dialogue Always 0 (already in window) Agent SOUL.md + ABOUT.md Always ~1K User Preferences, context Always (for current user) ~500 Project AGENTS.md + TOOLS.md + workspace On demand (first request to project) ~2–5K Total at startup: ~1.5K tokens. More than MemPalace (~600–900), but an order of magnitude less than OpenClaw (~10K). A compromise.\nWhy load Project on demand? Imagine: a user enters a chat, says hello. The agent doesn\u0026rsquo;t know which project the work will involve. Why spend 5K tokens loading AGENTS.md and TOOLS.md for a project that might never be needed? So: wait until the task requires project context — and only then surface the layer.\nAnd how does our L1 analog work — ranking \u0026ldquo;the most important\u0026rdquo;? We don\u0026rsquo;t have one. SOUL.md and ABOUT.md are themselves L0/L1, curated manually. No automatic ranking from 2000 drawers, but no risk of an algorithm with high emotional_weight crowding out a useful but \u0026ldquo;boring\u0026rdquo; fact either. For a platform with hundreds of agents, this is a deliberate choice: predictability over automation.\nTemporal validity: facts with expiration dates This is the second idea from MemPalace we ported. In MemPalace, the Knowledge Graph is built on SQLite: each fact is a triple (subject, predicate, object) with valid_from and valid_to fields (knowledge_graph.py). When a fact becomes stale, it\u0026rsquo;s not deleted — instead, valid_to is set. This enables queries like \u0026ldquo;what was true on date X?\u0026rdquo; via filtering: valid_from \u0026lt;= X AND (valid_to \u0026gt;= X OR valid_to IS NULL).\n# MemPalace: adding a temporal fact kg.add_triple(\u0026#34;Kai\u0026#34;, \u0026#34;works_on\u0026#34;, \u0026#34;Orion\u0026#34;, valid_from=\u0026#34;2025-06-01\u0026#34;) # Kai leaves Orion kg.invalidate(\u0026#34;Kai\u0026#34;, \u0026#34;works_on\u0026#34;, \u0026#34;Orion\u0026#34;, ended=\u0026#34;2026-03-01\u0026#34;) # Query: what\u0026#39;s true now? kg.query_entity(\u0026#34;Kai\u0026#34;) # → [Kai → works_on → Orion (ended), Kai → recommended → Clerk] # Query: what was true on Jan 20, 2026? kg.query_entity(\u0026#34;Kai\u0026#34;, as_of=\u0026#34;2026-01-20\u0026#34;) # → [Kai → works_on → Orion (active)] Why is this needed? The most common memory problem isn\u0026rsquo;t missing facts — it\u0026rsquo;s stale facts. The agent remembers \u0026ldquo;the project uses REST API,\u0026rdquo; but the team migrated to gRPC three months ago. Instead of the right answer, the agent drags a false fact into context. Temporal validity solves this: every fact has an expiration date, expired ones are automatically excluded from context.\nOur implementation is simpler than MemPalace\u0026rsquo;s: instead of a separate SQLite graph, we use metadata in the Virtual FS. Each agent memory file (MEMORY.md, ABOUT.md) has last_modified — and if a fact hasn\u0026rsquo;t been updated longer than TTL (configurable), it\u0026rsquo;s marked stale and excluded from context. No full graph query with as_of, but for our use case (a platform with hundreds of agents, not a single coding assistant) this is sufficient. Graph queries are Zep\u0026rsquo;s territory, and if we need logical inference over a fact graph, we know where to look.\nRuntime architecture: from request to context Above I described abstract memory layers. Here\u0026rsquo;s what it looks like at runtime — at the code level.\nThe main orchestrator is Chat UseCase. Every user request goes through an 8-stage pipeline:\ngraph TB REQ[\"📥 User Request\"] --\u003e PREP[\"1️⃣ prepareSession\nLoad Session + AgentVersion\"] PREP --\u003e REG[\"2️⃣ RegisterInAgent\nBind tools to agent\"] REG --\u003e PRE[\"3️⃣ ExecutePreAgent\nRAG pre-search\"] PRE --\u003e ASM[\"4️⃣ Context Assembly\nSession[] + SystemPrompt + Tool Injections\"] ASM --\u003e LLM[\"5️⃣ LLM react-loop\neino/adk Runner\"] LLM --\u003e|\"needs context\"| TOOLS[\"6️⃣ Tool Execution\nWorkspace files / RAG / Skills\"] TOOLS --\u003e|\"loaded into context\"| LLM LLM --\u003e|\"done\"| POST[\"7️⃣ PostAgent + Hooks\ncleanup + side effects\"] POST --\u003e SAVE[\"8️⃣ Update Session\nContext[]\"] style REQ fill:#1565c0,color:#fff style TOOLS fill:#e65100,color:#fff style SAVE fill:#2e7d32,color:#fff Key insight: context is assembled before the LLM starts generating. By the time runner.Run() is called, everything is already assembled — session, system prompt, tool injections. The LLM receives a ready-made context and works with it.\nBut this doesn\u0026rsquo;t mean all memory is loaded at once. SOUL.md and ABOUT.md are already in the system prompt — this is \u0026ldquo;hot\u0026rdquo; memory. Files from Workspace the agent requests itself during the react-loop (step 5→6 on the diagram), when it realizes it lacks context. This is on-demand loading from MemPalace — except instead of MCP tools we use eino tools.\nFive Workspace scopes and their S3 mapping:\nScope Virtual Path S3 Prefix Used by session /storage/session/... projects/{pid}/sessions/{sid}/workspace/ Current dialogue artifacts user /storage/user/... projects/{pid}/users/{uid}/workspace/ User files and preferences agent_user /storage/agent_user/... projects/{pid}/agents/{aid}/users/{uid}/workspace/ Agent-user pair memory skills /storage/skills/... projects/{pid}/skills/ SKILL.md, project documents hooks /storage/hooks/... projects/{pid}/hooks/ Hook scripts (post-agent) And unlike both OpenClaw and MemPalace, we don\u0026rsquo;t need an external memory service — everything runs inside the platform via Workspace (Virtual FS → S3) and Session Context (PostgreSQL).\nDifference from OpenClaw: we don\u0026rsquo;t store raw daily logs. Instead — curated memory with automatic compression. Difference from mem0: no external service dependency — everything runs inside the platform via Virtual FS and S3 mount.\n7. Takeaways Five approaches to agent memory — from file-based brain to managed service:\nApproach Project Boot cost Storage Management Best for File-First OpenClaw ~10K tokens Markdown files Manually curated Coding agents, full control Structured Compression MemPalace ~600–900 tokens Hierarchy + ChromaDB Auto (ranking + temporal) Multi-project agents, strict budget Managed Layer mem0 ~6.8K tokens Vector DB Auto (SaaS) Quick start, no infra overhead Vector + Graph Zep Depends on query Vectors + Knowledge Graph Auto (self-hosted) Structured relationships, privacy Embedded SDK LangMem Depends on type LangGraph store Auto + procedural memory LangChain ecosystem No silver bullet. OpenClaw gives maximum control but burns tokens. MemPalace is elegant but requires structural discipline. mem0 is easy to integrate but vendor-locked. Zep is powerful for graph relationships but heavy to deploy. LangMem is perfect for LangChain but useless outside it.\nWe chose a hybrid: memory layers like MemPalace, file structure like OpenClaw, runtime assembly instead of static boot. Because our task isn\u0026rsquo;t a coding agent or a chatbot — it\u0026rsquo;s a platform where agents of different types live and work together. And memory must serve that diversity.\nPrevious article in the series: Part 4: Multi-Agent Patterns\n","permalink":"https://triumphpc.github.io/blog/posts/ai-agent-design-patterns-5-memory/","summary":"Context window ≠ memory. I break down agent amnesia, the academic taxonomy (Du et al., 2026), five approaches to memory — from OpenClaw\u0026rsquo;s file-based brain to mem0\u0026rsquo;s managed layer — and how our team solves this in practice.","title":"AI Agent Design Patterns. Part 5: Agent Memory Management"},{"content":"1. Introduction In Part 1 I covered ReAct — a pattern where the agent interleaves reasoning and action. The problem: ReAct plans one step ahead. For tasks that need a global plan — incident analysis, multi-step debugging, service orchestration — the myopic approach breaks. Plan-and-Execute solves this by separating planning from execution.\n2. What is Plan-and-Execute History In May 2023, Wang et al. published Plan-and-Solve Prompting — a zero-shot strategy that first devises a plan to divide a task into subtasks, then carries out the subtasks. This was a direct response to three pitfalls of Zero-shot-CoT: missing-step errors, calculation errors, and semantic misunderstanding errors.\nIn March 2023, Shen et al. introduced HuggingGPT — an LLM-powered controller that plans a task, selects AI models from Hugging Face based on their descriptions, executes each subtask with the chosen model, and summarizes the result. This was the first production-scale implementation of the pattern.\nSo. The pattern crystallized: a Planner (strategist), an Executor (tactician), and an optional Reviser (reviewer). LangChain — a Python framework for building LLM applications, including agent primitives, RAG pipelines, and orchestration tools — popularized this as the Plan-and-Execute Agent in 2024, but the idea originated in 2023.\nTeam analogy Imagine: a tech lead (Planner) decomposes a task into tickets, a developer (Executor) picks up a ticket and implements it, a QA engineer (Reviser) verifies — and if they find a problem, the ticket goes back for rework. ReAct is a solo developer who sets their own one-step task, does it, checks it, sets the next one. For simple tasks — fine. For complex ones — you need a manager.\nFormal definition Plan-and-Execute operates in three phases:\nPlan: The Planner generates a list of steps [S₁, S₂, ..., Sₙ] to solve the task. Execute: The Executor executes each step Sᵢ, calling tools and receiving observations. Revise: The Reviser evaluates progress and modifies the remaining plan if necessary. The key difference from ReAct: planning is global, not step-by-step. The Planner sees the entire task and builds an N-step plan. The Executor receives one step at a time — not the full plan, not the user input. This creates a natural security boundary.\n3. What problem it solves ReAct\u0026rsquo;s problem: myopic planning ReAct, as I showed in Part 1, plans one step ahead. Each Thought is a reaction to the previous Observation, not to an overall strategy. For tasks with known structure (log analysis → filtering → correlation → diagnosis), this is inefficient: the agent \u0026ldquo;wanders\u0026rdquo; instead of following a plan.\nReAct\u0026rsquo;s problem: cost Each ReAct step is a full LLM call with the entire context (all previous Thoughts + Actions + Observations). With 10 steps — 10 calls to an expensive model. ReWOO (Xu et al., 2023) showed this is unnecessary: up to 5x reduction in token consumption on HotpotQA with a 4.4% accuracy improvement (Xu et al., 2023).\nReAct\u0026rsquo;s problem: no explicit re-planning ReAct doesn\u0026rsquo;t revise a plan — there simply isn\u0026rsquo;t one. If a step leads to a dead end, the model \u0026ldquo;implicitly\u0026rdquo; corrects through the next Thought. But that\u0026rsquo;s not re-planning — it\u0026rsquo;s reactive wandering. Plan-and-Execute makes re-planning explicit through the Reviser.\nWhat Plan-and-Execute provides Problem Plan-and-Execute solution Myopic planning Planner sees the whole task, builds an N-step plan High cost Executor can use a cheap model — it only needs one step No re-planning Reviser explicitly evaluates progress and modifies the plan Prompt injection Executor doesn\u0026rsquo;t see user input or the full plan Opacity The plan is a readable, auditable artifact 4. Architecture Plan → Execute → Revise sequenceDiagram participant U as User participant P as Planner participant E as Executor participant T as Tools participant R as Reviser U-\u003e\u003eP: Task P-\u003e\u003eP: Generate plan [S₁, S₂, ..., Sₙ] loop For each step P-\u003e\u003eE: Step Sᵢ E-\u003e\u003eT: Call tool T--\u003e\u003eE: Observation E--\u003e\u003eR: Step result R-\u003e\u003eR: Evaluate progress alt Plan needs revision R-\u003e\u003eP: Revised plan else Step OK R-\u003e\u003eE: Next step Sᵢ₊₁ end end R--\u003e\u003eU: Final answer The Planner receives the task and generates a plan. The Executor executes step by step, calling tools. The Reviser checks each step\u0026rsquo;s result and decides: continue the plan, revise it, or finish.\nEino State Graph In Eino, the pattern is implemented via compose.Graph — a directed graph with three agent nodes:\ngraph TB START((START)) --\u003e PL[PlannerBaseChatModel] PL --\u003e EX[ExecutorToolCallingChatModel+ ToolsNode] EX --\u003e RV{ReviserBaseChatModel} RV --\u003e|needs revision| PL RV --\u003e|step OK, more steps| EX RV --\u003e|all done| END((END)) The state struct stores: messages (history), current plan (steps), step number. The maxStep parameter limits iterations — protection against infinite loops.\nPattern evolution graph LR C[\"CoTWei 2022\"] --\u003e R[\"ReActYao 2022\"] R --\u003e PS[\"Plan-and-SolveWang 2023\"] PS --\u003e PA[\"Plan-and-ExecuteLangChain 2024\"] PA --\u003e RW[\"ReWOOXu 2023\"] PA --\u003e LC[\"LLMCompilerKim 2023\"] CoT (Chain-of-Thought, Wei et al., 2022) gave step-by-step reasoning without actions. ReAct (Yao et al., 2022) added actions. Plan-and-Solve (Wang et al., 2023) introduced explicit planning. Plan-and-Execute formalized this as an architectural pattern with separate agents. ReWOO and LLMCompiler are optimizations of the base pattern.\n5. Variants ReWOO: Reasoning Without Observation ReWOO (Xu et al., 2023) eliminates interleaved LLM calls. The Planner generates a plan with variables once, and the Executor substitutes results — without calling the LLM at each step. Result: 5x token reduction and +4.4% accuracy on HotpotQA. An additional bonus: you can distill the Planner from 175B (GPT-3.5) to 7B (LLaMA) — and it works.\nHere\u0026rsquo;s how it looks on our log analysis scenario:\ngraph LR subgraph \"1. Planner (LLM, 1 call)\" P[\"Plan:#E1 = query_logs('error spike')#E2 = filter_by_time(#E1, 'last hour')#E3 = correlate_deployments(#E1)#E4 = suggest_fix(#E2, #E3)\"] end subgraph \"2. Executor (no LLM)\" E1[\"#E1 → query_logs→ result into #E1\"] E2[\"#E2 → filter_by_time(#E1)→ result into #E2\"] E3[\"#E3 → correlate(#E1)→ result into #E3\"] E4[\"#E4 → suggest_fix(#E2,#E3)→ final answer\"] end P --\u003e E1 E1 --\u003e E2 E1 --\u003e E3 E2 --\u003e E4 E3 --\u003e E4 style P fill:#4a9eff,color:#fff style E1 fill:#2d8659,color:#fff style E2 fill:#2d8659,color:#fff style E3 fill:#2d8659,color:#fff style E4 fill:#2d8659,color:#fff The key insight: the Planner uses the LLM once to generate the plan. Then the Executor simply calls tools and substitutes results into variables — like a template engine. The LLM is only needed for the next Planner call if the Reviser decides to re-plan.\nLLMCompiler: parallel execution LLMCompiler (Kim et al., 2023) borrows from classical compilers: the Planner generates a DAG (Directed Acyclic Graph) of step dependencies, the Task Fetching Unit dispatches independent steps in parallel, and the Executor runs them concurrently. Result: 3.7x latency speedup, 6.7x cost savings, ~9% accuracy improvement over ReAct.\nOn our scenario it looks like this:\ngraph LR subgraph \"1. Planner → DAG\" P[\"Plan with dependencies:#E1 = query_logs('error')#E2 = filter_by_time(#E1)#E3 = correlate_deployments(#E1)#E4 = query_metrics('cpu')#E5 = suggest_fix(#E2, #E3, #E4)\"] end subgraph \"2. Parallel execution\" E1[\"#E1 query_logs⏱ 200ms\"] E2[\"#E2 filter_by_time⏱ 100ms\"] E3[\"#E3 correlate⏱ 150ms\"] E4[\"#E4 query_metrics⏱ 120ms\"] E5[\"#E5 suggest_fix⏱ 300ms\"] end P --\u003e E1 E1 --\u003e E2 E1 --\u003e E3 E1 --\u003e E4 E2 --\u003e E5 E3 --\u003e E5 E4 --\u003e E5 style P fill:#4a9eff,color:#fff style E1 fill:#2d8659,color:#fff style E2 fill:#e6a817,color:#333 style E3 fill:#e6a817,color:#333 style E4 fill:#e6a817,color:#333 style E5 fill:#2d8659,color:#fff Yellow highlights the steps that run in parallel: after #E1 (query_logs) completes, steps #E2, #E3, and #E4 launch simultaneously — each depends only on #E1, not on each other. Total time: 200ms + max(100, 150, 120) + 300ms = 650ms instead of sequential 870ms. With dozens of steps in real tasks, the gain grows up to 3.7x.\nHuggingGPT: LLM as controller HuggingGPT (Shen et al., 2023) is a specific implementation: ChatGPT plans the task, selects AI models from Hugging Face based on their descriptions, executes each subtask with the chosen model, and summarizes the result. Not a separate variant, but an illustration of the pattern\u0026rsquo;s applicability to real systems.\ngraph LR U[\"User:Generate an imageand describe it with voice\"] P[\"ChatGPT(Planner)\"] M1[\"Stable Diffusion(text-to-image)\"] M2[\"Whisper(speech-to-text)\"] M3[\"Bark(text-to-speech)\"] S[\"ChatGPT(Summarizer)\"] U --\u003e P P --\u003e|\"1. image generation\"| M1 P --\u003e|\"2. describe image\"| M2 P --\u003e|\"3. voice output\"| M3 M1 --\u003e S M2 --\u003e S M3 --\u003e S S --\u003e U style P fill:#4a9eff,color:#fff style M1 fill:#9b59b6,color:#fff style M2 fill:#9b59b6,color:#fff style M3 fill:#9b59b6,color:#fff style S fill:#4a9eff,color:#fff style U fill:#555,color:#fff Purple — specialized AI models from Hugging Face. ChatGPT acts as the Planner: it parses the request, selects models based on their capability descriptions, passes results between them, and forms the final answer. Unlike ReWOO and LLMCompiler, the Executor here isn\u0026rsquo;t an LLM — it\u0026rsquo;s external models. But the pattern is the same: planning → execution → summarization.\nFull deep-dive on ReWOO and LLMCompiler — in Part 4 of this series.\n6. Security: Control-Flow Integrity And here\u0026rsquo;s where it gets interesting. Separating the Planner and Executor isn\u0026rsquo;t just an architectural flourish. It\u0026rsquo;s a defense against prompt injection (an attack where a malicious instruction is embedded in data that an LLM processes — for example, in tool output or user-supplied content).\nDel Rosario et al. (2025) in \u0026ldquo;Architecting Resilient LLM Agents\u0026rdquo; formulate the principle of control-flow integrity: if the Executor only sees one step of the plan (not the full plan, not the user input), then an attacker controlling tool output cannot redirect the entire workflow. The blast radius is limited to one step.\nContrast with ReAct: in a ReAct agent, the full context (including user input) is available at every step. If a tool returns malicious content, the model may interpret it as a new instruction — and change behavior. In Plan-and-Execute, the Executor receives an isolated task: \u0026ldquo;call tool X with parameters Y.\u0026rdquo;\nAdditional defenses from Del Rosario 2025:\nPrinciple of Least Privilege: each step gets the minimum set of tools Task-scoped tool access: the Executor for \u0026ldquo;filter logs\u0026rdquo; shouldn\u0026rsquo;t have access to \u0026ldquo;delete records\u0026rdquo; Sandboxed execution: code generated by the Executor runs in a sandbox HITL (Human-in-the-Loop): critical steps require human confirmation These aren\u0026rsquo;t theoretical recommendations. Del Rosario et al. provide implementation blueprints for LangGraph, CrewAI, and AutoGen — with working code.\n7. Practice: Plan-and-Execute with Eino, Go Scenario: production incident analysis An on-call engineer gets an alert: latency spike on the orders service. Need to: query logs → filter by time → correlate with deployments → query metrics → suggest fix. Five steps, known structure — a perfect Plan-and-Execute case.\nInstallation go get github.com/cloudwego/eino@latest go get github.com/cloudwego/eino-ext/components/model/openai@latest Minimal Plan-and-Execute agent package main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; \u0026#34;github.com/cloudwego/eino/components/tool\u0026#34; \u0026#34;github.com/cloudwego/eino/compose\u0026#34; \u0026#34;github.com/cloudwego/eino/flow/agent/multiagent/planexecute\u0026#34; \u0026#34;github.com/cloudwego/eino/schema\u0026#34; ) func main() { ctx := context.Background() plannerModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ Model: \u0026#34;gpt-4o\u0026#34;, }) executorModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ Model: \u0026#34;gpt-4o-mini\u0026#34;, }) reviserModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ Model: \u0026#34;gpt-4o\u0026#34;, }) agent, _ := planexecute.NewAgent(ctx, \u0026amp;planexecute.AgentConfig{ Planner: plannerModel, Reviser: reviserModel, Executor: executorModel, ToolsConfig: compose.ToolsNodeConfig{ InvokableTools: []tool.InvokableTool{ queryLogsTool(), filterByTimeTool(), correlateDeploymentsTool(), queryMetricsTool(), suggestFixTool(), }, }, MaxStep: 10, }) msg, _ := agent.Generate(ctx, []*schema.Message{ schema.UserMessage( \u0026#34;Latency spike on the orders service. \u0026#34; + \u0026#34;Find the root cause and suggest a fix.\u0026#34;, ), }) fmt.Println(msg.Content) } Note: Planner and Reviser use gpt-4o (strategic decisions), while Executor uses gpt-4o-mini (calling a single tool). This saves cost: a cheap model for tactical work, an expensive one for strategy. In ReAct, this optimization isn\u0026rsquo;t available — every step requires a full-capability model.\nTools are defined via utils.NewTool — the same way as in Part 1 for ReAct. Full example with tools and prompts: eino-examples/flow/agent/multiagent/plan_execute.\nExecution trace Expand the blocks below to see intermediate agent outputs:\nStep 1: Planner — plan generation Task: Latency spike on the orders service. Find the root cause and suggest a fix. Plan: 1. query_logs — query orders service logs for the last hour 2. filter_by_time — filter logs to the spike window (14:00-14:30) 3. correlate_deployments — check deployments in the same window 4. query_metrics — request CPU/memory metrics for the orders service 5. suggest_fix — based on the data, suggest a fix Step 2: Executor — query_logs Call: query_logs(service=\u0026#34;orders\u0026#34;, level=\u0026#34;ERROR\u0026#34;, limit=100) Result: 47 ERROR records in the last hour. Main errors: - \u0026#34;timeout connecting to payments service\u0026#34; (23 occurrences) - \u0026#34;connection pool exhausted\u0026#34; (18 occurrences) - \u0026#34;slow query: SELECT * FROM orders WHERE...\u0026#34; (6 occurrences) Step 3: Executor — filter_by_time + correlate_deployments Call: filter_by_time(logs=\u0026lt;step2_result\u0026gt;, start=\u0026#34;14:00\u0026#34;, end=\u0026#34;14:30\u0026#34;) Result: Spike started at 14:12. Before 14:12 — normal error levels. Call: correlate_deployments(service=\u0026#34;orders\u0026#34;, start=\u0026#34;14:00\u0026#34;, end=\u0026#34;14:30\u0026#34;) Result: Deploy of payments-service v2.3.1 at 14:10 (2 minutes before spike). Step 4: Executor — query_metrics Call: query_metrics(service=\u0026#34;orders\u0026#34;, metric=\u0026#34;cpu,memory\u0026#34;, start=\u0026#34;14:00\u0026#34;, end=\u0026#34;14:30\u0026#34;) Result: orders CPU — normal (45%). Memory — normal (62%). But: payments-service latency jumped from 50ms to 2000ms starting at 14:10. Step 5: Reviser — validation + suggest_fix Reviser: Root cause found — deploy of payments-service v2.3.1 at 14:10 caused latency increase in payments, which led to timeouts in orders. Plan complete, no re-planning needed. Call: suggest_fix(root_cause=\u0026#34;payments-service v2.3.1 latency regression\u0026#34;, affected_service=\u0026#34;orders\u0026#34;) Result: 1. Immediate: rollback payments-service to v2.3.0 2. Short-term: increase timeout and connection pool for orders→payments 3. Long-term: add circuit breaker between orders and payments Key parameters Planner — model for plan generation (BaseChatModel interface). Reviser — model for progress evaluation and re-planning (BaseChatModel). Executor — model for step execution (ToolCallingChatModel + ToolsNode). MaxStep — graph iteration limit (default 12). Protection against infinite loops. ToolsConfig — tools available to the Executor. Links for deeper study Eino Plan-and-Execute Agent Manual — complete parameter reference eino-examples/flow/agent/multiagent/plan_execute — full working example GitHub: cloudwego/eino — source code 8. Comparative tables Architecture: ReAct vs Plan-and-Execute Dimension ReAct Plan-and-Execute Planning horizon Myopic: 1 step Global: N steps LLM calls per step Full model every step Cheap Executor per step, expensive Planner once Context management Scratchpad: all Thoughts/Actions/Observations accumulate State: plan + step results Re-planning mechanism Implicit: through next Thought Explicit: Reviser modifies the plan Error recovery Retry in-place: model corrects on next step Re-plan from checkpoint: Reviser rebuilds remaining plan Architectural complexity Single loop (ChatModel → ToolsNode) Three agents (Planner → Executor → Reviser) Prompt injection resilience Low: full context available at every step Higher: Executor sees only one step (Del Rosario et al., 2025) Performance and cost Metric ReAct Plan-and-Execute (naive) ReWOO LLMCompiler Token consumption (vs ReAct) 1x ~1.2x (Planner/Reviser overhead) 0.2x (Xu 2023) ~0.5x (parallel execution) LLM calls N steps × full model 1 Planner + N × Executor + k × Reviser 1 Planner + N × tool calls (no LLM) 1 Planner + parallel Executors Latency Sequential: N × T_step Sequential: T_plan + N × T_exec Sequential but no LLM per step Parallel: ~T_plan + T_slowest_step (Kim 2023) Accuracy (HotpotQA) Baseline Comparable +4.4% (Xu 2023) +~9% vs ReAct (Kim 2023) Latency speedup 1x ~1x ~1x 3.7x (Kim 2023) Cost savings 1x ~0.8x (cheap Executor) ~5x (Xu 2023) 6.7x (Kim 2023) Plan-and-Execute variants Variant Tokens vs ReAct Speedup Accuracy (delta) Complexity Key innovation Naive (LangChain-style) ~1.2x ~1x ~0% Low Planning + sequential execution ReWOO (Xu 2023) 0.2x (-5x) ~1x +4.4% Medium Variable substitution, LLM only for Planner LLMCompiler (Kim 2023) ~0.5x 3.7x +~9% High DAG dependencies, parallel execution Pattern selection guide Task characteristic Recommended pattern Example Simple, single API call with parsing ReAct \u0026ldquo;What\u0026rsquo;s the weather in Paris?\u0026rdquo; → one weather API call Complex, multi-step, known structure Plan-and-Execute Production incident analysis: logs → filter → correlate → diagnose Parallelizable subtasks LLMCompiler \u0026ldquo;Compare iPhone prices across 5 stores\u0026rdquo; → 5 parallel calls Cost-sensitive with fixed tool set ReWOO Regular monitoring: same step sequence, same tools Requires learning from past errors Reflexion Code review: agent checks, finds bug, re-checks the fix 9. When NOT to use The Microsoft Azure Architecture Center in their AI agent orchestration guide formulates the principle: use the minimum level of complexity that solves the task.\nIf your task\u0026hellip; \u0026hellip;consider this alternative Is solved by a direct model call Direct model call — classification, summarization Needs one or two tools, steps unknown in advance ReAct — as covered in Part 1 Has a fixed workflow without re-planning ReWOO — cheaper, no Reviser overhead Requires trees of hypotheses Tree of Thoughts (Yao et al., 2023) Spans multiple domains, different security boundaries Multi-agent — Part 5 of this series Plan-and-Execute is a middle complexity tier between ReAct and multi-agent. Don\u0026rsquo;t jump to it if ReAct solves the task. Don\u0026rsquo;t stay on it if you need multi-agent coordination.\n10. What\u0026rsquo;s next Part 3: Reflexion Pattern — ReAct + self-evaluation: the agent learns from its own mistakes through verbal reinforcement. Part 4: ReWOO and LLMCompiler — deep dive into two Plan-and-Execute optimizations: variable substitution and parallel DAG execution. Part 5: Pattern Comparison + Multi-Agent Orchestration — when to choose which pattern, and how multiple agents coordinate to solve complex tasks. Discussion welcome — comments on the site or GitHub Issues.\n11. References Research papers Wang et al., 2023 — Plan-and-Solve Prompting: Zero-shot task decomposition into subtasks. Addresses missing-step errors of Zero-shot-CoT. arXiv:2305.04091 Shen et al., 2023 — HuggingGPT: LLM as a controller orchestrating AI models from Hugging Face. First production-scale implementation of the pattern. arXiv:2303.17580 Xu et al., 2023 — ReWOO: Decoupling reasoning from observations. Variable substitution, 5x token efficiency, +4.4% accuracy on HotpotQA. 175B→7B distillation. arXiv:2305.18323 Kim et al., 2023 — LLMCompiler: Parallel function calling via DAG. 3.7x latency speedup, 6.7x cost savings, +~9% accuracy. arXiv:2312.04511 Del Rosario et al., 2025 — Plan-then-Execute Security: Control-flow integrity as prompt injection defense. Least privilege, task-scoped tool access, sandboxed execution. arXiv:2509.08646 Yao et al., 2022 — ReAct: The base Reasoning + Acting pattern from which Plan-and-Execute evolves. arXiv:2210.03629 Wei et al., 2022 — Chain-of-Thought: Step-by-step reasoning — the predecessor of all agent patterns. arXiv:2201.11903 Documentation and examples Eino Plan-and-Execute Agent Manual — complete parameter reference Eino Open Source Announcement — framework overview GitHub: cloudwego/eino — source code GitHub: eino-examples/flow/agent/multiagent/plan_execute — full working example Architecture guides AI agent design patterns — Microsoft Azure Architecture Center — complexity hierarchy, orchestration patterns, production recommendations Next article: AI Agent Design Patterns. Part 3: Reflexion Pattern — ReAct + self-evaluation: the agent learns from its own mistakes through verbal reinforcement and episodic memory.\n","permalink":"https://triumphpc.github.io/blog/posts/ai-agent-design-patterns-2-plan-execute/","summary":"Plan-and-Execute separates strategic planning from tactical execution. I break down the Planner+Executor+Reviser architecture, ReWOO and LLMCompiler variants with numbers from original papers, prompt injection defense via control-flow integrity, and a minimal working Go example with Eino.","title":"AI Agent Design Patterns. Part 2: Plan-and-Execute Pattern"},{"content":"1. Introduction In Part 1, I covered ReAct — a pattern where an agent alternates between reasoning and action. In Part 2, I covered Plan-and-Execute, where a separate Planner builds an N-step plan. Both patterns share one flaw: the agent doesn\u0026rsquo;t learn from its mistakes. Every run starts from scratch. The Reflexion pattern fixes this: the agent makes an attempt, evaluates the result, reflects — and uses the accumulated experience on the next try.\n2. What is Reflexion History In December 2022, Anthropic published Constitutional AI — an approach where a language model critiques its own responses and rewrites them to reduce harmfulness. This was the first large-scale example of the self-critique pattern: generate → self-critique → revise. But Constitutional AI used critique for training (finetuning via RLAIF), not for inference.\nIn March 2023, Madaan et al. published Self-Refine — iterative improvement through self-feedback. The same LLM plays three roles: generator, critic, and refiner. Result: ~20% average improvement across 7 tasks (Madaan et al., 2023). But there\u0026rsquo;s a catch — on reasoning tasks (Math Reasoning), the improvement is 0%: the model cannot reliably determine whether its reasoning is correct or not.\nAnd here\u0026rsquo;s where it gets interesting. In the same month of March 2023, Shinn et al. published Reflexion — solving the core problem of Self-Refine by adding episodic memory and external evaluation. Instead of a single critique-refine cycle — a multi-trial process where the agent accumulates verbal \u0026ldquo;lessons\u0026rdquo; and uses them on subsequent attempts. Result: 91% pass@1 on HumanEval (vs. 80% for GPT-4) and +22% absolute on AlfWorld (Shinn et al., 2023).\nAnalogy with a developer Imagine: a junior writes code, runs tests — they fail. What do they do? They don\u0026rsquo;t rewrite from scratch — they read the error, understand the cause, and note \u0026ldquo;next time I\u0026rsquo;ll check the nil edge case.\u0026rdquo; That\u0026rsquo;s reflection: not just fixing, but extracting a lesson for the future. ReAct is a junior without memory: making the same mistakes every time. Plan-and-Execute is a junior with a plan but without retrospection. Reflexion is a junior who keeps an error diary.\nFormal definition Reflexion operates in three phases, repeated cyclically:\nAct: The Actor (LLM) generates actions and receives observations from the environment. Evaluate: The Evaluator assesses the result — with a scalar score or free-form text. Reflect: The Self-Reflection model generates verbal feedback — what went wrong and how to fix it. The reflection is stored in episodic memory. On the next attempt, the Actor receives the contents of episodic memory in context — and can avoid repeating past mistakes.\n3. What problem it solves ReAct\u0026rsquo;s problem: no learning from mistakes ReAct, as I showed in Part 1, alternates Thought-Action-Observation in a single loop. If the task isn\u0026rsquo;t solved — the agent simply starts over. Past experience? Lost. Every run is the first and last.\nPlan-and-Execute\u0026rsquo;s problem: no retrospection Plan-and-Execute builds a plan and executes it. The Reviser adjusts the plan when needed — but within a single run. Between runs — clean slate. No learning from past mistakes.\nSelf-Refine\u0026rsquo;s problem: no memory between attempts Self-Refine does critique-refine in a single LLM call. There\u0026rsquo;s improvement on generation tasks (style, format) — but on reasoning tasks 0%, because the model cannot reliably assess the correctness of its own reasoning without an external arbiter (Madaan et al., 2023, Table 1: Math Reasoning).\nWhat Reflexion provides Problem Reflexion\u0026rsquo;s solution No learning from mistakes Episodic memory stores verbal lessons between attempts Unreliable self-assessment Evaluator — external arbiter (tests, compiler, environment) No retrospection Each attempt is enriched with reflections from past ones Myopic fixes Reflection focuses on the cause of the error, not the symptom 4. Architecture Actor → Evaluator → Reflector → Memory → retry sequenceDiagram participant U as User participant A as Actor participant E as Evaluator participant R as Reflector participant M as Episodic Memory U-\u003e\u003eA: Task + reflections from memory A-\u003e\u003eA: Generate actions (ReAct loop) A-\u003e\u003eE: Trajectory (actions + observations) alt Result is correct E--\u003e\u003eU: ✅ Success else Result has errors E-\u003e\u003eR: Trajectory + evaluation R-\u003e\u003eR: Generate verbal reflection\"I failed because...\" R-\u003e\u003eM: Store reflection M--\u003e\u003eA: Enriched context for next attempt Note over A,M: Retry with accumulated experience end Components Actor — an LLM generating actions. Can be a ReAct agent or a simple ChatModel. The key difference from regular ReAct: the Actor receives episodic memory contents in the system prompt, allowing it to account for past mistakes.\nEvaluator — assesses the Actor\u0026rsquo;s result. Can be:\nDeterministic: unit tests, compiler, game environment — gives an objective score LLM-based: another language model assesses quality — less reliable but applicable for open-ended tasks Why does this matter? It\u0026rsquo;s the external verifier that solves Self-Refine\u0026rsquo;s problem, where the model cannot assess its own correctness.\nSelf-Reflection model — an LLM generating verbal reflection. Receives: the Actor\u0026rsquo;s trajectory (actions + observations), the Evaluator\u0026rsquo;s assessment, past reflections from memory. Generates text like: \u0026ldquo;I made a mistake handling the empty list edge case. Next time I need to check len() \u0026gt; 0 before accessing an element.\u0026rdquo;\nEpisodic Memory — a store of reflections. Simple structure: a list of text strings injected into the Actor\u0026rsquo;s context on the next attempt. The more attempts — the richer the memory.\n5. Evolution of self-critique Three self-critique patterns emerged within 3 months — each solving the previous one\u0026rsquo;s problem:\nflowchart TD CAI[\"🛡️ Constitutional AIDec 2022 • Bai/AnthropicSelf-critique → revisefor training (finetuning)Goal: harmlessness\"] SRF[\"🔄 Self-RefineMar 2023 • Madaan/CMUSame LLM: generate → critique → refinefor inference (1 call)~20% improvement on generation\"] REF[\"🧠 ReflexionMar 2023 • Shinn/Princeton+NEUActor → Evaluator → Reflectorfor inference (N attempts)+ Episodic Memory\"] HUANG[\"⚠️ Huang et al.Oct 2023 • ICLR 2024LLMs Cannot Self-CorrectReasoning YetWithout external feedback — doesn't work\"] RRR[\"🚀 Reflect, Retry, RewardMay 2025 • Bensal/WriterRL-trained reflections1.5B-7B beats 10x models\"] CAI --\u003e|\"addedinference-timecritique\"| SRF SRF --\u003e|\"addedepisodic memory+ external eval\"| REF REF --\u003e|\"showedlimitationswithout verifier\"| HUANG HUANG --\u003e|\"RL-trainingbetter reflections\"| RRR style CAI fill:#2e7d32,color:#fff style SRF fill:#1565c0,color:#fff style REF fill:#e65100,color:#fff style HUANG fill:#c62828,color:#fff style RRR fill:#6a1b9a,color:#fff Comparison of three patterns Aspect Constitutional AI Self-Refine Reflexion Date Dec 2022 Mar 2023 Mar 2023 Goal Harmlessness (safety) Output quality Output quality + learning Critique Self-critique Self-feedback External evaluator + self-reflection Memory No No Episodic memory Attempts 1 1 (iterations inside) N (multi-trial) Application Finetuning (offline) Inference (online) Inference (online) Result Improved harmlessness +20% on generation, 0% on reasoning +11% HumanEval (91% vs 80% GPT-4), +22% AlfWorld Evaluation type RLAIF (RL from AI judge) Self-judge External verifier The pattern is clear: each subsequent pattern adds what the previous one lacked. Constitutional AI had no memory and no multi-trial capability. Self-Refine added inference-time critique, but without memory and without an external verifier. Reflexion closed the loop: the external verifier solves the unreliable self-assessment problem, and episodic memory enables learning between attempts.\n6. When it works / when it doesn\u0026rsquo;t Why a dedicated section on limitations? Because in October 2023, Huang et al. published \u0026ldquo;Large Language Models Cannot Self-Correct Reasoning Yet\u0026rdquo; — and proved that self-correction without external feedback degrades results.\nflowchart TD START[Agent task] --\u003e Q{Externalverifier available?} Q --\u003e|Yes| Q2{Low initialaccuracy?} Q --\u003e|No| FAIL[\"❌ Reflexion won't helpSelf-correction degrades results(Huang et al., 2023)\"] Q2 --\u003e|Yes| WORKS[\"✅ Reflexion works+11-22% improvement(HumanEval, AlfWorld)\"] Q2 --\u003e|No| WARN[\"⚠️ May not be worth itRisk of degradation on simple tasksDiminishing returns\"] WORKS --\u003e REC1[\"Recommendation:max 3-5 attempts,evaluate cost/benefit\"] WARN --\u003e REC2[\"Recommendation:no more than 2 attempts,monitor quality\"] style FAIL fill:#c62828,color:#fff style WORKS fill:#2e7d32,color:#fff style WARN fill:#e65100,color:#fff Numbers from Huang et al. Without an external verifier (intrinsic self-correction), quality drops across all models:\nModel GSM8K (before → after) CommonSenseQA (before → after) GPT-3.5 75.9 → 74.7 75.8 → 41.8 GPT-4 95.5 → 89.0 82.0 → 80.0 Llama-2-70b 62.0 → 36.5 64.0 → 36.5 The reason: LLMs are more likely to change a correct answer to an incorrect one than vice versa. The fundamental problem is that the model cannot reliably assess the correctness of its own reasoning (Huang et al., 2023, Figure 1).\nWhen Reflexion works Condition Why External verifier (tests, compiler, env) Objective assessment → accurate reflection Low initial accuracy Room for improvement Multi-trial scenario Memory accumulates lessons Tasks with objective success criterion Clear error signal When Reflexion does NOT work Condition Why No external verifier Model can\u0026rsquo;t assess its own correctness High initial accuracy Risk of degradation (correct → incorrect) Open-ended tasks without criteria Nothing to evaluate → inaccurate reflection Simple tasks Cost of reflection isn\u0026rsquo;t justified 7. Implementation variants in Eino Eino doesn\u0026rsquo;t provide a ready-made reflection.NewAgent() — unlike react.NewAgent() from Part 1 or planexecute.NewAgent() from Part 2. But that\u0026rsquo;s actually a good thing: Reflexion is not a separate agent type, but a composition pattern that can be implemented in several ways.\nVariant A: compose.Graph with a cycle Eino Graph supports cycles via AddEdge from a node to itself + WithMaxRunSteps to limit iterations. This is the most natural implementation of Reflexion:\nflowchart LR START((\"START\")) --\u003e Actor[\"🎬 Actor(react.Agent)\"] Actor --\u003e Evaluator[\"🔍 Evaluator(Lambda: run tests)\"] Evaluator --\u003e Branch{\"Pass?\"} Branch --\u003e|Yes| END((\"END\")) Branch --\u003e|No| Reflector[\"🪞 Reflector(ChatModel)\"] Reflector --\u003e Memory[\"💾 Memory(State: []string)\"] Memory --\u003e Actor style START fill:#2e7d32,color:#fff style END fill:#2e7d32,color:#fff style Branch fill:#e65100,color:#fff style Actor fill:#1565c0,color:#fff style Evaluator fill:#1565c0,color:#fff style Reflector fill:#6a1b9a,color:#fff style Memory fill:#ad1457,color:#fff Pros: explicit retry cycle, branch support, checkpoint via WithCheckPointStore, can embed a ReAct agent via ExportGraph().\nCons: Graph API uses implicit data passing (entire output → entire input), need to manage state carefully.\nVariant B: compose.Workflow (linear, no cycle) Workflow is a declarative graph with explicit field mapping. Problem: Workflow does not support cycles — always AllPredecessor. For Reflexion, this is fatal: no retry-loop.\nBut if the cycle is implemented externally (in Go code), and Workflow is used for a single Actor → Evaluator → Reflector iteration — it works:\n// External retry-loop for attempt := 0; attempt \u0026lt; maxAttempts; attempt++ { result, err := workflow.Invoke(ctx, input) if result.Passed { break } memory = append(memory, result.Reflection) // Inject memory into the next call input.Reflections = memory } Pros: explicit field mapping, type safety, easier to test a single iteration.\nCons: no built-in cycle — have to implement manually, no checkpoint between iterations.\nVariant C: deer-go pattern (State Graph) deer-go is a Go implementation of ByteDance\u0026rsquo;s DEER-flow on Eino Graph. It uses Goto-based routing: each node writes the next target node to state.Goto, and an agentHandOff function directs execution.\nFor Reflexion, you could add a Critic node and a Critic → Actor edge (retry). This extends the existing architecture, but deer-go doesn\u0026rsquo;t contain ready-made reflection patterns — only a re-planning loop.\nPros: ready state graph architecture, checkpoint, human-in-the-loop via InterruptAndRerun.\nCons: more complex, requires HTTP server and MCP tools, overkill for simple scenarios.\nVariant comparison Aspect Graph + cycle Workflow + external loop deer-go Retry cycle Built-in External (Go code) Via Goto Checkpoint ✅ WithCheckPointStore ❌ Manual ✅ Built-in Complexity Medium Low High Field mapping Implicit Explicit Via State Production readiness High Medium High My choice for the example: compose.Graph — native cycle support, checkpoint, and direct embedding of react.Agent via ExportGraph().\n8. Code example Implementing Reflexion on compose.Graph: Actor (ReAct agent) generates Go code, Evaluator runs tests, Reflector analyzes errors, Memory accumulates reflections.\nflowchart TD START((\"START\")) --\u003e Actor[\"🎬 Actorreact.Agent+ write_code tool\"] Actor --\u003e Evaluator[\"🔍 EvaluatorLambda: go test\"] Evaluator --\u003e Branch{\"Tests pass?\"} Branch --\u003e|Yes| END((\"END ✅\")) Branch --\u003e|No| Reflector[\"🪞 ReflectorChatModel: analyzetest failures\"] Reflector --\u003e Memory[\"💾 Append reflectionto episodic memory\"] Memory --\u003e Actor style START fill:#2e7d32,color:#fff style END fill:#2e7d32,color:#fff style Branch fill:#e65100,color:#fff package main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/cloudwego/eino/compose\u0026#34; \u0026#34;github.com/cloudwego/eino/components/tool\u0026#34; \u0026#34;github.com/cloudwego/eino/components/tool/utils\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; \u0026#34;github.com/cloudwego/eino/flow/agent/react\u0026#34; \u0026#34;github.com/cloudwego/eino/schema\u0026#34; ) // reflexionState — shared state for a single Reflexion execution. // Created fresh with each Invoke. type reflexionState struct { // Task — the original task (e.g., \u0026#34;write a function that sorts a list\u0026#34;) Task string // Reflections — accumulated verbal reflections from past attempts Reflections []string // Attempt — current attempt number (1-based) Attempt int // MaxAttempts — maximum number of attempts MaxAttempts int // Code — generated code (Actor output) Code string // TestResult — test run result (Evaluator output) TestResult string // Passed — flag: did tests pass? Passed bool } // writeCodeTool — Actor\u0026#39;s tool: \u0026#34;writes\u0026#34; code to a file. // In a real application, this would write to disk. type writeCodeTool struct{} func (t *writeCodeTool) Info(ctx context.Context) (*schema.ToolInfo, error) { return \u0026amp;schema.ToolInfo{ Name: \u0026#34;write_code\u0026#34;, Desc: \u0026#34;Write Go code to solve the task. The code will be tested automatically.\u0026#34;, }, nil } func (t *writeCodeTool) InvokableRun(ctx context.Context, args string, opts ...tool.Option) (string, error) { // In a real application: write args to a .go file return fmt.Sprintf(\u0026#34;Code written (%d bytes)\u0026#34;, len(args)), nil } func main() { ctx := context.Background() // 1. Create model for Actor and Reflector chatModel, err := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ Model: \u0026#34;gpt-4o\u0026#34;, }) if err != nil { panic(err) } // 2. Create ReAct agent as Actor // ExportGraph() allows embedding it into compose.Graph codeTool := utils.NewTool(\u0026amp;writeCodeTool{}, nil) actor, err := react.NewAgent(ctx, \u0026amp;react.AgentConfig{ ToolCallingModel: chatModel, ToolsConfig: compose.ToolsNodeConfig{ Tools: []tool.BaseTool{codeTool}, }, MaxStep: 5, }) if err != nil { panic(err) } // 3. Build the Reflexion Graph g := compose.NewGraph[string, string]( compose.WithGenLocalState(func(ctx context.Context) *reflexionState { return \u0026amp;reflexionState{ MaxAttempts: 3, } }), compose.WithMaxRunSteps(20), // cycle limit ) // Actor: embed ReAct agent via ExportGraph actorGraph, actorOpts := actor.ExportGraph() g.AddGraphNode(\u0026#34;actor\u0026#34;, actorGraph, actorOpts...) g.AddEdge(compose.START, \u0026#34;actor\u0026#34;) // Evaluator: Lambda node that \u0026#34;runs tests\u0026#34; g.AddLambdaNode(\u0026#34;evaluator\u0026#34;, compose.InvokableLambda(func(ctx context.Context, code string) (string, error) { // In a real application: exec.Command(\u0026#34;go\u0026#34;, \u0026#34;test\u0026#34;, \u0026#34;./...\u0026#34;) // Here: simulation if len(code) \u0026gt; 10 { return \u0026#34;PASS: all tests passed\u0026#34;, nil } return \u0026#34;FAIL: TestSortEmpty - expected [], got nil\u0026#34;, nil }), ) g.AddEdge(\u0026#34;actor\u0026#34;, \u0026#34;evaluator\u0026#34;) // Reflector: ChatModel analyzes errors g.AddChatModelNode(\u0026#34;reflector\u0026#34;, chatModel) g.AddEdge(\u0026#34;evaluator\u0026#34;, \u0026#34;reflector\u0026#34;) // Conditional Branch: pass → END, fail → back to Actor g.AddBranch(\u0026#34;evaluator\u0026#34;, compose.NewGraphBranch( func(ctx context.Context, testResult string) (string, error) { // Read state to check attempt count if err := compose.ProcessState[*reflexionState](ctx, func(ctx context.Context, s *reflexionState) error { s.Attempt++ s.TestResult = testResult // Simple heuristic criterion s.Passed = len(testResult) \u0026gt; 4 \u0026amp;\u0026amp; testResult[:4] == \u0026#34;PASS\u0026#34; }, ); err != nil { return \u0026#34;\u0026#34;, err } var target string _ = compose.ProcessState[*reflexionState](ctx, func(ctx context.Context, s *reflexionState) error { if s.Passed || s.Attempt \u0026gt;= s.MaxAttempts { target = compose.END } else { target = \u0026#34;reflector\u0026#34; // reflect first, then retry } return nil }, ) return target, nil }, map[string]bool{compose.END: true, \u0026#34;reflector\u0026#34;: true}, )) // Reflector → Actor: retry with reflection in context g.AddEdge(\u0026#34;reflector\u0026#34;, \u0026#34;actor\u0026#34;) // END g.AddEdge(compose.END, compose.END) // 4. Compile and run runnable, err := g.Compile(ctx) if err != nil { panic(fmt.Sprintf(\u0026#34;compile error: %v\u0026#34;, err)) } result, err := runnable.Invoke(ctx, \u0026#34;Write a function that sorts a slice of integers\u0026#34;) if err != nil { panic(fmt.Sprintf(\u0026#34;invoke error: %v\u0026#34;, err)) } fmt.Println(\u0026#34;Result:\u0026#34;, result) } What\u0026rsquo;s happening here State — reflexionState stores the current attempt, reflections, and evaluation result. Created via WithGenLocalState on each graph run.\nActor — embedded via ExportGraph(). A ReAct agent with a write_code tool. On re-entry (after reflection), it receives updated context.\nEvaluator — a Lambda node that \u0026ldquo;runs tests\u0026rdquo;. In a real application — exec.Command(\u0026quot;go\u0026quot;, \u0026quot;test\u0026quot;). In the example — simulation: if the code is longer than 10 bytes — PASS.\nReflector — a ChatModel analyzing errors. Receives test results and generates verbal reflection.\nBranch — conditional branching after Evaluator: PASS → END, FAIL → Reflector → Actor (retry). Checks Attempt \u0026lt; MaxAttempts.\nCycle — g.AddEdge(\u0026quot;reflector\u0026quot;, \u0026quot;actor\u0026quot;) closes the loop. WithMaxRunSteps(20) limits the total number of steps (protection against infinite loops).\n9. Engineering scenario Code generation with tests is the ideal scenario for Reflexion. Why? Because there\u0026rsquo;s an objective external verifier: the compiler and unit tests. This isn\u0026rsquo;t a subjective LLM assessment — it\u0026rsquo;s a binary PASS/FAIL.\nTDD for agents sequenceDiagram participant Dev as Developer participant A as Actor Agent participant T as go test participant R as Reflector participant M as Memory Dev-\u003e\u003eA: \"Write sort([]int)\" Note over A: Attempt 1 A-\u003e\u003eT: sort.go T--\u003e\u003eA: ❌ FAIL: TestSortEmptyexpected [], got nil A-\u003e\u003eR: Trajectory + test failure R-\u003e\u003eR: \"Forgot to handleempty slice\" R-\u003e\u003eM: Store reflection #1 Note over A: Attempt 2 (with reflection) M--\u003e\u003eA: \"Check empty slice first\" A-\u003e\u003eT: sort_v2.go T--\u003e\u003eA: ❌ FAIL: TestSortStableunstable sort on equal elements A-\u003e\u003eR: Trajectory + test failure R-\u003e\u003eR: \"Used unstable sort,need stable\" R-\u003e\u003eM: Store reflection #2 Note over A: Attempt 3 (with 2 reflections) M--\u003e\u003eA: \"Check empty slice firstUse stable sort\" A-\u003e\u003eT: sort_v3.go T--\u003e\u003eA: ✅ All tests passed A--\u003e\u003eDev: sort_v3.go ✅ Why this works better than Self-Refine Self-Refine in the same scenario would give 0% improvement on Math Reasoning (Madaan et al., 2023). Why? Without tests, the model can\u0026rsquo;t distinguish sort([]int{}) from sort([]int{1}) — it lacks an objective signal. Reflexion with go test as Evaluator solves this problem: tests provide precise error diagnostics → reflection focuses on the real problem → the Actor fixes exactly what\u0026rsquo;s needed.\nProduction implementation In production, the scenario expands:\nComponent Example In production Actor react.NewAgent with write_code + read_file, search_docs, lint_code Evaluator go test ./... + go vet, golangci-lint, coverage ≥ 80% Reflector ChatModel \u0026ldquo;analyze failures\u0026rdquo; Prompt with specific error patterns Memory []string in state Redis / file with reflection history Max attempts 3 5 (HumanEval: 91% achieved in 12 attempts, but 3-5 usually sufficient) 10. Practical recommendations When to apply Reflexion Scenario Applicability Rationale Code generation + tests ✅ Excellent Objective verifier (compiler/tests) Game agents ✅ Excellent Environment provides clear reward signal Data pipeline with validation ✅ Good Schema validation as verifier Code review automation ⚠️ Cautious LLM assessment less reliable than tests Creative writing ⚠️ Cautious No objective success criterion Math / reasoning ❌ Not recommended Without external verifier — degrades results Hyperparameter tuning Max attempts: 3-5 for tasks with a fast verifier (tests). HumanEval reached 91% in 12 attempts, but diminishing returns start after 3-5. More is more expensive, but not better.\nEpisodic memory size: store the last 5-10 reflections. Too many — the context grows and the model loses focus. Too few — doesn\u0026rsquo;t account for older mistakes.\nEvaluator choice: a deterministic verifier (tests, compiler) is always better than LLM-based. If the verifier is unreliable — Reflexion degrades into Self-Refine with its problems.\nReflector prompt: specific \u0026gt; abstract. Not \u0026ldquo;analyze the error\u0026rdquo;, but \u0026ldquo;identify: (1) which test failed, (2) what input caused the failure, (3) what assumption was wrong, (4) what to change in the code\u0026rdquo;.\nCost management Each Reflexion attempt = a full Actor + Evaluator + Reflector cycle. With 3 attempts — 3x cost. Mitigations:\nUse a cheap model for the Evaluator (deterministic checking doesn\u0026rsquo;t require GPT-4) Stop early: if 2 attempts didn\u0026rsquo;t help — the third probably won\u0026rsquo;t either Cache reflections for similar tasks 11. Summary The Reflexion pattern is ReAct + self-assessment + episodic memory. Key takeaways:\nExternal verifier is mandatory. Without it, self-correction degrades results (Huang et al., 2023). With it — Reflexion delivers +11% on HumanEval and +22% on AlfWorld.\nEpisodic memory is the key difference from Self-Refine. Not just critique-refine, but accumulating verbal lessons between attempts. This transforms a one-shot agent into a learning one.\nNot a silver bullet. Reflexion doesn\u0026rsquo;t work on tasks without an objective success criterion and can degrade results when initial accuracy is already high.\nIn Eino — compose.Graph with a cycle. Workflow doesn\u0026rsquo;t work (no cycles). Graph + AddEdge(\u0026quot;reflector\u0026quot;, \u0026quot;actor\u0026quot;) + WithMaxRunSteps — the natural implementation.\nWhat\u0026rsquo;s next? In Part 4 — Multi-Agent patterns: when one agent isn\u0026rsquo;t enough, and you need a team. And the topic of agent memory — long-term, episodic, semantic — I\u0026rsquo;ll cover in detail in Part 6.\nReflexion: Shinn, N., Cassano, F., Labash, A., Gopinath, A., Narasimhan, K., \u0026amp; Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. Self-Refine: Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., \u0026hellip; \u0026amp; Yang, D. (2023). Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. Constitutional AI: Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., \u0026hellip; \u0026amp; Kaplan, J. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. Huang et al.: Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, Q., \u0026amp; Zhou, D. (2023). Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024. Reflect, Retry, Reward: Bensal, Y., Kim, S., Bosselut, A., \u0026amp; Guha, N. (2025). Reflect, Retry, Reward: Training LLM Agents to Reflect and Retry with Reward-Guided Self-Reflection. Eino: CloudWeGo. Eino: The ultimate LLM/AI application development framework in Go. deer-go: CloudWeGo. DEER-flow Go implementation. ","permalink":"https://triumphpc.github.io/blog/posts/ai-agent-design-patterns-3-reflexion/","summary":"Reflexion is a pattern where an agent learns from its own mistakes through verbal reinforcement and episodic memory. I trace the evolution of self-critique from Constitutional AI to Reflexion, honestly show its limitations (doesn\u0026rsquo;t work without an external verifier), and implement a working example in Go using Eino compose.Graph with an Actor → Evaluator → Reflector → retry loop.","title":"AI Agent Design Patterns. Part 3: Reflexion Pattern"},{"content":"1. Introduction I\u0026rsquo;m starting a series on AI agent design patterns. The goal: a reference built on original research papers, working Go code, and honest limitation analysis. Every claim traces back to the source — no secondary summaries. Audience — experienced engineers who don\u0026rsquo;t need basics explained.\nReAct Pattern (Yao et al., 2022) is the foundation upon which Reflexion, Plan-and-Execute, ReWOO, and all agent architectures are built. Without it, the other patterns don\u0026rsquo;t form a coherent system.\n2. What is ReAct Pattern History In October 2022, a team from Princeton University and Google Brain — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao — published «ReAct: Synergizing Reasoning and Acting in Language Models». Project page: react-lm.github.io.\nThe idea is simple: an LLM should not only reason (as in Chain-of-Thought) or only act (as in tool-use agents), but interleave reasoning and action, forming a closed feedback loop.\nAnalogy with Human Thinking When you solve a complex problem — say, navigating an unfamiliar city — you don\u0026rsquo;t plan the entire route in your head and then execute. You: check the map → think → walk → see a sign → adjust → keep going. This very pattern — interleaved Reasoning + Acting — is what ReAct formalizes.\nContrast with Predecessors Approach Reasoning Action Grounding Self-correction CoT (Wei et al., 2022, arXiv:2201.11903) Yes No No No Act-only No Yes Yes No ReAct Yes Yes Yes Yes CoT generates a chain of reasoning but cannot verify facts. Act-only calls tools but doesn\u0026rsquo;t reflect on results. ReAct combines both worlds.\nFormal Definition A ReAct agent operates in a loop:\ngraph LR Q((Question)) --\u003e T1[Thought] T1 --\u003e A1[Action] A1 --\u003e O1[Observation] O1 --\u003e T2[Thought] T2 --\u003e A2[Action] A2 --\u003e O2[Observation] O2 --\u003e TN[...] TN --\u003e Ans((Answer)) Example trace (task: \u0026ldquo;What\u0026rsquo;s the weather in Paris?\u0026rdquo;):\nThought: I need to check the current weather in Paris Action: get_weather(city=\u0026#34;Paris\u0026#34;) Observation: 18°C, clear, humidity 45% Thought: Data received, I can answer now Answer: It\u0026#39;s currently 18°C in Paris, clear skies, humidity 45% Each Thought is the model\u0026rsquo;s reasoning about what to do next. Each Action is a tool invocation. Each Observation is a result the model considers in the next step.\n3. What Problem Does It Solve The CoT Problem: Hallucination Chain-of-Thought impresses on tasks where the answer can be derived from context. But as soon as external facts are needed — the model hallucinates. Classic example: CoT confidently generates \u0026ldquo;the president of Nepal is Hari Bahadur Basnet\u0026rdquo; — the name sounds plausible but is factually wrong. No grounding — no guarantee.\nThe Act-only Problem: No Reflection An agent that only acts can call a tool and get a result, but cannot synthesize an answer. It doesn\u0026rsquo;t \u0026ldquo;think\u0026rdquo; about the observation — just passes it along. This works for simple queries but breaks on multi-step tasks.\nThe Error Propagation Problem In long CoT chains, an error at an early step goes unnoticed and cascades, making the final answer nonsensical. Without the ability to verify through an external environment, the model cannot course-correct.\nWhat ReAct Provides Problem ReAct Mechanism Hallucination Grounding through tool calls (Observation = fact) No reflection Thought after each Observation — model analyzes the result Error propagation Self-correction: model sees erroneous result and adjusts plan Opacity Interpretability: each Thought is a readable reasoning trace Rigid plan Dynamic planning: plan is revised at each step Numbers from Yao et al., 2022 Key results from the original paper:\nHotpotQA + FEVER: ReAct overcomes CoT hallucination through Wikipedia API — the model retrieves facts instead of making them up (Yao et al., 2022, §4.1). ALFWorld: +34% success rate over imitation learning — in a text-based home interaction environment, ReAct significantly outperforms the baseline (Yao et al., 2022, §4.2). WebShop: +10% success rate over reinforcement learning — even in the complex online shopping task, ReAct demonstrates an advantage (Yao et al., 2022, §4.2). Pattern Applicability Importantly: the pattern has been validated across a broad class of models and tasks. The specific numbers from Yao et al. were obtained on PaLM-540B, but the interleaved reasoning + acting principle is model-independent — it works with GPT-4, Claude, and open-source models that support tool calling.\n4. Architecture and Diagrams The Thought → Action → Observation Loop sequenceDiagram participant U as User participant A as ReAct Agent participant T as Tools U-\u003e\u003eA: Question loop ReAct Loop A-\u003e\u003eA: Thought (reasoning) A-\u003e\u003eT: Action (tool call) T--\u003e\u003eA: Observation (result) end A-\u003e\u003eA: Final Thought A--\u003e\u003eU: Answer At each iteration, the agent forms a Thought — internal reasoning about the current state and next step. Then it executes an Action — calling one of the available tools. The resulting Observation is added to the context, and the cycle repeats. State (message history) accumulates: all previous Thoughts, Actions, and Observations are available at the next step.\nState-Graph Representation In frameworks like Eino, ReAct is implemented as a directed graph (State Graph):\ngraph TB START((START)) --\u003e CM[ChatModel] CM --\u003e|tool_calls| B{Branch} CM --\u003e|no tool_calls| END((END)) B --\u003e|has tool calls| TN[ToolsNode] B --\u003e|no tool calls| END TN --\u003e CM The graph representation matters for three reasons:\nState: all messages are stored in a single graph state — no manual context management needed. Streaming: each graph node can stream its result — the user sees the agent\u0026rsquo;s reasoning in real time. Callbacks: handlers can be attached to each node — logging, metrics, tracing. Evolution of Approaches 2022–2025 graph LR A[\"Prompt-basedReAct (2022)\"] --\u003e B[\"Tool CallingAPI (2023)\"] B --\u003e C[\"AgentFrameworks (2024)\"] C --\u003e D[\"Multi-AgentOrchestration (2025)\"] Prompt-based ReAct (2022): original implementation — few-shot prompts, tools via text interface. Worked but fragile and non-scalable. Tool Calling API (2023): models gained native function calling support — tools became structured and reliable. Schick et al., 2023 (arXiv:2302.04761) showed that LLMs can learn to call tools autonomously. Agent Frameworks (2024): LangGraph, LlamaIndex, Eino — frameworks that encapsulate ReAct into reusable components with graph architecture. Multi-Agent Orchestration (2025): multiple ReAct agents coordinate to solve complex tasks — each specialized in its domain. 5. Framework Landscape ReAct is a universal pattern, implemented across all major frameworks. Summary table:\nFramework Language Implementation Key Feature LangGraph Python create_react_agent De facto standard in Python, graph model LlamaIndex Python ReActAgent Deep RAG and index integration OpenAI Agents SDK Python Agent + tools Native GPT-4/GPT-4o integration Anthropic Claude API Python/TS Tool use + system prompt Maximum reasoning via extended thinking Google ADK Python Agent + tools Gemini and Google Cloud integration Eino Go react.NewAgent Go-native, production-tested at ByteDance LangGraph — De Facto Standard LangGraph has become the standard for agent systems in the Python world. The create_react_agent function creates a ready-made ReAct agent from a model and tool list in a few lines. The graph architecture allows conditional transitions, loops, and human-in-the-loop patterns. If you\u0026rsquo;re in Python — this is the first candidate.\nEino — Why Go Eino is a framework from CloudWeGo (ByteDance), production-tested in Doubao, TikTok, and Coze. Chosen for the practical section for three reasons:\nGo-native: typed tools, interfaces, no interface{} chaos. Production-tested: serves hundreds of millions of requests per day inside ByteDance. Graph architecture: compose.Graph under the hood of the ReAct agent — the same model as LangGraph, but in Go. No Python code examples are provided in this article — this is intentional. This blog focuses on Go.\n6. Practice: ReAct with Eino, Go Installation go get github.com/cloudwego/eino@latest go get github.com/cloudwego/eino-ext/components/model/openai@latest API documentation: pkg.go.dev/github.com/cloudwego/eino\nMinimal ReAct Agent package main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;github.com/cloudwego/eino-ext/components/model/openai\u0026#34; \u0026#34;github.com/cloudwego/eino/components/tool\u0026#34; \u0026#34;github.com/cloudwego/eino/compose\u0026#34; \u0026#34;github.com/cloudwego/eino/flow/agent/react\u0026#34; \u0026#34;github.com/cloudwego/eino/schema\u0026#34; ) func main() { ctx := context.Background() // Model with tool calling support chatModel, _ := openai.NewChatModel(ctx, \u0026amp;openai.ChatModelConfig{ Model: \u0026#34;gpt-4o\u0026#34;, }) // Create agent with a single tool agent, _ := react.NewAgent(ctx, \u0026amp;react.AgentConfig{ ToolCallingModel: chatModel, ToolsConfig: compose.ToolsNodeConfig{ InvokableTools: []tool.InvokableTool{weatherTool()}, }, }) // Invoke agent msg, _ := agent.Generate(ctx, []*schema.Message{ schema.UserMessage(\u0026#34;What\u0026#39;s the weather in Paris?\u0026#34;), }) fmt.Println(msg.Content) } Typed Tool via utils.NewTool type WeatherRequest struct { City string `json:\u0026#34;city\u0026#34; jsonschema:\u0026#34;description=City to get weather for\u0026#34;` } type WeatherResponse struct { City string `json:\u0026#34;city\u0026#34;` Temperature int `json:\u0026#34;temperature\u0026#34;` Condition string `json:\u0026#34;condition\u0026#34;` } func weatherTool() tool.InvokableTool { return utils.NewTool( \u0026amp;schema.ToolInfo{ Name: \u0026#34;get_weather\u0026#34;, Desc: \u0026#34;Get current weather for a specified city\u0026#34;, ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ \u0026#34;city\u0026#34;: {Type: \u0026#34;string\u0026#34;, Desc: \u0026#34;City\u0026#34;, Required: true}, }), }, func(ctx context.Context, input *WeatherRequest) (*WeatherResponse, error) { // Real weather API call goes here return \u0026amp;WeatherResponse{ City: input.City, Temperature: 18, Condition: \u0026#34;clear\u0026#34;, }, nil }, ) } Key Parameters (without code) ToolCallingModel — model must support tool calling (ToolCallingChatModel interface). ToolsConfig — tool node configuration: InvokableTools and StreamableTools. MaxStep — graph step limit (default 12 = up to 6 full ChatModel + Tools cycles). MessageModifier — function to modify messages before model call (e.g., adding system prompt). ToolReturnDirectly — tools whose result is returned directly to the user, bypassing the next model call. Links for Deep Dive ReAct Agent Manual — complete parameter reference eino-examples/flow/agent/react — full working example (Food Recommender demo) GitHub: cloudwego/eino — source code 7. When ReAct is NOT the Right Choice ReAct is not a silver bullet. Each Thought→Action→Observation cycle is a separate LLM call, meaning: cost grows linearly with the number of steps, latency accumulates, and the planning horizon is limited by the model\u0026rsquo;s context window.\nHierarchy of Complexity The Microsoft Azure Architecture Center\u0026rsquo;s guide to AI agent orchestration articulates a principle: use the lowest level of complexity that reliably solves the problem.\nLevel Description When it\u0026rsquo;s enough Direct model call Single LLM call, no tools Classification, summarization, translation Single agent + tools (ReAct) Agent with reasoning and tools Dynamic tool selection within a single domain Multi-agent orchestration Multiple specialized agents Cross-domain tasks, distinct security boundaries ReAct occupies the middle level. If a direct model call solves the task — you don\u0026rsquo;t need an agent. If a single agent can\u0026rsquo;t cope due to prompt complexity, tool overload, or security requirements — move to multi-agent. But not before.\nIf your task\u0026hellip; \u0026hellip;consider this alternative Has a fixed workflow with known steps Plan-and-Solve — planning without iterative search Is cost-sensitive (many LLM calls) ReWOO — planning without interleaved model calls (Xu et al., 2023) Requires learning from past mistakes Reflexion (a.k.a. maker-checker, evaluator-optimizer) — ReAct + self-evaluation (Shinn et al., 2023) Needs a tree of hypotheses Tree of Thoughts — branching reasoning (Yao et al., 2023) Has a long planning horizon Plan-and-Execute — decomposition into subtasks A detailed comparison of patterns — in Part 5 of this series.\n8. What\u0026rsquo;s Next in the Series Part 2: Plan-and-Execute Pattern — decomposing tasks into subtasks with a separate planner and executor. Part 3: Reflexion Pattern — ReAct + self-evaluation: an agent that learns from its own mistakes. Part 4: ReWOO Pattern — planning without interleaved model calls: cheaper, faster, but without dynamic adjustment. Part 5: Pattern Comparison + Multi-Agent Orchestration — when to choose which pattern, and how multiple agents coordinate for complex tasks. Discussion welcome — comments on the site or GitHub Issues.\n9. References Research Papers Yao et al., 2022 — ReAct: Synergizing reasoning and acting in language models. Formalization of the Thought→Action→Observation loop. arXiv:2210.03629 Wei et al., 2022 — Chain-of-Thought: Step-by-step reasoning without external environment interaction. Predecessor to ReAct. arXiv:2201.11903 Schick et al., 2023 — Toolformer: LLMs learn to call tools autonomously. Bridge between prompt-based and API-based approaches. arXiv:2302.04761 Shinn et al., 2023 — Reflexion: Extending ReAct with self-evaluation and verbal reinforcement. arXiv:2303.11366 Yao et al., 2023 — Tree of Thoughts: Generalizing CoT to a tree of hypotheses with search. arXiv:2305.10601 Xu et al., 2023 — ReWOO: Planning without interleaved model calls — demonstrates ReAct\u0026rsquo;s cost/latency limitations. arXiv:2305.18323 Documentation and Examples Eino ReAct Agent Manual — all parameters and configuration Eino Open Source Announcement — framework overview GitHub: cloudwego/eino — source code GitHub: eino-examples/flow/agent/react — full working example (Food Recommender) LangGraph ReAct Agent template — Python implementation React Project Page — original paper\u0026rsquo;s project page Architecture Guides AI agent design patterns — Microsoft Azure Architecture Center — complexity hierarchy (direct call → single agent → multi-agent), orchestration patterns, production recommendations for reliability, security, cost optimization ","permalink":"https://triumphpc.github.io/blog/posts/ai-agent-design-patterns-1-react/","summary":"ReAct (Reasoning + Acting) — the foundational AI agent design pattern that interleaves reasoning and action in a single loop. I break down the architecture, numbers from Yao et al. 2022, framework landscape, and a minimal working Go example via Eino.","title":"AI Agent Design Patterns. Part 1: ReAct Pattern"},{"content":"Why Hugo? After evaluating custom SSG options and Go-based generators, I chose Hugo for three reasons:\nCriteria Hugo Custom SSG Likho Build speed ~40ms N/A ~200ms Theme ecosystem 400+ None Minimal Markdown features Full DIY Basic Maintenance Community You You Hugo gives me speed, a mature ecosystem, and the PaperMod theme that provides everything I need out of the box.\nArchitecture Overview The blog follows a simple pipeline — write Markdown, push to GitHub, and everything else is automated:\ngraph LR A[\"✏️ Markdown\"] --\u003e B[\"🔨 Hugo Build\"] B --\u003e C[\"⏱️ GitHub Actions\"] C --\u003e D[\"🚀 GitHub Pages\"] style A fill:#24283b,stroke:#7aa2f7,color:#c0caf5 style B fill:#24283b,stroke:#bb9af7,color:#c0caf5 style C fill:#24283b,stroke:#7dcfff,color:#c0caf5 style D fill:#24283b,stroke:#9ece6a,color:#c0caf5 Setup Highlights Hugo Modules over Git Submodules hugo mod init github.com/triumumphc/blog In hugo.yaml:\nmodule: imports: - path: github.com/adityatelange/hugo-PaperMod No git submodule headaches — just Go modules, versioned and reproducible.\nPaperMod Dark Mode Dark theme by default, with a toggle for light mode. The color palette is inspired by Tokyo Night:\npie title Color Palette Distribution \"Primary (#7aa2f7)\" : 35 \"Accent (#bb9af7)\" : 25 \"Success (#9ece6a)\" : 20 \"Warning (#e0af68)\" : 10 \"Error (#f7768e)\" : 10 Code Copy Buttons Every code block gets a copy button automatically. Go code looks like this:\nfunc main() { mux := http.NewServeMux() mux.HandleFunc(\u0026#34;/\u0026#34;, handler) srv := \u0026amp;http.Server{ Addr: \u0026#34;:8080\u0026#34;, Handler: mux, ReadTimeout: 5 * time.Second, WriteTimeout: 10 * time.Second, } log.Fatal(srv.ListenAndServe()) } Mermaid Diagrams Diagrams are rendered client-side via CDN, only loaded when the mermaid shortcode is used:\nsequenceDiagram participant W as Writer participant G as Git participant CI as GitHub Actions participant P as GitHub Pages W-\u003e\u003eG: git push G-\u003e\u003eCI: webhook trigger CI-\u003e\u003eCI: hugo --minify CI-\u003e\u003eP: deploy artifact P--\u003e\u003eW: site live ✅ C4 architecture style:\ngraph TB subgraph \"Authoring\" MD[Markdown Files] IMG[Images \u0026 Assets] end subgraph \"Build Pipeline\" HUGO[Hugo v0.162] CSS[Custom CSS] SC[Shortcodes] end subgraph \"Hosting\" GHA[GitHub Actions] GHP[GitHub Pages] end MD --\u003e HUGO IMG --\u003e HUGO CSS --\u003e HUGO SC --\u003e HUGO HUGO --\u003e GHA GHA --\u003e GHP Math with KaTeX KaTeX renders math client-side, loaded only when the katex shortcode is present.\nEuler\u0026rsquo;s Identity The most beautiful equation in mathematics:\n$$ e^{i\\pi} + 1 = 0 $$Gaussian Integral $$ \\int_{-\\infty}^{\\infty} e^{-x^2} dx = \\sqrt{\\pi} $$Normal Distribution The probability density function of a normal distribution:\n$$ f(x) = \\frac{1}{\\sigma\\sqrt{2\\pi}} e^{-\\frac{1}{2}\\left(\\frac{x-\\mu}{\\sigma}\\right)^2} $$Inline math works too: the mean is $\\mu$ and standard deviation is $\\sigma$.\nMatrix Operations $$ A = \\begin{pmatrix} a_{11} \u0026 a_{12} \\\\ a_{21} \u0026 a_{22} \\end{pmatrix}, \\quad \\det(A) = a_{11}a_{22} - a_{12}a_{21} $$Shortcodes Reference Shortcode Purpose Example {{\u0026lt; mermaid \u0026gt;}} Mermaid diagrams Flowcharts, sequences, C4 $$ ... $$ Display math Centered equations $ ... $ Inline math $\\mu$ renders as μ Custom Styling The dark theme extends PaperMod with:\nCode blocks — custom background + border with rounded corners Links — subtle underline on hover Blockquotes — accent-colored left border with surface background Tables — bordered with surface header row CI/CD Pipeline GitHub Actions workflow handles everything:\nname: Deploy Hugo Blog on: push: branches: [main] The pipeline:\nInstalls Hugo Extended Checks out the repo Caches Hugo Modules Builds with hugo --minify Deploys to GitHub Pages What\u0026rsquo;s Next Go concurrency patterns — goroutines, channels, errgroup LLM integration — RAG pipelines, prompt engineering AI Agent architecture — agentic orchestration, tool use Stay tuned!\n","permalink":"https://triumphpc.github.io/blog/posts/getting-started/","summary":"How this blog was born — Hugo, PaperMod theme, Mermaid diagrams, KaTeX math, custom dark styling, and fully automated CI/CD via GitHub Actions.","title":"Getting Started with Hugo + PaperMod"},{"content":"Hi, I\u0026rsquo;m Sebastian PE — a full-stack software engineer with 15+ years of experience building web products, backend systems, and developer tools.\nWhat I Do I work across the entire product lifecycle — from architecture and prototyping to launch and operation. My current focus:\nGo — high-performance backend services, concurrent systems, microservices AI \u0026amp; LLM Agents — prompt engineering, RAG pipelines, agentic orchestration PHP \u0026amp; JavaScript — full-stack web development, API design DevOps — CI/CD, observability, infrastructure as code I\u0026rsquo;ve built e-commerce platforms, web portals, CRM and ERP systems, and business tools. I also have extensive experience as a team lead and head of development, managing engineering teams and designing software architecture.\nTeaching \u0026amp; Community I share my knowledge through courses and content:\nUdemy — Design Patterns, Domain-Driven Design for Junior YouTube — channel (architecture, methodologies, dev tools) Habr — blog posts Connect GitHub: triumphpc Udemy: instructor profile This blog is built with Hugo and the PaperMod theme.\n","permalink":"https://triumphpc.github.io/blog/pages/about/","summary":"\u003cp\u003eHi, I\u0026rsquo;m \u003cstrong\u003eSebastian PE\u003c/strong\u003e — a full-stack software engineer with \u003cstrong\u003e15+ years\u003c/strong\u003e of experience building web products, backend systems, and developer tools.\u003c/p\u003e\n\u003ch2 id=\"what-i-do\"\u003eWhat I Do\u003c/h2\u003e\n\u003cp\u003eI work across the entire product lifecycle — from architecture and prototyping to launch and operation. My current focus:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGo\u003c/strong\u003e — high-performance backend services, concurrent systems, microservices\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAI \u0026amp; LLM Agents\u003c/strong\u003e — prompt engineering, RAG pipelines, agentic orchestration\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePHP \u0026amp; JavaScript\u003c/strong\u003e — full-stack web development, API design\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDevOps\u003c/strong\u003e — CI/CD, observability, infrastructure as code\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI\u0026rsquo;ve built e-commerce platforms, web portals, CRM and ERP systems, and business tools. I also have extensive experience as a team lead and head of development, managing engineering teams and designing software architecture.\u003c/p\u003e","title":"About"},{"content":"Open Source Project Description blog This blog — Hugo + PaperMod, auto-deployed via GitHub Actions Teaching Udemy courses on software engineering — profile\n","permalink":"https://triumphpc.github.io/blog/pages/projects/","summary":"\u003ch2 id=\"open-source\"\u003eOpen Source\u003c/h2\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eProject\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eDescription\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ca href=\"https://github.com/triumphpc/blog\"\u003eblog\u003c/a\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eThis blog — Hugo + PaperMod, auto-deployed via GitHub Actions\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"teaching\"\u003eTeaching\u003c/h2\u003e\n\u003cp\u003eUdemy courses on software engineering — \u003ca href=\"https://www.udemy.com/user/sergei-1146/\"\u003eprofile\u003c/a\u003e\u003c/p\u003e","title":"Projects"}]