Implementing Semantic Memory in AI Agents: Vector-Backed Long-Term Context

# Implementing Semantic Memory in AI Agents: Vector-Backed Long-Term Context


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The Context-Window Amnesia Problem

Every multi-day agent deployment I have audited has the same failure mode, and it never shows up in a stack trace. The agent works beautifully on day one — it remembers the ticket, the customer, the constraints, the decisions it already made. Then the session ends, the process restarts, and on day two the agent greets the same user as a total stranger. It re-asks for information it was given yesterday. It re-proposes an approach that was explicitly rejected. It contradicts a decision that took an hour of back-and-forth to reach.

This is **context-window amnesia**, and it is the defining architectural weakness of long-horizon agentic systems. The context window is a cache, not a memory. It is scoped to a single session, it is volatile the moment the process exits, and it is fundamentally finite — 128K, 200K, a million tokens, the ceiling keeps moving but the constraint never disappears. An agent that runs a procurement workflow over three weeks, or a support escalation that spans four handoffs, or a research task that accumulates forty documents, cannot fit its working history into any window. Even when it technically could, stuffing a week of transcript into every prompt is the fastest way to burn budget and degrade reasoning quality through lost-in-the-middle dilution.

The industry’s default response has been *bigger windows*. That is a hardware answer to a data-architecture problem. The right answer is **memory as a first-class system**: durable, queryable, scoped to a user, and hydrated into the context window on demand. When memory is externalized into a vector-backed store, the agent stops treating “what happened before” as a scrolling transcript and starts treating it as a retrieval problem — because that is exactly what it is.

This post walks through how we design that system in production: a tiered memory architecture, the retrieval patterns that hydrate context at runtime, the multi-tenant isolation model that keeps user data from bleeding across boundaries, a working Python + pgvector implementation of a memory manager, and the ROI math that makes durable memory a cost-reduction program rather than a nice-to-have.

—

## 2. Architecture: A Tiered Memory System

The mistake most teams make is treating “memory” as a single thing — one vector store, one `remember()` function, one `recall()` at the top of every turn. That collapses three genuinely different access patterns into one, and it produces a system that is simultaneously too slow for hot-path context and too coarse for long-term recall.

We design memory in three tiers, each optimized for its own access pattern and retention horizon.

| Tier | Scope | Storage | Access latency | Retention | Typical content |
|—|—|—|—|—|—|
| **Buffer** | Current turn | In-process | Microseconds | Seconds | Latest message, tool output, scratchpad |
| **Short-Term** | Current session/task | Redis / in-memory KV | Single-digit ms | Hours to days | Rolling summary, active plan, recent turns |
| **Long-Term** | Cross-session, per-user | Postgres + pgvector | 10–50 ms | Indefinite | Facts, preferences, decisions, episodic events |

### The buffer: the working set

The buffer is the raw, unprocessed tail of the conversation — the current turn, the last tool result, the model’s scratchpad. It is deliberately ephemeral and must never be persisted to durable storage as-is. Nothing valuable survives here; its job is to give the model immediate continuity within a turn.

### The short-term tier: the session working memory

Short-term memory covers everything the agent needs to stay coherent *within* a task, without carrying a full transcript. Two mechanisms do the work:

– **A rolling summary**, regenerated incrementally. Rather than keeping every turn verbatim, the agent maintains a compressed narrative — decisions made, open questions, current state — refreshed every N turns or when the token budget for the session crosses a threshold.
– **Structured task state**, kept outside the prompt entirely: the active plan, step index, scratch variables, idempotency keys. This is state management, not text, and it should be read and written as data — an agent reading its own scratch state from a KV store is far more reliable than one trying to re-parse its own summary prose.

The short-term tier is also where **eviction policy** lives. When the session budget is exceeded, the agent summarizes, promotes anything durable to long-term memory, and drops the rest. This is the moment where an important fact either graduates to durable memory or is lost forever — get the promotion heuristic right and the agent’s memory compounds; get it wrong and it forgets exactly the things that matter.

### The long-term tier: durable, semantic, queryable

Long-term memory is where the vector database earns its place. Every durable memory is stored as a record with an embedding, so it can be retrieved by *meaning* rather than by timestamp or key. We generally store four categories of memory, each with its own embedding strategy and decay behavior:

– **Semantic facts** — stable truths about the user or domain (“uses Oracle EBS 12.2”, “prefers concise summaries”). Long-lived, rarely invalidated.
– **Episodic events** — what happened and when (“escalated ticket #4821 on March 3”). Queryable by recency and similarity.
– **Procedural memory** — how this user’s environment works (“their API requires a PATCH with a session header”). The highest-value category in enterprise deployments, because it is pure accumulated operational knowledge.
– **Preferences and constraints** — explicit guardrails the agent must never violate.

The critical design property is that long-term memory is **retrieved, not injected wholesale**. The context window receives only the handful of memories relevant to the current turn, which keeps prompts small and signals dense.

—

## 3. Retrieval Patterns: Hydrating Context at Runtime

A tiered store is only as good as the retrieval that feeds it into the prompt. The naive pattern — embed the user’s message, fetch the top-5, paste them in — is a starting point, not a design. Production semantic memory uses a richer set of patterns.

**Query construction is the first lever.** The raw user message is a noisy query. The reliable approach is to build the retrieval query from *three* inputs: the current message, the active task summary, and the last assistant action. This produces a query that reflects intent rather than surface wording. When the agent is mid-plan, the task summary often matters more than the literal turn.

**Hybrid retrieval, always.** Dense vectors are excellent at fuzzy semantic match and routinely terrible at exact identifiers — order numbers, part codes, policy IDs, account names. Fuse BM25 keyword search with vector similarity using reciprocal rank fusion so that a memory containing “ticket #4821” is retrievable both when the user says “#4821” and when they describe the underlying issue in different words.

**Recency-aware ranking, not pure similarity.** Long-term memory has a time dimension that generic vector search ignores. Two memories can be equally similar to the query, but the fresher one is usually more relevant — especially in episodic memory, where a preference stated last month should often outrank one from last year. We blend similarity with a recency decay to produce the final score:

“`
final_score = α · cosine_similarity(query, memory)
+ β · recency_decay(now – memory.created_at)
+ γ · importance(memory)
“`

Tune α/β/γ per memory category. Procedural memory leans on similarity; episodic memory leans on recency; constraints (preferences) are allowed to bypass ranking entirely and be injected whenever they match the domain.

**A hit threshold with a floor.** Never inject a memory just because it is the least-bad of the top-5. Apply a minimum score, and if nothing clears it, inject *nothing*. A wrong memory is worse than no memory — it actively misleads the model, and the model has no way to know the memory is stale.

**Bounded hydration.** Set a hard token budget for the memory block (we typically cap it at 15–25% of the total context) and enforce it at retrieval time, not in the prompt template. Sort by final score, append until the budget is exhausted, and stop. This is the single most effective control against the exact context bloat that memory systems are supposed to eliminate.

**Write-side discipline matters as much as read.** Deduplicate near-identical memories before insert (a similarity check against recent entries), version and supersede contradictory facts rather than accumulating them, and attach provenance to every memory so the agent can cite *why* it believes something. Memory that cannot be audited will eventually be trusted too much.

—

## 4. Privacy: User-Scoped Memory Isolation in Multi-Tenant Systems

Semantic memory is the most privacy-sensitive component in an agent stack. It stores exactly the things that must never leak: preferences, identities, past decisions, operational details. In a multi-tenant deployment, a single filtering bug is a data-breach event, not a quality regression.

The non-negotiable principle is **isolation enforced at the storage layer, not in the prompt**. Never retrieve broadly and then filter with application logic, and never rely on the model to “only use the current user’s memories.” Both approaches fail under prompt injection and under ordinary coding mistakes.

The patterns we implement:

**Tenant-scoped partitions or schemas.** Separate Postgres schemas per tenant, or at minimum a mandatory `tenant_id` column enforced by row-level security (RLS). With RLS, even a query that forgets its `WHERE` clause cannot return another tenant’s rows:

“`sql
ALTER TABLE agent_memories ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON agent_memories
USING (tenant_id = current_setting(‘app.current_tenant’)::uuid);
“`

Every connection sets `app.current_tenant` at session start, and the database — not the application — guarantees the boundary.

**User-scoped namespaces inside the tenant.** Memory is not just per-tenant; it is per-user within a tenant. Vector indexes should be partitioned so that a nearest-neighbor search physically cannot traverse another user’s embeddings. In pgvector, combine the RLS policy above with a `user_id` predicate pushed into the index scan, and consider partial indexes per tenant for large deployments.

**The tenant key must be bound to identity, not to input.** Derive `tenant_id` and `user_id` from the authenticated session — a signed token, an OAuth subject claim — never from a request parameter the caller can set. This is the same discipline as identity propagation in MCP tool calls: the boundary is established once, cryptographically, and carried through every downstream operation without re-derivation.

**Isolation must extend to the caching layer.** A semantic cache or query-result cache keyed only on the query text will happily serve one user’s memory to another. Key every cache entry on `(tenant_id, user_id, query_hash)` or disable cross-user caching entirely for memory-derived context.

**Support deletion and export as first-class operations.** GDPR and CCPA require the ability to delete an individual’s data on request. Because memories are embedded, “delete the row” is not enough — the vector, any cached summaries, and any derived records must all be removed. Design a user-scoped deletion path from day one; retrofitting it after you have millions of embeddings is painful.

**Encrypt at rest and minimize what you store.** Encrypt embeddings and payload columns, and apply a retention policy that expires episodic memory that no longer has operational value. The safest memory is the one you never had to keep.

—

## 5. Implementation: A Python + pgvector Memory Manager

Here is a compact but production-shaped implementation. It assumes Postgres with the `pgvector` extension and `psycopg` (v3) with the `pgvector` Python adapter, plus any embedding function (OpenAI, Cohere, or a local sentence-transformer — the interface is the same).

“`python
import hashlib
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

import psycopg
from pgvector.psycopg import register_vector

@dataclass
class Memory:
content: str
kind: str # semantic | episodic | procedural | preference
importance: float # 0.0 – 1.0, set by the writer
created_at: datetime

class AgentMemoryManager:
“””User-scoped, tiered memory with vector recall.”””

def __init__(self, dsn: str, embed_fn, tenant_id: str, user_id: str,
score_floor: float = 0.35):
self.embed = embed_fn
self.tenant_id = tenant_id
self.user_id = user_id
self.score_floor = score_floor
self.conn = psycopg.connect(dsn, autocommit=True)
register_vector(self.conn)
# Bind the tenant to the DB session; RLS enforces isolation.
self.conn.execute(
“SELECT set_config(‘app.current_tenant’, %s, false)”,
(tenant_id,),
)

# ———- write path ———-
def remember(self, content: str, kind: str = “semantic”,
importance: float = 0.5, dedupe_at: float = 0.95) -> None:
vec = self.embed(content)
# Supersede near-duplicates instead of accumulating them.
dup = self.conn.execute(
“””
SELECT id, 1 – (embedding <=> %s::vector) AS sim
FROM agent_memories
WHERE tenant_id = %s AND user_id = %s AND kind = %s
ORDER BY embedding <=> %s::vector
LIMIT 1
“””,
(vec, self.tenant_id, self.user_id, kind, vec),
).fetchone()
if dup and dup[1] >= dedupe_at:
self.conn.execute(
“UPDATE agent_memories SET content = %s, embedding = %s, ”
“created_at = now(), importance = %s WHERE id = %s”,
(content, vec, importance, dup[0]),
)
return
self.conn.execute(
“””
INSERT INTO agent_memories
(tenant_id, user_id, kind, content, embedding, importance)
VALUES (%s, %s, %s, %s, %s, %s)
“””,
(self.tenant_id, self.user_id, kind, content, vec, importance),
)

# ———- read path ———-
def recall(self, query: str, k: int = 6, max_tokens: int = 1200,
half_life_days: float = 30.0) -> list[str]:
qvec = self.embed(query)
rows = self.conn.execute(
“””
SELECT content, importance, created_at,
1 – (embedding <=> %s::vector) AS similarity
FROM agent_memories
WHERE tenant_id = %s AND user_id = %s
ORDER BY embedding <=> %s::vector
LIMIT %s
“””,
(qvec, self.tenant_id, self.user_id, qvec, k * 4),
).fetchall()

now = datetime.now(timezone.utc)
scored = []
for content, importance, created_at, sim in rows:
age_days = (now – created_at).total_seconds() / 86400
recency = 0.5 ** (age_days / half_life_days) # exponential decay
score = 0.7 * sim + 0.2 * recency + 0.1 * importance
if sim >= self.score_floor:
scored.append((score, content))

scored.sort(key=lambda x: x[0], reverse=True)

# Bound hydration by token budget, not by count.
hydrated, used = [], 0
for _, content in scored[:k]:
cost = len(content) // 4 # rough token estimate
if used + cost > max_tokens:
break
hydrated.append(content)
used += cost
return hydrated
“`

The DDL that makes this safe is equally short:

“`sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE agent_memories (
id BIGSERIAL PRIMARY KEY,
tenant_id UUID NOT NULL,
user_id UUID NOT NULL,
kind TEXT NOT NULL,
content TEXT NOT NULL,
embedding vector(1536) NOT NULL,
importance REAL NOT NULL DEFAULT 0.5,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

— RLS: the database enforces the tenant boundary.
ALTER TABLE agent_memories ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON agent_memories
USING (tenant_id = current_setting(‘app.current_tenant’)::uuid);

— Approximate-nearest-neighbor index for fast recall.
CREATE INDEX ON agent_memories
USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON agent_memories (tenant_id, user_id, created_at DESC);
“`

Three notes on production hardening. First, `SET set_config(…)` must run on *every* connection in a pool — wrap it in a connection factory, not at the call site. Second, the `k * 4` over-fetch before re-ranking exists because the similarity-only ordering is not the final ordering; the blended score needs headroom. Third, tune HNSW’s `m` and `ef_construction` against your recall target — default parameters are a starting point, not a guarantee.

—

## 6. ROI: Why Durable Memory Pays for Itself

Durable semantic memory is often framed as a capability feature. In production it is a **cost-reduction and reliability program**, and the numbers are not subtle.

**Token cost falls through the roof being removed.** An agent with durable memory does not need to carry a full session transcript forward, because it can retrieve the three facts that matter instead of re-reading the ninety turns that produced them. A 50-turn session that would have shipped 40K tokens of history per request now ships a 1.5K-token memory block. Across thousands of sessions, that is not a marginal saving; it is an order-of-magnitude change in cost per resolved task. And because the memory block is bounded by token budget, the saving is *structural* — it does not degrade as sessions get longer, which is precisely where transcript-stuffing collapses.

**Context stays dense, so reasoning improves.** Pruning the window to relevant memories reduces the lost-in-the-middle dilution that degrades answer quality as context grows. Teams consistently find that memory-augmented agents answer multi-session questions *more* accurately than long-context baselines, even at a fraction of the token spend.

**Task completion rates rise because re-work disappears.** The expensive part of a multi-day agent task is not the compute — it is the human time consumed by an agent that has forgotten the state of play: re-asking for credentials, re-running rejected approaches, re-discovering environment quirks. Procedural memory eliminates this class of failure outright. In our builds, the metric that moves first is not latency or cost but **first-pass task completion**, and it is the one that matters most to the business, because a task that completes in one pass costs a fraction of one that needs two.

**Humans stop being the memory.** The hidden cost of amnesiac agents is that operators become the persistence layer — copying context between sessions, re-explaining the customer, babysitting the handoff. Durable memory moves that burden back into the system, which is where it belongs.

The economics compound in the same direction as the technology: memory is written once and retrieved cheaply forever, while transcript-stuffing pays full token price on every single turn. Over a long-horizon workload, that asymmetry is decisive.

—

## Conclusion: Build the Memory Layer Before It Bites You

Context-window amnesia is not a model limitation you can wait out. Bigger windows delay the problem and inflate the bill; they do not give an agent persistence, scope, or the ability to forget selectively. The durable solution is architectural: a tiered memory system, semantic retrieval that hydrates the context window on demand, and hard tenant isolation enforced in the database rather than in a prompt.

If you are standing up long-horizon agents against real systems — Oracle, MS SQL, Postgres, Snowflake, your CRM — and you need memory that is fast, private, and auditable, the OnlineCloud team builds exactly this layer: vector-backed memory, MCP-native tool surfaces, and the eval harness that proves it works before it touches a customer. [Talk to the OnlineCloud engineers](https://www.onlinecloud.dev/contact) about mapping your first memory-backed vertical slice — scoped in a discovery week, built by senior engineers only, with no offshore handoff.