Letta is the long-term memory layer for Open-LLM-VTuber. It runs as a stateful agent server: every interaction passes through it, and it owns the LLM connection, the conversation history, and the agent’s evolving memory.
The Open-LLM-VTuber backend connects to it as a thin client. Instead of sending raw prompts to Ollama, the backend sends a user message to Letta; Letta assembles the full context (system prompt + memory + recent conversation), calls the LLM, streams the reply back, and persists everything to its own PostgreSQL database for next time.
localhost:8283gemma4:e4b via Ollama at
http://10.100.0.2:11434/v1, or any model served through
OpenRouternomic-embed-text via Ollama (auto-detected
once pulled)~/.letta/.persist/pgdata — a
PostgreSQL volume holding agents, memory blocks, and conversation
history. Restarting the container preserves all of it.Before any commands make sense, the terminology:
agent-7f4a479f-….persona
(description of the agent’s character, speaking style, language rules)
and human (what the agent knows about the user). Both are
included in every LLM call. The agent can rewrite them mid-conversation
via tool calls — that’s how it “remembers” the user across
sessions.archival_memory_insert tool and searches with
archival_memory_search when it needs to recall something
old. Requires an embedding model.provider/model. Examples:
ollama/gemma4:e4b,
ollama/nomic-embed-text:latest. Letta discovers these by
querying each provider’s /v1/models endpoint.podman run --rm \
-v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
-p 8283:8283 \
-e OLLAMA_BASE_URL=http://10.100.0.2:11434/v1 \
-e OPENROUTER_API_KEY="sk-or-v1-..." \
-e EXA_API_KEY=$EXA_API_KEY \
letta/letta:latestOLLAMA_BASE_URL is what lets Letta discover Ollama
models (including nomic-embed-text for embeddings).
OPENROUTER_API_KEY (optional) lets Letta discover any model
available through OpenRouter — models appear under the
openrouter/ prefix. EXA_API_KEY (optional)
enables the builtin web_search and
fetch_webpage tools — reference Giving the agent tools.
~/.letta should be declared persistent in
impermance.nix so the database survives reboots.
Archival memory needs an embedding model. Pull one into Ollama before creating the agent:
ollama pull nomic-embed-textLetta picks it up automatically and exposes it as the handle
ollama/nomic-embed-text:latest. If skipped, agent creation
still works but archival memory won’t.
There is no graphical UI in the self-hosted build — the cloud-only ADE is unavailable. Agents are created through the REST API. Run this once:
import httpx
import sys
BASE = "http://127.0.0.1:8283"
# Edit these before running. All three are free-form text.
NAME = "<name>" # any identifier, used to find the agent later
PERSONA = "PLACEHOLDER" # describes who the agent is (can update later)
HUMAN = "PLACEHOLDER" # what the agent knows about the user (can update later)
MODEL = "<model>" # the ollama model to use, e.g. 'ollama/gemma4:e4b'
client = httpx.Client(base_url=BASE, timeout=30)
resp = client.post("/v1/agents/", json={
"name": NAME,
"agent_type": "letta_v1_agent",
"model": MODEL,
"memory_blocks": [
{"label": "persona", "value": PERSONA, "limit": 2000},
{"label": "human", "value": HUMAN, "limit": 2000},
],
})
if resp.status_code not in (200, 201):
print(f"Failed ({resp.status_code}): {resp.text}")
sys.exit(1)
agent = resp.json()
print(agent["id"])The printed ID is what needs to be pasted into conf.yaml
below. To edit PERSONA/HUMAN later with
multi-line text, reference Editing
memory blocks.
letta_v1_agent is the current agent type. The older
memgpt_agent requires the LLM to respond exclusively via
tool calls, which gemma4:e4b doesn’t reliably do — symptom
is llm_api_error: No tool calls found in response.
conf.yamlIn the backend’s conf.yaml:
conversation_agent_choice: 'letta_agent'
agent_settings:
letta_agent:
host: '127.0.0.1'
port: 8283
id: 'agent-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
faster_first_response: true
segment_method: 'pysbd'Use 127.0.0.1 rather than localhost.
localhost resolves to IPv6 first, and the SSE stream the
backend opens against Letta over IPv6 + uvloop closes immediately on
first read.
If the agent ID was lost or never noted, list all agents on the server:
curl -sL http://127.0.0.1:8283/v1/agents/ | jq '.[] | {id, name}'The rest of the commands below use two shell variables — set them once per session:
HOST=http://127.0.0.1:8283
AGENT=agent-xxxxxxxx-...Inspect an agent — LLM config, embedding config, attached tools, memory blocks. Useful for confirming the agent is wired up correctly
curl -sL $HOST/v1/agents/$AGENT | jq '{name, llm_config, embedding_config, memory}'See the actual prompt the LLM will receive — the compiled context window, with system prompt, memory blocks, and recent messages all stitched together
curl -sL $HOST/v1/agents/$AGENT/context | jqList recent conversation messages
curl -sL "$HOST/v1/agents/$AGENT/messages?limit=20" | jqEdit a single message — fix one bad turn without nuking the history, e.g. a garbled ASR transcription. Individual messages cannot be deleted, only rewritten. First grab the message ID
curl -sL "$HOST/v1/agents/$AGENT/messages?limit=10" | \
jq '.[] | {id, message_type, content}'Then rewrite it:
curl -s -X PATCH "$HOST/v1/agents/$AGENT/messages/message-xxxx-..." \
-H "Content-Type: application/json" \
-d '{"message_type": "user_message", "content": "what was actually said"}'The same endpoint rewrites the agent’s replies with
"message_type": "assistant_message" — worth doing when a
reply was derived from the garbled input, since the LLM imitates its own
prior turns. The edit affects all future turns (while the message
remains in the context window) but regenerates nothing: an
already-spoken reply stays as-is unless rewritten too. For multi-line
content, use the same jq -Rs heredoc trick as in Editing memory blocks, wrapping stdin
as {message_type: "user_message", content: .}.
Clear the in-context conversation but keep the memory blocks — the LLM will respond as if no prior turns happened. The endpoint requires a JSON body, even if empty
curl -s -X PATCH $HOST/v1/agents/$AGENT/reset-messages \
-H "Content-Type: application/json" -d '{}'This only clears the agent’s context window
(agent.message_ids drops to just the system message). The
full message archive lives separately in Postgres and is still returned
by the messages endpoint above — Letta keeps it so the
agent’s conversation_search tool can still recall past
turns. To truly delete every message including the archive, recreate the agent.
Check the current LLM
curl -sL $HOST/v1/agents/$AGENT | jq -r '.model'Swap the LLM — change which model the agent uses
without recreating it. The handle has to be one of those listed under
/v1/models/
curl -s -X PATCH $HOST/v1/agents/$AGENT \
-H "Content-Type: application/json" \
-d '{"model":"ollama/qwen2.5:3b"}'Memory blocks and conversation history are preserved across the swap; only the LLM backing the agent changes.
| Model handle | Description | Cost |
|---|---|---|
ollama/gemma4:e4b |
Local 9B model on GPU, fast with zero network latency | Free (self-hosted) |
openrouter/openai/gpt-4o-mini |
Best quality-to-price. Fast, vision-capable, reliable tool calling | $0.15/M in, $0.60/M out |
openrouter/google/gemini-2.5-flash-lite |
Strong multilingual (Spanish/Japanese/German), 1M context, vision | $0.10/M in, $0.40/M out |
openrouter/google/gemma-4-26b-a4b-it |
Same model family as local, cheap, vision-capable | $0.06/M in, $0.33/M out |
Rename the agent — update the display name without recreating it
curl -s -X PATCH $HOST/v1/agents/$AGENT \
-H "Content-Type: application/json" \
-d '{"name": "new-name"}'See what LLMs and embedding models Letta discovered
curl -sL $HOST/v1/models/ | jq '.[].handle'
curl -sL $HOST/v1/models/embedding | jq '.[].handle'Letta caches each provider’s model catalog at startup, so
ollama pull <new> won’t show up until Letta is
restarted. The built-in ollama provider has no refresh
endpoint exposed — restarting the container is the supported path.
Read a single block:
curl -sL $HOST/v1/agents/$AGENT/core-memory/blocks/persona | jq .value -r
curl -sL $HOST/v1/agents/$AGENT/core-memory/blocks/human | jq .value -rUpdate a block. Multi-line text breaks single-line JSON, so
jq encodes it from a heredoc:
cat <<'EOF' | jq -Rs '{value: .}' | curl -s -X PATCH \
"$HOST/v1/agents/$AGENT/core-memory/blocks/persona" \
-H "Content-Type: application/json" -d @-
First line of the persona.
Second line, with "quotes" and ñ accents — both fine.
Third line.
EOFjq -Rs reads all of stdin as a single raw string and
wraps it in {"value": ...} — newlines, quotes, and unicode
are escaped correctly. The <<'EOF' form (quoted
heredoc) prevents the shell from doing any variable expansion inside the
text.
Replace persona with human in the URL to
update the other block. Changes take effect on the agent’s next
message.
When the context window fills past ~90%, Letta evicts the oldest
messages and replaces them with a summary written by a dedicated model —
configured per agent in compaction_settings, independent of
the chat model:
curl -sL $HOST/v1/agents/$AGENT | jq '.compaction_settings.model'curl -s -X PATCH $HOST/v1/agents/$AGENT \
-H "Content-Type: application/json" \
-d '{"compaction_settings": {"model": "<model-handle>"}}'The summarizer must be a model that reliably answers: if it fails
(e.g. a rate-limited :free OpenRouter handle), the failure
takes the whole turn down with it — replies end in retry loops and
LLMRateLimitError in the container logs once the threshold
is reached. The agent’s own model handle is a safe default; compaction
only runs when the context fills. compaction_settings also
accepts a custom prompt controlling what the summaries
preserve.
The agent reaches into archival memory automatically — when it learns
something worth remembering long-term, it calls
archival_memory_insert; when it wants to recall something,
it calls archival_memory_search. Manual access is rarely
needed but available:
# List all stored passages
curl -sL $HOST/v1/agents/$AGENT/archival-memory | jq
# Insert one manually
curl -s $HOST/v1/agents/$AGENT/archival-memory \
-H "Content-Type: application/json" \
-d '{"text":"User mentioned they like atmospheric phonk."}'
# Search by semantic similarity
curl -sL "$HOST/v1/agents/$AGENT/archival-memory/search?query=music" | jq
# Delete a passage
curl -s -X DELETE $HOST/v1/agents/$AGENT/archival-memory/<memory-id>The agent’s toolbox is server-side state on Letta, not part of the
Open-LLM-VTuber config. The letta_agent backend in
Open-LLM-VTuber is a thin HTTP proxy — the use_mcpp /
mcp_enabled_servers keys that work for
basic_memory_agent are not honored here, because the
conversation is delegated to the remote agent and that agent is what
owns its toolbox.
Tools are added in two steps:
web_search, fetch_webpage,
run_code, …) this is already done; they ship with the
image. For MCP tools, reference below.PATCH /v1/agents/{agent}/tools/attach/{tool_id}.# All tools registered on the server (built-ins, MCP, custom)
curl -sL "$HOST/v1/tools/?limit=200" | jq '.[] | {name, tool_type}'
# Tools currently attached to the agent
curl -sL "$HOST/v1/agents/$AGENT" | jq '.tools[].name'Letta ships builtin tools that wrap Exa:
web_search (results + snippets) and
fetch_webpage (one URL → markdown). Free tier covers a few
hundred queries a month.
Sign up at exa.ai, copy the API key.
Restart the Letta container with -e EXA_API_KEY=…
(reference Starting the
Server).
Attach the tools:
for tool in web_search fetch_webpage; do
TID=$(curl -sL "$HOST/v1/tools/?limit=200" \
| jq -r --arg n "$tool" '.[] | select(.name==$n) | .id')
curl -s -X PATCH "$HOST/v1/agents/$AGENT/tools/attach/$TID" \
-H "Content-Type: application/json" >/dev/null
echo "attached $tool"
doneThe agent’s system prompt is automatically rebuilt to advertise the new tools; the LLM can call them on the next turn.
Letta’s structured prompt (base instructions + memory blocks + tool
definitions) is ~5× heavier than basic_memory_agent’s. A
typical tool-using turn sends ~20K tokens to the LLM. Ollama’s default
num_ctx is 4096 tokens and it silently truncates
anything longer — no warning, no error. When the user’s question falls
off the end of the truncated window, the model receives
[system, tools, tool_result] with no question to answer and
emits a generic “how can I help?” greeting, ignoring the perfectly good
tool output.
Symptom: the tool fires, Letta logs show successful Exa call, but the assistant reply has nothing to do with the question.
Fix lives on the Ollama service, not anywhere in Letta. In
modules/features/ollama.nix:
services.ollama.environmentVariables = {
OLLAMA_CONTEXT_LENGTH = "32768";
# … other env vars
};Applies server-wide to every loaded model. Costs a few hundred extra
MB of KV cache per model, negligible on any GPU sized for these models
in the first place. The OpenAI-compatible
/v1/chat/completions endpoint Letta uses doesn’t expose
num_ctx per-request, so this env var is the cleanest
lever.
run_codeThe builtin run_code (and
run_code_with_tools) executes generated code in a “local”
sandbox — a subprocess inside the Letta Podman container, with network
access and visibility into the container’s env vars. It can’t touch the
host filesystem, but it can hit the public internet, read its own
Postgres data, and leak any env var (including
EXA_API_KEY). The realistic risk is prompt-injection: a
webpage returned by fetch_webpage convinces the LLM to
write code that exfiltrates secrets.
If code execution is actually needed, configure the E2B or Modal
backend (ephemeral cloud sandboxes) via
POST /v1/sandbox-config/e2b/default instead of the local
default.
Letta supports MCP servers via
POST /v1/tools/mcp/servers/connect in three flavors:
stdio, sse, and streamable_http.
Stdio requires the MCP binary inside the Letta container — the default
image has no uvx or node for typical MCP
servers, so using stdio means building a custom image. SSE
/ HTTP work if the MCP server is reachable from the container as a
network service (separate daemon or mcp-proxy wrapping a
stdio server).
Once a server is registered, list its tools via
GET /v1/tools/mcp/servers/{name}/tools, then attach
individual tools to the agent the same way as the builtins. Reference https://docs.letta.com/guides/agents/tools for the
latest API.
To change the LLM model or start with a clean slate, delete and re-create:
curl -s -X DELETE $HOST/v1/agents/$AGENTAfter deletion, re-run the Python script under Create an Agent and update the new ID in
conf.yaml. Memory blocks and conversation history are
deleted along with the agent.
The full REST API is browsable at http://localhost:8283/docs. Useful for discovering endpoints not covered here — every operation the cheatsheet uses is documented there with request/response schemas. The graphical ADE shown in Letta’s marketing is cloud-only and not bundled with the self-hosted Docker/Podman image.