Letta

Overview

Letta is the long-term memory layer for Open-LLM-VTuber. It runs as a stateful agent server: every interaction passes through it, and it owns the LLM connection, the conversation history, and the agent’s evolving memory.

The Open-LLM-VTuber backend connects to it as a thin client. Instead of sending raw prompts to Ollama, the backend sends a user message to Letta; Letta assembles the full context (system prompt + memory + recent conversation), calls the LLM, streams the reply back, and persists everything to its own PostgreSQL database for next time.

Vocabulary

Before any commands make sense, the terminology:

Starting the Server

podman run --rm \
  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
  -p 8283:8283 \
  -e OLLAMA_BASE_URL=http://10.100.0.2:11434/v1 \
  -e OPENROUTER_API_KEY="sk-or-v1-..." \
  -e EXA_API_KEY=$EXA_API_KEY \
  letta/letta:latest

OLLAMA_BASE_URL is what lets Letta discover Ollama models (including nomic-embed-text for embeddings). OPENROUTER_API_KEY (optional) lets Letta discover any model available through OpenRouter — models appear under the openrouter/ prefix. EXA_API_KEY (optional) enables the builtin web_search and fetch_webpage tools — reference Giving the agent tools.

~/.letta should be declared persistent in impermance.nix so the database survives reboots.

Pull an Embedding Model

Archival memory needs an embedding model. Pull one into Ollama before creating the agent:

ollama pull nomic-embed-text

Letta picks it up automatically and exposes it as the handle ollama/nomic-embed-text:latest. If skipped, agent creation still works but archival memory won’t.

Create an Agent

There is no graphical UI in the self-hosted build — the cloud-only ADE is unavailable. Agents are created through the REST API. Run this once:

import httpx
import sys

BASE = "http://127.0.0.1:8283"

# Edit these before running. All three are free-form text.
NAME = "<name>"          # any identifier, used to find the agent later
PERSONA = "PLACEHOLDER"  # describes who the agent is (can update later)
HUMAN = "PLACEHOLDER"    # what the agent knows about the user (can update later)
MODEL = "<model>"        # the ollama model to use, e.g. 'ollama/gemma4:e4b'

client = httpx.Client(base_url=BASE, timeout=30)

resp = client.post("/v1/agents/", json={
    "name": NAME,
    "agent_type": "letta_v1_agent",
    "model": MODEL,
    "memory_blocks": [
        {"label": "persona", "value": PERSONA, "limit": 2000},
        {"label": "human", "value": HUMAN, "limit": 2000},
    ],
})

if resp.status_code not in (200, 201):
    print(f"Failed ({resp.status_code}): {resp.text}")
    sys.exit(1)

agent = resp.json()
print(agent["id"])

The printed ID is what needs to be pasted into conf.yaml below. To edit PERSONA/HUMAN later with multi-line text, reference Editing memory blocks.

letta_v1_agent is the current agent type. The older memgpt_agent requires the LLM to respond exclusively via tool calls, which gemma4:e4b doesn’t reliably do — symptom is llm_api_error: No tool calls found in response.

Wire Up conf.yaml

In the backend’s conf.yaml:

conversation_agent_choice: 'letta_agent'
agent_settings:
  letta_agent:
    host: '127.0.0.1'
    port: 8283
    id: 'agent-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
    faster_first_response: true
    segment_method: 'pysbd'

Use 127.0.0.1 rather than localhost. localhost resolves to IPv6 first, and the SSE stream the backend opens against Letta over IPv6 + uvloop closes immediately on first read.

REST Cheatsheet

If the agent ID was lost or never noted, list all agents on the server:

curl -sL http://127.0.0.1:8283/v1/agents/ | jq '.[] | {id, name}'

The rest of the commands below use two shell variables — set them once per session:

HOST=http://127.0.0.1:8283
AGENT=agent-xxxxxxxx-...

Inspect an agent — LLM config, embedding config, attached tools, memory blocks. Useful for confirming the agent is wired up correctly

curl -sL $HOST/v1/agents/$AGENT | jq '{name, llm_config, embedding_config, memory}'

See the actual prompt the LLM will receive — the compiled context window, with system prompt, memory blocks, and recent messages all stitched together

curl -sL $HOST/v1/agents/$AGENT/context | jq

List recent conversation messages

curl -sL "$HOST/v1/agents/$AGENT/messages?limit=20" | jq

Edit a single message — fix one bad turn without nuking the history, e.g. a garbled ASR transcription. Individual messages cannot be deleted, only rewritten. First grab the message ID

curl -sL "$HOST/v1/agents/$AGENT/messages?limit=10" | \
  jq '.[] | {id, message_type, content}'

Then rewrite it:

curl -s -X PATCH "$HOST/v1/agents/$AGENT/messages/message-xxxx-..." \
  -H "Content-Type: application/json" \
  -d '{"message_type": "user_message", "content": "what was actually said"}'

The same endpoint rewrites the agent’s replies with "message_type": "assistant_message" — worth doing when a reply was derived from the garbled input, since the LLM imitates its own prior turns. The edit affects all future turns (while the message remains in the context window) but regenerates nothing: an already-spoken reply stays as-is unless rewritten too. For multi-line content, use the same jq -Rs heredoc trick as in Editing memory blocks, wrapping stdin as {message_type: "user_message", content: .}.

Clear the in-context conversation but keep the memory blocks — the LLM will respond as if no prior turns happened. The endpoint requires a JSON body, even if empty

curl -s -X PATCH $HOST/v1/agents/$AGENT/reset-messages \
  -H "Content-Type: application/json" -d '{}'

This only clears the agent’s context window (agent.message_ids drops to just the system message). The full message archive lives separately in Postgres and is still returned by the messages endpoint above — Letta keeps it so the agent’s conversation_search tool can still recall past turns. To truly delete every message including the archive, recreate the agent.

Check the current LLM

curl -sL $HOST/v1/agents/$AGENT | jq -r '.model'

Swap the LLM — change which model the agent uses without recreating it. The handle has to be one of those listed under /v1/models/

curl -s -X PATCH $HOST/v1/agents/$AGENT \
  -H "Content-Type: application/json" \
  -d '{"model":"ollama/qwen2.5:3b"}'

Memory blocks and conversation history are preserved across the swap; only the LLM backing the agent changes.

Other model handles
Model handle Description Cost
ollama/gemma4:e4b Local 9B model on GPU, fast with zero network latency Free (self-hosted)
openrouter/openai/gpt-4o-mini Best quality-to-price. Fast, vision-capable, reliable tool calling $0.15/M in, $0.60/M out
openrouter/google/gemini-2.5-flash-lite Strong multilingual (Spanish/Japanese/German), 1M context, vision $0.10/M in, $0.40/M out
openrouter/google/gemma-4-26b-a4b-it Same model family as local, cheap, vision-capable $0.06/M in, $0.33/M out

Rename the agent — update the display name without recreating it

curl -s -X PATCH $HOST/v1/agents/$AGENT \
  -H "Content-Type: application/json" \
  -d '{"name": "new-name"}'

See what LLMs and embedding models Letta discovered

curl -sL $HOST/v1/models/          | jq '.[].handle'
curl -sL $HOST/v1/models/embedding | jq '.[].handle'

Letta caches each provider’s model catalog at startup, so ollama pull <new> won’t show up until Letta is restarted. The built-in ollama provider has no refresh endpoint exposed — restarting the container is the supported path.

Editing Memory Blocks

Read a single block:

curl -sL $HOST/v1/agents/$AGENT/core-memory/blocks/persona | jq .value -r
curl -sL $HOST/v1/agents/$AGENT/core-memory/blocks/human   | jq .value -r

Update a block. Multi-line text breaks single-line JSON, so jq encodes it from a heredoc:

cat <<'EOF' | jq -Rs '{value: .}' | curl -s -X PATCH \
  "$HOST/v1/agents/$AGENT/core-memory/blocks/persona" \
  -H "Content-Type: application/json" -d @-
First line of the persona.
Second line, with "quotes" and ñ accents — both fine.
Third line.
EOF

jq -Rs reads all of stdin as a single raw string and wraps it in {"value": ...} — newlines, quotes, and unicode are escaped correctly. The <<'EOF' form (quoted heredoc) prevents the shell from doing any variable expansion inside the text.

Replace persona with human in the URL to update the other block. Changes take effect on the agent’s next message.

Compaction

When the context window fills past ~90%, Letta evicts the oldest messages and replaces them with a summary written by a dedicated model — configured per agent in compaction_settings, independent of the chat model:

curl -sL $HOST/v1/agents/$AGENT | jq '.compaction_settings.model'
curl -s -X PATCH $HOST/v1/agents/$AGENT \
  -H "Content-Type: application/json" \
  -d '{"compaction_settings": {"model": "<model-handle>"}}'

The summarizer must be a model that reliably answers: if it fails (e.g. a rate-limited :free OpenRouter handle), the failure takes the whole turn down with it — replies end in retry loops and LLMRateLimitError in the container logs once the threshold is reached. The agent’s own model handle is a safe default; compaction only runs when the context fills. compaction_settings also accepts a custom prompt controlling what the summaries preserve.

Archival Memory

The agent reaches into archival memory automatically — when it learns something worth remembering long-term, it calls archival_memory_insert; when it wants to recall something, it calls archival_memory_search. Manual access is rarely needed but available:

# List all stored passages
curl -sL $HOST/v1/agents/$AGENT/archival-memory | jq

# Insert one manually
curl -s $HOST/v1/agents/$AGENT/archival-memory \
  -H "Content-Type: application/json" \
  -d '{"text":"User mentioned they like atmospheric phonk."}'

# Search by semantic similarity
curl -sL "$HOST/v1/agents/$AGENT/archival-memory/search?query=music" | jq

# Delete a passage
curl -s -X DELETE $HOST/v1/agents/$AGENT/archival-memory/<memory-id>

Giving the Agent Tools

The agent’s toolbox is server-side state on Letta, not part of the Open-LLM-VTuber config. The letta_agent backend in Open-LLM-VTuber is a thin HTTP proxy — the use_mcpp / mcp_enabled_servers keys that work for basic_memory_agent are not honored here, because the conversation is delegated to the remote agent and that agent is what owns its toolbox.

Tools are added in two steps:

  1. Register the tool with the Letta server — for the builtin tools (web_search, fetch_webpage, run_code, …) this is already done; they ship with the image. For MCP tools, reference below.
  2. Attach the tool to the agent via PATCH /v1/agents/{agent}/tools/attach/{tool_id}.

Browsing what’s available

# All tools registered on the server (built-ins, MCP, custom)
curl -sL "$HOST/v1/tools/?limit=200" | jq '.[] | {name, tool_type}'

# Tools currently attached to the agent
curl -sL "$HOST/v1/agents/$AGENT" | jq '.tools[].name'

Letta ships builtin tools that wrap Exa: web_search (results + snippets) and fetch_webpage (one URL → markdown). Free tier covers a few hundred queries a month.

  1. Sign up at exa.ai, copy the API key.

  2. Restart the Letta container with -e EXA_API_KEY=… (reference Starting the Server).

  3. Attach the tools:

    for tool in web_search fetch_webpage; do
      TID=$(curl -sL "$HOST/v1/tools/?limit=200" \
            | jq -r --arg n "$tool" '.[] | select(.name==$n) | .id')
      curl -s -X PATCH "$HOST/v1/agents/$AGENT/tools/attach/$TID" \
           -H "Content-Type: application/json" >/dev/null
      echo "attached $tool"
    done

The agent’s system prompt is automatically rebuilt to advertise the new tools; the LLM can call them on the next turn.

Gotcha — raise Ollama’s context window first

Letta’s structured prompt (base instructions + memory blocks + tool definitions) is ~5× heavier than basic_memory_agent’s. A typical tool-using turn sends ~20K tokens to the LLM. Ollama’s default num_ctx is 4096 tokens and it silently truncates anything longer — no warning, no error. When the user’s question falls off the end of the truncated window, the model receives [system, tools, tool_result] with no question to answer and emits a generic “how can I help?” greeting, ignoring the perfectly good tool output.

Symptom: the tool fires, Letta logs show successful Exa call, but the assistant reply has nothing to do with the question.

Fix lives on the Ollama service, not anywhere in Letta. In modules/features/ollama.nix:

services.ollama.environmentVariables = {
  OLLAMA_CONTEXT_LENGTH = "32768";
  # … other env vars
};

Applies server-wide to every loaded model. Costs a few hundred extra MB of KV cache per model, negligible on any GPU sized for these models in the first place. The OpenAI-compatible /v1/chat/completions endpoint Letta uses doesn’t expose num_ctx per-request, so this env var is the cleanest lever.

Avoid attaching run_code

The builtin run_code (and run_code_with_tools) executes generated code in a “local” sandbox — a subprocess inside the Letta Podman container, with network access and visibility into the container’s env vars. It can’t touch the host filesystem, but it can hit the public internet, read its own Postgres data, and leak any env var (including EXA_API_KEY). The realistic risk is prompt-injection: a webpage returned by fetch_webpage convinces the LLM to write code that exfiltrates secrets.

If code execution is actually needed, configure the E2B or Modal backend (ephemeral cloud sandboxes) via POST /v1/sandbox-config/e2b/default instead of the local default.

Native MCP (advanced)

Letta supports MCP servers via POST /v1/tools/mcp/servers/connect in three flavors: stdio, sse, and streamable_http. Stdio requires the MCP binary inside the Letta container — the default image has no uvx or node for typical MCP servers, so using stdio means building a custom image. SSE / HTTP work if the MCP server is reachable from the container as a network service (separate daemon or mcp-proxy wrapping a stdio server).

Once a server is registered, list its tools via GET /v1/tools/mcp/servers/{name}/tools, then attach individual tools to the agent the same way as the builtins. Reference https://docs.letta.com/guides/agents/tools for the latest API.

Recreate the Agent

To change the LLM model or start with a clean slate, delete and re-create:

curl -s -X DELETE $HOST/v1/agents/$AGENT

After deletion, re-run the Python script under Create an Agent and update the new ID in conf.yaml. Memory blocks and conversation history are deleted along with the agent.

Swagger UI

The full REST API is browsable at http://localhost:8283/docs. Useful for discovering endpoints not covered here — every operation the cheatsheet uses is documented there with request/response schemas. The graphical ADE shown in Letta’s marketing is cloud-only and not bundled with the self-hosted Docker/Podman image.