Skip to main content

Give any LiteLLM provider persistent memory

· 2 min read
Anurag Atulya

atulya-litellm intercepts LiteLLM completions: recall before the call, retain after. Works across OpenAI, Anthropic, Groq, Azure, Bedrock, Vertex, and 100+ providers with one bank.

The problem​

Every provider API call is stateless. Sliding-window chat history is not memory: old preferences and project context fall off the cliff.

Three-line setup​

import atulya_litellm

atulya_litellm.configure(atulya_api_url="http://localhost:8888")
atulya_litellm.set_defaults(bank_id="my-agent")
atulya_litellm.enable()
response = atulya_litellm.completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Help with my Python project"}],
    atulya_query="What do I know about the user's Python project?",
)

atulya_query drives retrieval. Switch models freely; the bank stays the same.

Any provider​

query = "What have I discussed with this user?"
messages = [{"role": "user", "content": "What did we discuss last time?"}]

atulya_litellm.completion(model="gpt-4o", messages=messages, atulya_query=query)
atulya_litellm.completion(model="claude-sonnet-4-20250514", messages=messages, atulya_query=query)
atulya_litellm.completion(model="groq/llama-3.1-70b-versatile", messages=messages, atulya_query=query)

Recall vs reflect​

Recall (default): numbered fact list in the prompt. Best for precise data points.

atulya_litellm.set_defaults(bank_id="my-agent", use_reflect=False)

Reflect: synthesized paragraph. Best for conversational context.

atulya_litellm.set_defaults(
    bank_id="my-agent",
    use_reflect=True,
    reflect_context="Coding assistant for Python projects.",
)

Direct APIs​

from atulya_litellm import recall, reflect, retain

memories = recall("what projects is the user on?", budget="mid")
result = reflect("user preferences?", context="Support agent.")
retain(content="Switching from Flask to FastAPI", context="Frameworks")

Async: arecall, areflect, aretain.

Per-call overrides​

atulya_litellm.completion(
    model="gpt-4o-mini",
    messages=[...],
    atulya_query="What do I know about Alice?",
    atulya_bank_id="other-agent",
    atulya_budget="high",
)

Native SDK wrappers​

from openai import OpenAI
from atulya_litellm import wrap_openai

wrapped = wrap_openai(OpenAI(), bank_id="my-agent", atulya_api_url="http://localhost:8888")
wrapped.chat.completions.create(model="gpt-4o", messages=[...])

Anthropic: wrap_anthropic with the same pattern.

Context manager​

from atulya_litellm import atulya_memory
import litellm

with atulya_memory(atulya_api_url="http://localhost:8888", bank_id="user-123"):
    litellm.completion(model="gpt-4o", messages=[...], atulya_query="greeting")

Bank mission​

atulya_litellm.configure(
    atulya_api_url="http://localhost:8888",
    mission="Route support issues to billing, technical, or sales. Track resolutions.",
    bank_name="Support Router",
)

Next steps​