Skip to main content

Production config, observability, and MCP mental models

· 2 min read
Anurag Atulya

Capabilities for production deployments: hierarchical configuration, direct LiteLLM SDK access, broader database options, request tracing, and MCP mental model management.

Hierarchical configuration​

Override operational settings per memory bank: retain chunk size, extraction mode, custom instructions, and more.

curl -X PATCH http://localhost:8888/v1/default/banks/my-bank/config \
  -H "Content-Type: application/json" \
  -d '{
    "updates": {
      "retain_chunk_size": 1000,
      "retain_extraction_mode": "custom",
      "retain_custom_instructions": "Keep specific details about incidents, ignore complaints."
    }
  }'

Cascade: environment defaults → tenant overrides → bank settings. Bank config API is on by default; disable with ATULYA_API_ENABLE_BANK_CONFIG_API=false. See the configuration guide.

LiteLLM SDK integration​

Use LiteLLM for embeddings and reranking without a proxy server:

# Proxy mode (still supported)
export ATULYA_API_EMBEDDINGS_PROVIDER=litellm
export ATULYA_API_EMBEDDINGS_LITELLM_API_BASE=http://localhost:4000

# Direct SDK
export ATULYA_API_EMBEDDINGS_PROVIDER=litellm-sdk
export ATULYA_API_EMBEDDINGS_LITELLM_SDK_API_KEY=your-api-key
export ATULYA_API_EMBEDDINGS_LITELLM_SDK_MODEL=cohere/embed-english-v3.0

Same pattern for reranking: ATULYA_API_RERANKER_PROVIDER=litellm-sdk. Proxy when you need centralized rate limits; SDK for simpler topologies.

Expanded database support​

  • TimescaleDB pg_textsearch for time-series full-text workloads (compose example)
  • vchord and pgvector options (compose example)
  • Better compatibility with external Postgres configurations

OpenTelemetry tracing​

Request-level traces across retain, recall, reflect, and async jobs. Token usage reflects actual LLM consumption.

Local stack: ./scripts/dev/start-monitoring.sh (Grafana LGTM: Loki, Grafana, Tempo, Mimir).

MCP mental models​

MCP clients can create, read, update, and delete mental models, not only query them.

Documentation skill​

The docs skill helps documentation-aware assistants access and reason over your doc corpus.

Reverse proxy support​

Configure base path when Atulya sits behind nginx or another reverse proxy. Example: nginx compose.

Also worth knowing​

  • Helm: split TEI for embeddings/reranking, PodDisruptionBudgets, per-component affinity, GKE port fixes
  • Slim Docker image includes tiktoken to avoid runtime download failures

Changelog · GitHub Discussions