Skip to main content

File ingestion, batch retain, Go SDK, and DiskANN

· 2 min read
Anurag Atulya

Ingest documents without pre-extracting text, run cheaper async retain, call Atulya from Go, and index large banks with DiskANN-class vector backends.

File ingestion​

Retain PDFs, images, and Office formats (DOCX, PPTX, XLSX) directly.

from atulya_sdk import AtulyaClient

client = AtulyaClient()

with open("report.pdf", "rb") as f:
    client.retain_files("my-bank", files=[("file", ("report.pdf", f, "application/pdf"))])

REST:

curl -X POST http://localhost:8888/v1/default/banks/my-bank/files/retain \
  -F "file=@report.pdf;type=application/pdf"

Parsers: Markitdown (default) or Iris for harder layouts:

export ATULYA_API_FILE_PARSER=iris   # or markitdown

Batch API for async retain​

~50% token discount on OpenAI and Groq batch APIs when async retain is enabled:

export ATULYA_API_RETAIN_BATCH_ENABLED=true

Requires async=true on retain. Atulya submits extraction jobs, polls, and processes results. Ideal for backfills and large archives where background latency is acceptable.

Go client SDK​

Generated from OpenAPI 3.1:

import atulya "github.com/eight-atulya/atulya/atulya-clients/go"

cfg := atulya.NewConfiguration()
cfg.Servers = atulya.ServerConfigurations{{URL: "http://localhost:8888"}}
client := atulya.NewAPIClient(cfg)

retainReq := atulya.RetainRequest{
    Items: []atulya.MemoryItem{
        {Content: "The deployment succeeded at 14:32 UTC.", Tags: []string{"ops"}},
    },
}
client.MemoryAPI.RetainMemories(ctx, "my-bank").RetainRequest(retainReq).Execute()

Go SDK docs

DiskANN vector indexing​

pgvectorscale (DiskANN) for self-hosted scale:

docker compose -f docker/docker-compose/timescale/docker-compose.yml up

Azure pg_diskann on Azure Database for PostgreSQL Flexible Server.

export ATULYA_API_VECTOR_INDEX_BACKEND=diskann        # pgvectorscale
export ATULYA_API_VECTOR_INDEX_BACKEND=azure_diskann  # Azure

Drop-in for default pgvector HNSW at large recall volumes.

Also worth knowing​

  • Vercel AI SDK integration: cleaner types and tool definitions
  • Python client: async consistency and keepalive fixes under load
  • OpenClaw: safer execFile, HTTP dual mode, per-user bank isolation, reinit cooldown

Changelog