How It Works

Baymax Chatbot

Responses are generated using a chain agentic workflow with self-critique mechanism, orchestrated using LangGraph.

Model Eval →

Caching

A user prompt first checks two cache layers, so repeated or reworded questions return instantly without reaching the LLM and spending tokens.

1. exact cache

Identical questions are served instantly from a hashed lookup.

Redis, 24h

2. semantic cache

Reworded questions with the same meaning reuse a stored answer, matched by similarity above a threshold (e.g., "what's my latest job" ≈ "where does Jonas work now"). Embedded with bge-large-en-v1.5. Entries older than 7 days are treated as a miss (7d soft TTL).

Vector, 7d

Orchestration Pipeline

1. retrieve

Gathers relevant knowledge for the question. The full knowledge base is passed while it is small; semantic search activates as it grows.

full context

2. generate

Produces an answer based on the retrieved context.

LLM: llama-3.3-70b-versatile

3. check

Reviews own answer for completeness and accuracy.

LLM: llama-3.3-70b-versatile

4. revise

If the check finds problems, rewrites the answer.

LLM: llama-3.3-70b-versatile

Backend

Orchestration LangGraph

Caching Upstash Redis (exact, 24h) and Upstash Vector (semantic, 7d, bge-large-en-v1.5)

Hosting Vercel serverless

Guardrails

Rate limiting 20 requests per minute per IP

Greetings "hi", "thank you", "bye" return a canned reply, no LLM call

Scope Only answers questions about Jonas; off-topic gets a polite refusal

Input limit Questions over 300 characters are rejected

Failure mode Friendly message instead of hanging when the model is rate-limited