The Assistant Answering This Site

A consultancy that sells AI systems cannot put a chatbot on its own site that makes things up. That is the whole brief, and it rules out the easy version of this project: wire a model up to a system prompt and hope. The assistant needed to answer only from real Munir Labs material, refuse cleanly when it did not know something, and never let a plausible-sounding guess slip through, because a single invented client claim or a softened "yes" on a question it should have declined would cost more credibility than the assistant is worth.

The corpus behind it is small on purpose: written site content, service scoping, engagement terms, FAQ, and project write-ups, on the order of 15,000 words, deliberately kept under a single model's context window. That constraint turns "build a RAG pipeline" into a sharper, more honest question: is retrieval actually earning its cost at this size, or would the same money be better spent just sending the model everything and skipping the infrastructure. The project exists to answer that with numbers, not assume it.

Corpus and exclusions

Ingestion came with hard, non-negotiable exclusions baked in at the corpus level rather than left to the prompt to police: no client counts or revenue figures, no fabricated statistics, specific named entities that never appear regardless of how the question is phrased, and a founder's incomplete university studies that must never be described as a finished degree. A retriever cannot read a positioning decision, so content that would let the assistant offer a service Munir Labs deliberately does not sell (website builds, chief among them) was left out of the corpus entirely rather than trusted to a refusal at answer time.

Retrieval and evaluation

Retrieval and evaluation were built from definitions, not a library: hit rate and MRR implemented by hand against a hand-written, provenance-tagged question bank, so the metric could be trusted before it was used to judge anything. That harness caught a real, specific weakness in the TF-IDF baseline before embeddings ever entered the picture: it scored 0.949 hit@5 on questions phrased like the source FAQ and 0.360 on paraphrased ones, the exact gap you would predict from a keyword-matching method and would otherwise only suspect. Swapping in embeddings (bge-small-en-v1.5, run through fastembed's ONNX runtime rather than PyTorch, since the build machine has no usable GPU) closed a real, measured slice of that gap: paraphrased-question accuracy went from 0.360 to 0.520, a 44% relative improvement, with zero questions regressing.

The head-to-head benchmark

Then came the question the whole project was set up to answer. A head-to-head benchmark, full corpus in the prompt versus retrieval's top 5 chunks, on the same 24 held-back test questions, judged by a second model rather than by eye. The raw numbers favored full context, 0.625 to 0.375 accuracy, and reporting that on its own would have been misleading and was not shipped as the headline. Digging into every miss showed something more useful: neither approach ever fabricated a fact across all 48 answers, every miss was a decline rather than a wrong answer, and most of both conditions' misses were on questions with no clean answer in the corpus to begin with, where declining is correct behavior, not failure. The real, comparable picture was 3 genuine misses out of 24 for full context against roughly 1 for retrieval, and every genuine full-context miss was a case where the fact was present in the prompt and the model still lost it, a real "lost in the middle" effect that a 22,000-token context window makes possible and retrieval's narrow, focused context structurally cannot. Retrieval also sends roughly 42 times fewer input tokens per query, a gap that only widens as the corpus grows, since retrieval's cost stays flat while full-context's scales with corpus size.

Guardrails

Guardrails came last and were tested adversarially, not iterated into shape: a focused suite covering every hard rule, plus a prompt-injection attempt and a plain control question, passed 12 for 12 with zero prompt rewrites needed. The harder cases show judgment, not just refusal: asked whether the founder holds a degree from a specific university, it states the honest, specific fact (studies, not a completed degree) rather than deflecting; asked for pricing, it declines to invent a number but still shares the real, published payment structure rather than stonewalling. The assistant is also multi-turn and can take a lead: name, contact detail, and a message, written to a Google Sheet with an email notification through a lightweight Apps Script webhook, with its own real bug caught in testing, an early version confirmed a lead was saved before checking whether the write had actually succeeded, fixed before it ever ran with a real visitor.

Deployment

It runs as a FastAPI service on fly.io, generation on Gemini for now with a second, parallel backend already built for Claude, a deliberate one-import swap once there is a working system to point it at. Shipping it live surfaced its own real problems: an out-of-memory crash loop traced to the embedding runtime's actual memory footprint rather than assumed away, fixed by moving to a larger instance tier once smaller ones were confirmed insufficient; and a chat widget built to match this site's exact visual language, hover behavior included, with its own typing-animation reveal and a guarded clear-conversation control that closes a real race condition between a visitor clearing the chat and an in-flight reply landing a moment later.

Building the measurement before trusting the result, and reporting the corrected number over the flattering one, is the discipline here: the raw full-context-versus-retrieval score said one thing, and the honest number, after checking why, said another. It also proves the work ships past the demo. A chatbot that answers correctly in a notebook is a claim; the same system live on a real site, rate-limited, guarded against a specific hard list of things it must never say, and caught mid-bug on its own memory footprint before anyone else saw it fail, is a working product.