Evals · run on every deploy
The assistant, measured in public.
Every build runs a golden set of 41 on-topic questions and 13 off-topic ones against the exact BM25 index the Worker serves. If retrieval regresses, it shows up here first.
corpus de379456be92 · 182 chunks · score floor 2.5 · coverage floor 0.65 · ran
- hit@195.1%right page ranked first
- hit@3100%right page in the top three
- mrr0.976mean reciprocal rank
- refusal100%off-topic questions refused
- answer rate100%on-topic questions answered
- p50 search0.03msBM25 over 182 chunks
Golden set · 41/41 passing
On-topic questions
A case passes when an expected page ranks in the top three and the refusal gate lets the question through.
| Question | Top page | Rank | Score | Coverage | Result |
|---|---|---|---|---|---|
| What did he build for State Street? | work/state-street-vm-factory-image-bakery | 2 | 8.22 | 1.00 | pass |
| Disaster recovery with Azure Site Recovery | work/state-street-azure-landing-zone-dr | 1 | 12.02 | 1.00 | pass |
| Data centre migration to Azure for State Street | work/state-street-azure-landing-zone-dr | 1 | 16.80 | 1.00 | pass |
| How many VMs did he build for State Street? | work/state-street-azure-landing-zone-dr | 1 | 10.52 | 1.00 | pass |
| Splitting a large Terraform state into workspaces | work/state-street-terraform-enterprise-split | 1 | 12.53 | 0.71 | pass |
| How many Terraform resources did he split per application? | work/state-street-terraform-enterprise-split | 1 | 13.93 | 1.00 | pass |
| Moving Terraform runs into Terraform Enterprise and pipelines to Harness | work/state-street-terraform-enterprise-split | 1 | 11.70 | 1.00 | pass |
| What is the VM factory? | work/state-street-vm-factory-image-bakery | 1 | 7.75 | 1.00 | pass |
| How much faster are VM builds with the golden image? | work/state-street-vm-factory-image-bakery | 1 | 12.83 | 0.67 | pass |
| Golden image bakery with Azure Image Builder or Packer | work/state-street-vm-factory-image-bakery | 1 | 19.06 | 1.00 | pass |
| Crédit du Maroc GitLab upgrade | work/credit-du-maroc-gitlab-upgrade | 1 | 21.96 | 1.00 | pass |
| How much did the Five Below audit save? | work/five-below-azure-landing-zone-audit | 1 | 12.43 | 0.63 | pass |
| Blue/green deployments at PlanHub | work/planhub-aws-platform | 1 | 17.88 | 1.00 | pass |
| Azure DevOps to Harness migration for Mott MacDonald | work/ado-to-harness-migration | 1 | 23.72 | 1.00 | pass |
| Reusable GitLab CI templates with OIDC | work/gitlab-ci-platform-templates | 1 | 18.23 | 1.00 | pass |
| DevOps for an agentic benchmark platform on Cloud Run | work/agentic-benchmark-devops-ai-eval | 1 | 17.14 | 1.00 | pass |
| Kubernetes platforms for RAK Bank and DU Telecom | work/gcc-kubernetes-platforms | 1 | 24.09 | 1.00 | pass |
| Migrating 30,000 websites to EKS | work/eks-30000-site-migration | 1 | 17.28 | 0.76 | pass |
| What is TalentLens? | projects/talentlens | 1 | 5.29 | 1.00 | pass |
| Agentic paper trading system | projects/agentic-paper-trader | 1 | 15.54 | 0.78 | pass |
| Local RAG platform retrieval evaluation | projects/local-rag-agent-platform | 1 | 15.92 | 1.00 | pass |
| Agent sandbox conformance tests | projects/agent-sandbox-conformance-kit | 1 | 16.01 | 1.00 | pass |
| OpenAI-compatible gateway for VS Code language models | projects/vscode-lm-openai-gateway | 1 | 25.17 | 1.00 | pass |
| YouTube Shorts generation pipeline | projects/shorts-engine | 1 | 10.39 | 0.62 | pass |
| Terraform modules for an Azure landing zone | projects/azure-landing-zone-terraform | 1 | 12.32 | 1.00 | pass |
| Jenkins pipelines on Kubernetes | projects/jenkins-k8s-pipeline-lab | 1 | 10.27 | 1.00 | pass |
| Interview coach for forward deployed engineers | projects/fde-coach | 1 | 13.02 | 0.71 | pass |
| 3D model of a house | projects/bhagamandala-house-3d | 1 | 13.16 | 1.00 | pass |
| Which certifications does he hold? | profile/certifications | 1 | 10.70 | 1.00 | pass |
| Is he a Certified Kubernetes Administrator? | profile/certifications | 1 | 13.71 | 1.00 | pass |
| Where did he study? | profile/education | 1 | 6.04 | 1.00 | pass |
| How can I contact him? | profile/contact | 1 | 9.51 | 1.00 | pass |
| What did he do at Capgemini? | profile/experience/capgemini | 1 | 8.43 | 1.00 | pass |
| How many engineers does he lead? | profile/summary | 1 | 5.53 | 1.00 | pass |
| Which observability tools has he used? | profile/skills/observability | 1 | 14.65 | 1.00 | pass |
| Has he worked on-site in Oman? | work/gcc-kubernetes-platforms | 1 | 9.32 | 1.00 | pass |
| How deep is his Kubernetes experience? | profile/experience/ozone | 1 | 6.66 | 1.00 | pass |
| Which clouds has he worked on? | profile/skills/cloud | 1 | 4.86 | 1.00 | pass |
| What does he do at Ensono? | profile/experience/ensono | 1 | 3.31 | 1.00 | pass |
| Has he taken LLM systems to production? | profile/skills/ai | 2 | 9.15 | 1.00 | pass |
| What is his phone number? | profile/contact | 1 | 6.33 | 1.00 | pass |
Refusals · 13/13 refused
Off-topic questions
These must be refused before any model is called. That keeps answers grounded and costs at zero.
| Question | Nearest page | Score | Coverage | Result |
|---|---|---|---|---|
| What is the capital of France? | – | 0.00 | 0.00 | refused |
| Write me a poem about cats | projects/agent-sandbox-conformance-kit | 2.92 | 0.18 | refused |
| Recipe for chocolate cake | – | 0.00 | 0.00 | refused |
| Does he know the Rust programming language? | projects/bhagamandala-house-3d | 4.81 | 0.58 | refused |
| What is his salary expectation? | – | 0.00 | 0.00 | refused |
| Who is the best footballer of all time? | profile/experience/intuitive | 2.44 | 0.21 | refused |
| Explain quantum entanglement | – | 0.00 | 0.00 | refused |
| Translate hello into Japanese | – | 0.00 | 0.00 | refused |
| What is the weather in Bengaluru today? | profile/contact | 4.40 | 0.27 | refused |
| Is he married? | – | 0.00 | 0.00 | refused |
| What is the meaning of life? | projects/gitlab-omnibus-upgrade-playbook | 2.87 | 0.26 | refused |
| Who won the cricket world cup? | projects/agentic-paper-trader | 5.50 | 0.45 | refused |
| How do I bake bread? | work/state-street-vm-factory-image-bakery | 3.61 | 0.42 | refused |
Method & limits
- The refusal gate has two signals: a minimum BM25 score, and IDF-weighted query coverage when a question mentions terms the corpus has never seen. Most off-topic questions are about something the site never mentions.
- Retrieval and refusal are measured deterministically at build time. Generated answers are not scored here, because no model is called during the build.
- At runtime the Worker also filters prompt-injection patterns, delimits context and drops any citation that doesn’t map to a retrieved chunk.
- The golden set is small and written by me, so treat it as a regression guard, not a benchmark.