Evals · run on every deploy

The assistant, measured in public.

Every build runs a golden set of 41 on-topic questions and 13 off-topic ones against the exact BM25 index the Worker serves. If retrieval regresses, it shows up here first.

corpus de379456be92 · 182 chunks · score floor 2.5 · coverage floor 0.65 · ran

Golden set · 41/41 passing

On-topic questions

A case passes when an expected page ranks in the top three and the refusal gate lets the question through.

QuestionTop pageRankScoreCoverageResult
What did he build for State Street?work/state-street-vm-factory-image-bakery28.221.00pass
Disaster recovery with Azure Site Recoverywork/state-street-azure-landing-zone-dr112.021.00pass
Data centre migration to Azure for State Streetwork/state-street-azure-landing-zone-dr116.801.00pass
How many VMs did he build for State Street?work/state-street-azure-landing-zone-dr110.521.00pass
Splitting a large Terraform state into workspaceswork/state-street-terraform-enterprise-split112.530.71pass
How many Terraform resources did he split per application?work/state-street-terraform-enterprise-split113.931.00pass
Moving Terraform runs into Terraform Enterprise and pipelines to Harnesswork/state-street-terraform-enterprise-split111.701.00pass
What is the VM factory?work/state-street-vm-factory-image-bakery17.751.00pass
How much faster are VM builds with the golden image?work/state-street-vm-factory-image-bakery112.830.67pass
Golden image bakery with Azure Image Builder or Packerwork/state-street-vm-factory-image-bakery119.061.00pass
Crédit du Maroc GitLab upgradework/credit-du-maroc-gitlab-upgrade121.961.00pass
How much did the Five Below audit save?work/five-below-azure-landing-zone-audit112.430.63pass
Blue/green deployments at PlanHubwork/planhub-aws-platform117.881.00pass
Azure DevOps to Harness migration for Mott MacDonaldwork/ado-to-harness-migration123.721.00pass
Reusable GitLab CI templates with OIDCwork/gitlab-ci-platform-templates118.231.00pass
DevOps for an agentic benchmark platform on Cloud Runwork/agentic-benchmark-devops-ai-eval117.141.00pass
Kubernetes platforms for RAK Bank and DU Telecomwork/gcc-kubernetes-platforms124.091.00pass
Migrating 30,000 websites to EKSwork/eks-30000-site-migration117.280.76pass
What is TalentLens?projects/talentlens15.291.00pass
Agentic paper trading systemprojects/agentic-paper-trader115.540.78pass
Local RAG platform retrieval evaluationprojects/local-rag-agent-platform115.921.00pass
Agent sandbox conformance testsprojects/agent-sandbox-conformance-kit116.011.00pass
OpenAI-compatible gateway for VS Code language modelsprojects/vscode-lm-openai-gateway125.171.00pass
YouTube Shorts generation pipelineprojects/shorts-engine110.390.62pass
Terraform modules for an Azure landing zoneprojects/azure-landing-zone-terraform112.321.00pass
Jenkins pipelines on Kubernetesprojects/jenkins-k8s-pipeline-lab110.271.00pass
Interview coach for forward deployed engineersprojects/fde-coach113.020.71pass
3D model of a houseprojects/bhagamandala-house-3d113.161.00pass
Which certifications does he hold?profile/certifications110.701.00pass
Is he a Certified Kubernetes Administrator?profile/certifications113.711.00pass
Where did he study?profile/education16.041.00pass
How can I contact him?profile/contact19.511.00pass
What did he do at Capgemini?profile/experience/capgemini18.431.00pass
How many engineers does he lead?profile/summary15.531.00pass
Which observability tools has he used?profile/skills/observability114.651.00pass
Has he worked on-site in Oman?work/gcc-kubernetes-platforms19.321.00pass
How deep is his Kubernetes experience?profile/experience/ozone16.661.00pass
Which clouds has he worked on?profile/skills/cloud14.861.00pass
What does he do at Ensono?profile/experience/ensono13.311.00pass
Has he taken LLM systems to production?profile/skills/ai29.151.00pass
What is his phone number?profile/contact16.331.00pass

Refusals · 13/13 refused

Off-topic questions

These must be refused before any model is called. That keeps answers grounded and costs at zero.

QuestionNearest pageScoreCoverageResult
What is the capital of France?–0.000.00refused
Write me a poem about catsprojects/agent-sandbox-conformance-kit2.920.18refused
Recipe for chocolate cake–0.000.00refused
Does he know the Rust programming language?projects/bhagamandala-house-3d4.810.58refused
What is his salary expectation?–0.000.00refused
Who is the best footballer of all time?profile/experience/intuitive2.440.21refused
Explain quantum entanglement–0.000.00refused
Translate hello into Japanese–0.000.00refused
What is the weather in Bengaluru today?profile/contact4.400.27refused
Is he married?–0.000.00refused
What is the meaning of life?projects/gitlab-omnibus-upgrade-playbook2.870.26refused
Who won the cricket world cup?projects/agentic-paper-trader5.500.45refused
How do I bake bread?work/state-street-vm-factory-image-bakery3.610.42refused

Method & limits

esc
↑↓ navigate ↵ opentry whoami or sudo hire rakshith