Turing (contract) · AI training & evaluation
Frontier-model coding data & DevOps for an agentic benchmark platform
On a short contract I built and validated Codeforces-style tasks for frontier-model training data (statements, reference solutions, reasoning traces, Docker test suites and plagiarism checks) and ran DevOps for an agentic computer-use benchmark platform on Cloud Run, standing up its develop branch and staging environment.
Context
My contract with Turing covered two jobs on either side of the same problem: producing data that teaches models to write code, and running infrastructure that measures how well AI agents perform.
The first was AI training for agentic coding. I took LLM-generated, Codeforces-style competitive programming problems and turned them into tasks fit for frontier-model training data: unambiguous, correct, original and still hard.
The second was DevOps for an agentic computer-use benchmark platform, a system that evaluates AI agents on tasks they complete by operating a computer. It runs on GCP Cloud Run, with Jenkins and GitHub Actions in the delivery path. I stood up its develop branch and staging environment.
Constraints
- Frontier models solve the easy problems. A task is only worth training on if it is precise and still challenging. Ambiguity is the worst failure: a model can be marked wrong for reading an underspecified statement differently.
- A wrong reference solution poisons the data. Training on an incorrect answer teaches the wrong thing, confidently. Solutions had to be checked, not trusted.
- Originality. A lightly edited copy of a known problem tests recall, not reasoning.
- Reproducible verdicts. A test that passes on one machine and fails on another is not a test.
- A live platform. Benchmark infrastructure changes while people depend on its results, so changes need somewhere to be proven before production.
- A short contract. Whatever I set up had to be simple enough for the team to keep running after I left.
What I did
Coding data for model training
- Refined problem statements. I edited LLM-generated statements for clarity, constraints and correctness, to remove ambiguity and make them reliably challenging for frontier models. The test I apply is whether two careful readers would write programs that agree on every valid input. That means explicit bounds, defined edge cases and an output format with no room for interpretation.
- Validated brute-force and optimal reference solutions. The two solutions do different jobs. The brute force is slow but obviously correct, which makes it an oracle: on small inputs, the optimal solution has to agree with it exactly. A disagreement means one of the solutions, or the statement itself, is wrong.
- Engineered reasoning traces and metadata. I wrote chain-of-thought reasoning traces that walk from the problem to the optimal approach, along with technical metadata for each task. The traces I aim for explain why the obvious approach fails before reaching the one that works, because that step is the part worth learning from.
- Built Docker-based test suites. Tests run in a container, so the verdict on a solution is the same on any machine and for any reviewer: same toolchain, same environment.
- Ran plagiarism checks on each task to verify its originality before it went into the dataset.
DevOps for an agentic computer-use benchmark platform
- Delivery on Cloud Run. I worked as the DevOps engineer on the platform’s GCP Cloud Run deployment and its Jenkins and GitHub Actions pipelines.
- Develop branch and staging environment. I stood up a develop branch and a staging environment, so changes integrate and run somewhere production-like before release. On Cloud Run, a staging environment can be a separate service with its own configuration that scales to zero when idle, which removes the usual cost argument against having one.
Key decisions
| Decision | Why | Trade-off |
|---|---|---|
| Treat the brute-force solution as the oracle | Simple code is easier to trust than clever code. Agreement on small inputs is strong evidence that the optimal solution is right. | Brute force only runs on small inputs, so behaviour at the limits needs its own tests. |
| Run every test suite in Docker | A verdict means the same thing on every machine and for every reviewer. | Images to maintain, and container start-up adds time to each run. |
| Check originality on every task | Duplicated problems inflate a dataset without adding anything a model can learn from. | Extra time on every task. |
| Pair a develop branch with a staging environment | Integration problems surface in staging rather than in front of the people relying on the benchmark. | Another environment to keep in step with production, and the discipline of merging through develop. |
Why evaluation data needs engineering discipline
Writing a training task looks like authoring, but its failure modes are the ones I know from platform work: an unstated assumption, a check nobody ran, an environment that behaves differently on someone else’s machine. The fixes are the same too. State the contract precisely, keep an independent oracle, and make the test environment reproducible. The DevOps half of the contract was the same principle applied to infrastructure: give every change a place to fail safely before it reaches production.
Outcome
- Coding tasks delivered with refined statements, validated brute-force and optimal solutions, reasoning traces, technical metadata, Docker test suites and plagiarism checks.
- The benchmark platform gained a develop branch and a staging environment, giving changes a place to be proven before production.
I’m not quoting task volumes or scores for this contract. The deliverables were the tasks themselves and a safer release path for the platform.
What I’d do next
- Automate the stress test. Put random input generation and the brute-force-versus-optimal comparison into CI for every task, so no solution can be submitted without passing it.
- Build once, promote by digest. Build the container image once and promote the same image digest from staging to production, so what was tested is exactly what ships.
- Preview environments per pull request. Cloud Run revision tags give a revision its own URL without sending it traffic, which makes short-lived previews cheap.
- One owner per stage. With Jenkins and GitHub Actions both in the delivery path, I would write down which system owns build, test and deploy, so a failure has one obvious place to look.
— Rakshith S P