RESEARCH IN PRACTICE
New material, existing report: what actually needs to change?
An existing report asked what coding-agent leaderboards can support in a procurement decision. Research Toolkit was used to continue that work: revisit relevant sources, locate affected claims and revise the complete report while retaining the original evidence. The central recommendation remained: use leaderboards to select candidates, then test them on the buyer’s own work.
Previous complete report · 14 October update · 15 October continuation
From source checks to specific revisions
The table summarizes revisions; it is not a new benchmark result. The reading notes retain source IDs and locators; the full report links to the original public sources.
| Previous report | Material checked | Revision in the report |
|---|---|---|
| Discussed defective evaluation tasks without explaining this failure mechanism. | S2: OpenAI’s coding-evaluation audit, OpenLibrary-77c16d5. | Added the one-versus-two leading-space mismatch: an answer can follow the prompt and still fail the test. |
| Did not spell out the prerequisites for a valid SWE-Lancer run. | S6: official running instructions. | Added disabled internet access and Linux iptables; an online Mac trial is not an equivalent score. |
| Did not distinguish the different Pro V2 repair counts. | S9/S10: V2 announcement and repository instructions. | Separated 69 contradictory-instruction tasks from 529 rewritten statements, 214 test patches and 38 reference patches. These counts are not interchangeable. |
| Final control outcomes alone hid first-pass infrastructure failures. | New S11: V2 release-control record. | Separated final 642/642 control passes from three first-pass infrastructure failures per arm followed by successful reruns. |
A real revision also introduced an omission
The first update dropped an existing explanation: differences between V1 and V2 scores cannot be read directly as model progress. Author-context self-review caught it. The next revision restored that explanation and moved the citation beside the 529/214/38 figures. The first update, needs_revision response and final pass remain accessible.
The next update: locate, read, then decide
On 15 October, saving two updated source records surfaced five linked claims and report locations. Reading them led to changes in three claims, not all five: the SWE-Lancer Manager monolith-image requirement, scoring of partial patches after the Pro V2 timeout, and a recommendation to separate shared setup from per-task costs. These were details already present in the sources, not newly published upstream changes.
Previous version · Source checks · Actual save response · Changed records · Separate-context Astra review and repairs
What this case establishes—and its limits
- The first update preserved the original task’s 50 files; the next preserved its 67 files. Existing source and claim IDs were reused. First execution record · Continuation record.
- Both continuations used Full/self in the author context. Later separate-context Astra reviews are archived separately; they are not CLI independent-backend receipts.
- No coding benchmark or complete model evaluation was rerun. There was no paired update-versus-rewrite experiment, so this case does not measure overall time saved or establish a general quality or cost advantage.
- Observed timings, response-byte measurements, repeated reads and missing usage data remain in the original case record and follow-up record. Response size is not overall token, time or monetary savings. The public archive is an evidence subset, not a portable complete task snapshot.
Use this approach on your own report
Give your agent the existing report, relevant source material and the decision the report must support. Ask it to identify affected passages, reuse sound evidence, revise the complete report and return a short change note. Choose Lite or Full for the task and state the review actually performed.
Continue existing research · Copy a research request · More cases