A script with no eyes beat the bar I set for a learning robot
On 25 September I posted a document saying what one robot task would be judged on — forty trials, eight positions, a pass bar of 22 — and that the result would publish on 30 September whether it passed or failed. On 29 September the result published. It passed: 29 of 40. The robot that passed had no camera and had learned nothing. It was the pre-registered fallback, a fixed script driven from the rig’s registration alone, and it ran because the learned policy I’d hoped to test was not ready by the deadline a gate had fixed in advance.
That is the finding, and it is more useful than a pass. The task, as I registered it, is too easy to tell a learned policy from a script. The next pre-registration’s bar goes above the script’s upper confidence bound, 84 %. Nobody had to argue about that; the null decided it.
The month also produced nine dated errata, and they are the part I’d ask you to read. The base of the arm turned about four degrees in 22 hours because it was never fixed to the table; the guard I designed to catch that couldn’t resolve its own bar, and after four dated revisions I demoted it to a disclosed measurement, in writing, with the reason. One trial recorded as a success was corrected to a failure because its own photograph showed the cube in my hand. A yaw tolerance I never registered was found afterwards and disclosed, not adjudicated. Five of my own anchor steps accepted garbage before they ever saw a robot, and a dry run once satisfied a safety gate. The instrument defects each carry a fix and a fixture; the rest carry a dated erratum that says what was done and what was not. All of it is in the certificate, and what it does not cover is stated before the result, on purpose.
“Certified” here means what PR-001 §1 says it means and nothing more. §1, quoted whole:
“Certified” means every gate’s pre-registered checks were run on named artifacts, and the results are published as measured. It is not a safety certification. It is not a performance guarantee outside the measured envelope. It is not conformance to any external standard. A FAIL certificate is published as a FAIL.
This one is a Certificate of Conformance to Pre-Registration PR-001. The document, the 72-row hash chain, 88 photographs, 41 step logs and the evidence manifest are at the tag cert-001. Verify the two documents with shasum -a 256; the README gives the recipe for the JSON artifacts.
If you build robots and want your policy judged this way — bar first, result either way — the first three are free.