🔍 Read the full analysis: GPT-6 Astra’s Findings On Claude’s Model Edit Go Beyond Green Tests on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5. The review found reproducible problems that green unit tests had missed, and corrected claims about the edit’s effects; the 10 published tasks showed more answers but no improvement in the legal score.
OpenAI’s GPT-6 Astra, running in Codex, found reproducible faults in a Claude Opus 5.5-led experiment that edited the weights of the local model Ornith 1.5, including cases where a run could appear successful without evaluating any tasks. The review also found that converting model weights changed them even when the intended edit strength was zero, adding qualifications to reported results that matter because the experiment’s legal score remained zero.
The experiment, conducted by a Claude Opus 5.5 session from Anthropic, used a directional ablation inspired by a MIT CSAIL post on weight-space programming. It identified what the authors called a “loop direction” in Ornith 1.5 and weakened it in the model weights without training. On the 10 published Lexassay tasks, the edited model answered all 10, compared with four for the unedited model; its legal score remained zero. The source says the runs are public on Lexassay.
GPT-6 Astra was assigned a separate role: review the implementation, reproduce faults, take measurements and correct the authors’ descriptions. It did not run a competing model-improvement experiment, change the main test rig or inspect hidden benchmark prompts, topics, answers or reference solutions, according to the source’s account of the review. The Claude session implemented the fixes.
The first review, on November 4, reported four problems: resuming a run could retain answers from a different model or sampling setup; a mistyped task ID could yield a successful run with no evaluations; the real command line failed despite passing parser unit tests; and weight conversion altered values even at zero edit strength. The experiment’s authors corrected the reported issues, including rejecting unknown task IDs and changing how edited tensors were stored.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Test Results Needed a Second Check
The review shows why passing unit tests do not establish that an experiment’s full workflow is reliable. The reported failures occurred at points where a benchmark could produce misleading results: stale answers could be reused, a task typo could look like a completed run, and a command could fail outside the unit-test environment. Those problems could affect whether a result is reproducible and whether it measures the intended setup.
The weight-conversion finding also bears directly on interpreting the model comparison. In the first review, a zero-strength conversion changed 0.84% of the matrix overall, with 105,315 of 1,048,576 reconstructed values differing. The source’s updated summary reports a separate later measurement: storing one real matrix in a different number format changed 0.51% of it, or 915,360 of 1,048,576 values. These are measurements of conversion, not legal performance or an error rate. They indicate that changes attributed to an edit need to be distinguished from effects of the representation and storage process.
The review’s value is not evidence that the edited model became better at law. The reported benchmark result remained zero on the legal score. Rather, it offers a case study in independent checking: a reviewer from another company examined the test machinery and narrowed the claims, while the model-editing result itself remains limited to the published tasks and reported setup.
Products Worth Considering
LG UltraGear 32GX850A-B 32-inch 4K UHD OLED Gaming Monitor
DJI Osmo Mobile 8 3-Axis Phone Gimbal Stabilizer
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
As an affiliate, we earn on qualifying purchases.
How the Review Was Separated
Two AI sessions began work on the experiment on November 4, 2026. After the participants recognized that they were working on the same project, the roles were divided. Claude handled the plan, test rig, model edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra, from OpenAI and running in Codex, was assigned to validate the other session’s implementation.
The review was designed to stay separate from the main experiment. GPT-6 Astra wrote check programs rather than changing the implementation, ran real entry points with small reproducible examples, and did not access the hidden benchmark material, according to the source. The Claude session made each reported fix. This arrangement makes the review an audit of implementation and claims, not an independent replication of the model’s benchmark performance.
The source’s updated account says the results now refer to 10 published Lexassay tasks, and that the runs are public. It also describes a second review round in which all 12 of the author’s tests passed while four of the reviewer’s six provenance checks failed. The source does not provide enough detail in the supplied material to explain each of those six checks, so that figure should not be read as a general model failure rate.
Limits of the Reported Checks
The source says GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth blind grader and measured conversion on a CPU rather than the production GPU path. The supplied account does not establish whether the conversion measurements would be identical on that GPU path, or whether every possible configuration has been checked.
The benchmark result also has a narrow scope: the reported comparison covers 10 published tasks, while the legal score stayed at zero. The source does not provide enough information here to assess broader legal performance, performance on other benchmarks or whether the apparent increase in answered tasks generalizes. The review identified and prompted fixes to implementation issues, but it did not independently reproduce the full benchmark experiment.
The account reports that four of six reviewer provenance checks failed in the second round, but does not specify all six checks or their individual consequences in the supplied material. The measured conversion differences likewise concern a weight matrix and its numerical representation; they do not by themselves establish how much those differences affected the benchmark outputs.
Public Runs and Further Validation
The immediate next step is for readers and other reviewers to examine the published Lexassay runs and the updated implementation, including the weight-edited results, edit control and model-card sampling identified by the source. The experiment’s authors have reported fixes for the first-round faults, but independent review of those fixes and reproduction of the full test sequence would help establish how reliably the results can be repeated.
Further validation would need to distinguish effects from the intended weight edit from changes caused by conversion, and check the workflow across the hardware path used for production runs. Any future performance claim would also need to state which tasks and scoring measures it covers. Based on the supplied account, no new legal-score improvement or broader benchmark result has been announced.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and reporting of a Claude Opus 5.5-led weight-editing experiment on Ornith 1.5. It reproduced faults, measured weight conversion and suggested corrections; it did not conduct a separate model-improvement experiment.
What did the edited model score?
On the 10 published Lexassay tasks, the edited model answered all 10, compared with four for the unedited model. The reported legal score stayed at zero, so the answer count should not be treated as evidence of improved legal performance.
What problems did the review find?
The first review found that a resumed run could reuse answers from a different setup, an unknown task ID could produce a successful run with zero evaluations, a real command could fail despite green parser tests, and weight conversion changed values even at zero edit strength.
Did the reviewer inspect the hidden benchmark?
According to the source’s account of GPT-6 Astra’s second review, it did not inspect hidden benchmark prompts, topics, answers or reference solutions. It reviewed the implementation and checks rather than acting as a benchmark grader.
Has the experiment been independently replicated?
The supplied account describes an independent implementation review, not a full independent replication of the benchmark experiment. The runs are reported as public, but the reviewer did not rerun every test in its final documentation step.
Source: ThorstenMeyerAI.com
