🔍 Read the full analysis: A Measured Win For Qwen: One Edit, Better JSON Tab Escaping on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
An experiment reported by ThorstenMeyerAI.com found that a targeted, rank-one edit to one Qwen3-4B-Instruct-2507 matrix improved exact answers on a frozen JSON-escaping test from 49 of 120 cases to 70. The narrow result did not improve table-row extraction, and 50 of 120 final test answers remained wrong; the work has not had an independent review.
An experiment reported November 5 by ThorstenMeyerAI.com found that a single matrix edit improved Qwen3-4B-Instruct-2507’s exact score on a frozen test of JSON text escaping from 49 of 120 cases to 70. The change, made by an OpenAI GPT-6 Astra agent, addressed a narrow failure involving tabs and other special characters; it did not improve a separate table-extraction task.
The test asked the model to place short texts containing tabs, line breaks, quotation marks or backslashes into a JSON object. Answers counted as correct only when a JSON parser could recover the original text exactly. In JSON strings, for example, a tab must be represented with the escape sequence \t, rather than inserted as a raw tab character.
According to the experiment report, the edited model answered 70 of 120 frozen final cases exactly, compared with 49 before the edit: a gain of 21 cases, or 17.5 percentage points. The report says 21 cases were fixed and none that had been correct became incorrect. The edit was a calculated rank-one update to the output projection of the model’s final MLP layer; it did not use gradient training. The other 397 stored tensors were unchanged.
The report also describes limits. The initial model answered 16 of 32 screening cases correctly, and a more explicit escaping prompt also scored 16 of 32. On the final test, 50 of 120 answers were still wrong. A separate task—extracting a table row—remained at zero of 30. The source characterizes the cases as synthetic and says no second AI has yet independently reviewed this experiment.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in Structured Output
The result matters as a small, measurable example of changing model behavior through a direct weight edit rather than relying only on prompt wording. The targeted problem is practical: software commonly exchanges data through JSON, and a string that fails to preserve tabs or other special characters can be unusable even if it looks close to the input. Here, correctness was judged by whether parsing returned the exact source text, not simply whether the output looked like valid JSON.
But the result does not establish that the edit makes Qwen broadly more reliable. It covers one model, one narrow task and 120 final test cases, with synthetic examples. Half of the final answers remained incorrect, and the unchanged table-extraction score indicates that the edit did not solve a general information-extraction problem. Readers should treat the reported improvement as evidence about this test, not a general performance guarantee.
Products Worth Considering
Amazon Kindle Paperwhite (2024), 16 GB, 7-inch Display, Black
Apple Watch SE 3 GPS Smartwatch, 40mm
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
As an affiliate, we earn on qualifying purchases.
The Clue From an Earlier Test
This was the third experiment in a week of work on changing model weights, according to the source. An earlier attempt by Anthropic’s Claude edited a local model called Ornith to make it finish more legal answers. Although the number of completed answers rose, the legal score remained zero. GPT-6 Astra later reviewed that work and, the source says, identified problems including a resume bug, a command-line fault and side effects from weight conversion.
Before the Qwen JSON-escaping test, GPT-6 Astra had tried a different edit on the same model for extracting five fields from synthetic business messages. Both edited and original versions scored 97 of 200 on the frozen final test. The report says both also produced 47 invalid answers, mostly by placing raw tabs in JSON strings. That earlier failure supplied the lead for the new experiment: instead of searching broadly, the agent targeted the known escaping error.
The source says the site owner instructed the agent on November 5 to inspect Qwen and try to improve it. GPT-6 Astra, operating as a Codex agent, planned and ran the experiment, including building tests, computing the edit and writing the report. Claude had no role in this experiment, although a Claude agent later helped research and draft the source article.
“The edited model got 70 of 120 test cases exactly right instead of 49.”
— ThorstenMeyerAI.com experiment report
Independent Checks Still Pending
The reported figures come from the experiment records described by ThorstenMeyerAI.com; the source says no independent second-AI review has yet checked this run. The test set was frozen before the result was measured, according to the report, but the material provided does not include an external replication or results on real-world JSON data.
It is also unclear whether the gain holds across different prompts, languages, longer inputs or other model versions. The source does not provide enough information to determine how representative the synthetic cases are of software users’ actual workloads. The measured improvement is specific to the described test and should not be taken as proof that the model now handles JSON escaping reliably in general.
Replication and Broader Tests
The source identifies independent review as the main missing check. A follow-up evaluation could test the same edit on a separately prepared set of examples, check whether the improvement persists under varied prompts and input lengths, and measure performance on data closer to real applications. It could also examine whether the edit affects tasks beyond escaping, since the table-row score remained unchanged in this experiment.
ThorstenMeyerAI.com’s report and accompanying video present the result as part of an ongoing series of weight-editing experiments. The source material does not give a date for a replication or another review, so no next test is confirmed. Until further checks are reported, the 49-to-70 result remains a narrow finding from one frozen synthetic test.
Key Questions
What changed in the Qwen model?
An OpenAI GPT-6 Astra agent made a calculated rank-one update to the output projection of Qwen3-4B-Instruct-2507’s final MLP layer. The source says the model’s other 397 stored tensors were unchanged.
How much did the test score improve?
The model’s exact score rose from 49 of 120 to 70 of 120 on the frozen final test. That is 21 additional correct cases, or 17.5 percentage points, according to the experiment report.
What did the test measure?
It measured whether the model could copy short texts containing special characters into JSON and preserve them exactly when the JSON was parsed. A tab, for example, needed to be represented as \t inside the JSON string.
Does the result show that Qwen is now reliable at JSON?
No. The reported result covers a narrow synthetic test, and 50 of 120 final answers remained incorrect. The source also reports no improvement on a separate table-row extraction task and says the experiment has not yet received independent review.
Source: ThorstenMeyerAI.com
