Research note · 2026-10-12
Learning which research methods earn their place
Research update · October 12, 2026
An autonomous researcher makes two kinds of decisions: what solution to try, and how to spend its next experiment. Our recent work tests the second question and builds toward a larger goal: researchers that improve their own methods.
The experiments below concern causal discovery on FML-Bench’s gCastle task. Structural Hamming distance (SHD) measures errors in the recovered causal graph; lower is better. They are separate development experiments, with different search budgets and runs—not one leaderboard.
More drafting helped in one comparison
We compared Gnosys’s existing sequential search, G0, with a draft-only variant. G0 begins by drafting approaches, then can improve or repair existing candidates. Draft-only spends every attempt on another approach, informed by the previous attempts.
Each arm received eight candidate-evaluation attempts per run, across two run seeds. The comparison preserved the prompts, model, scoring, selection, and reproduction process. All four selections were frozen before separate final evaluations.
| Run seed | G0 final SHD | Draft-only final SHD |
|---|---|---|
| 1 | 73 | 53 |
| 2 | 71 | 64 |
Draft-only achieved lower error in both runs. It also used about 50% more research time and 48% more estimated direct compute-plus-model cost. Equal attempt limits did not produce equal resource use. The cost estimate excludes storage, requests, transfer, retention, and shared overhead; it is not an invoice-reconciled total.
Three G0 attempts and two draft-only attempts failed and remain included in the budgets. Both arms completed the planned searches and reproduced their selections. The result supports fresh-task confirmation; it does not establish an efficiency advantage or justify a general claim that iteration is inferior.
A different intervention did not help
In a separate eight-attempt comparison, we tested more focused drafting prompts against G0 on two run seeds.
| Run seed | G0 final SHD | Focused drafting final SHD |
|---|---|---|
| 1 | 47 | 92 |
| 2 | 81 | 81 |
Focused drafting lost once and tied once. We retained G0 as the default. The G0 values differ between these experiments because they are separate runs; shared seed labels do not guarantee matched model randomness across experiments.
Together, these findings show why research methods need their own experiments. A plausible change can fail, and an improvement in final quality can require more resources.
A coordinated research team completed the loop
On October 10, a coordinated swarm completed a six-attempt gCastle smoke test, fresh reproduction, and separate final evaluation. Four search attempts succeeded and two failed. Specialist proposals, two independent reviewer ballots, a lead synthesis, and a challenger selection were exercised during the run.
The selected candidate reached SHD 43. This establishes that the coordinated loop ran end to end in this setting. No matched G0 or uncoordinated-specialist control ran alongside this smoke, so it does not establish that coordination improved quality or efficiency. It also did not exercise the full autonomous objective-engineering system.
Now testing the measurements that guide research
Our current CausalML study compares the same coordinated swarm with objective engineering enabled and disabled. In the enabled arm, agents can construct and revise proxies and diagnostics during search while the final benchmark target remains fixed.
Both arms receive ten candidate-evaluation attempts and use the same reserved measurement split, excluded from candidate training. Additional measurement costs are reported separately. This is a disclosed amendment to the development protocol, not an unchanged native benchmark comparison. MCGrad is used only for supported binary diagnostics, not as a calibration guarantee for continuous error.
As of this October 12 update, the paired study has no completed comparative readout. It tests the contribution of objective engineering within Gnosys; it does not test superiority over other frameworks or recursive self-improvement.
The next question: can a better researcher improve its successor?
We are designing a study in which an improved researcher takes over subsequent improvement rounds. The central test is whether that handoff produces stronger further improvements than continuing with the original researcher, on fresh problems and with comparable total budgets.
Our historical-science thesis explores a complementary training environment: discoveries unknown to the researcher, with observations grounded in existing measurements or validated simulations. Transfer from those environments to unfamiliar science remains a hypothesis.
Neither the search-policy experiments nor the swarm smoke demonstrate recursive self-improvement. They provide working components and evidence for deciding what to test next.
Scope and source records
The completed experiments above use one development-exposed task. Two run seeds are not two independent datasets, and these results do not establish broad statistical superiority. The policy comparisons used glm-5.3-flash; they are distinct from the earlier few-shot learning comparison.
Download the numerical summary and source hashes. This summary preserves both policy comparisons, the swarm smoke, and their limitations without exposing private execution records.
Explore the research vision or discuss a research partnership.