Research note · 2026-09-19
An early comparison of eight research agents
We compared eight research agents on one few-shot learning task in FML-Bench. Gnosys reached 78.45% test accuracy, compared with 79.21% for AIDE, the highest result in this run.
All eight used glm-5.3-flash with a 25-step search budget and seed 0. This was an exploratory comparison on one task, not a complete FML-Bench campaign or a study of performance at equal total cost.
The full comparison
The task was Data_Efficiency_easyfsl. The table reports test accuracy, with higher values indicating better performance.
| Agent | Test accuracy |
|---|---|
| AIDE | 79.21% |
| AIRA-MCTS | 78.92% |
| Autoresearch | 78.72% |
| AI Scientist v2 | 78.59% |
| Gnosys | 78.45% |
| OpenEvolve | 78.07% |
| AdaptiveSearch | 74.07% |
| The AI Scientist | 70.12% |
Gnosys finished fifth by test accuracy, 0.76 percentage points behind AIDE. A single observation cannot establish statistical equivalence between these systems.
What the timing record tells us
The historical report records 48 minutes for Gnosys and 116 minutes for AIDE, with 560,104 and 701,201 LLM tokens respectively. These observations motivated further investigation of research efficiency.
An October 5 audit found that the retained timing fields do not establish equivalent accounting boundaries across agents. The historical protocol also disclosed a shared machine. Full costs, auxiliary operations, reproduction, and final evaluation require consistent accounting before we can claim an efficiency advantage.
We therefore report those times as historical metadata, not verified speedup or dollar savings. An equal number of search steps is not necessarily an equal research budget.
A failed reproduction check belongs in the result
The retained report records that Gnosys's promotion re-execution failed because of a specification-validation error, which was fixed afterward. The recorded test score does not erase that failure. This run does not satisfy the stronger requirement that a selected result reproduce successfully before acceptance.
The agents also differed in successful execution counts: the retained summaries report 22 valid steps out of 25 for Gnosys and 25 out of 25 for AIDE. Neither execution count nor a shorter observed runtime establishes stronger research quality.
What we learned
Gnosys trailed AIDE on both its best validation score and selected test score. That makes candidate generation and productive use of the research budget important questions alongside final selection.
We are testing how exploration and verification should share that budget. The goal is better independently evaluated results for the total resources spent. A gain must survive fresh tasks and account for failed attempts as well as successful ones.
Scope and evidence
This comparison was recorded on September 19, 2026. Its task had already informed development. It establishes neither cross-domain method transfer nor autonomous recursive self-improvement, and it is not a comparison with RRSI.
Download the numerical readout
The readout transcribes the retained historical table and identifies its source digest. It is not a complete reproduction package. The timing and reproduction qualifications above are part of the result.