Open-ended scientific discovery with Station

Our earlier Station studies showed that AI agents could improve scientific benchmarks and discover new mathematical constructions. In those tasks, a score helped agents tell whether a new attempt was better than the last. But much of science begins without such a measure. The theories of relativity and evolution grew from attempts to explain the world, requiring new questions and new ways of thinking about familiar observations.

For AI to contribute more broadly to science, it must also pursue questions whose most important findings cannot be specified in advance. An experiment may reveal something unexpected, challenge an explanation or suggest a better question. Deciding what to investigate is part of the research.

In this study, we asked whether Station could support that kind of inquiry. We first tested whether agents could rediscover findings from three AI interpretability studies, then gave them two exploratory questions about language models. They recovered substantial findings from the published studies and developed new insights into how models learn and behave.

Could agents rediscover the science?

To evaluate open-ended research, we needed a way to judge the findings without telling agents what to find. We chose three recent ICLR oral papers that investigate how AI works. Each offered a research question, an experimental setup and a body of findings against which we could assess the agents’ work.

We gave the agents the questions and the resources needed to study them, such as a trained model or code for training one. We withheld the papers’ conclusions and disabled web access. Before the runs began, we divided each paper’s findings into a checklist. A finding counted only when the agents explained it and supported it with evidence from their investigation.

The three questions concerned different aspects of AI. Does a reinforcement learning model form internal plans while solving a puzzle? Do language-model outputs share a mathematical structure that can help us predict them? How does a neural network decide what to remember and what to forget? These questions call for explanations, so a better performance score alone would not answer them.

We ran each task three times, with six agents powered by GPT, Claude and Gemini. Under matched cumulative experiment time, Station recovered 62.7% of the evaluated findings, compared with 15.4% for Codex Multiagent-v2 and up to 20.6% for AI Scientist-v2. Blinded assessment by human experts closely agreed with our automated evaluation.

Following an observation to an explanation

One task asked whether a model trained to solve Sokoban, a box-pushing puzzle, had learned to plan. The model had been trained through rewards for its actions, without an explicit planning algorithm. Yet its internal activity might still represent where a box should go or which moves it should make next.

The agents found representations of future actions and states inside the model. They then changed some of those representations to test whether they influenced its behavior. This is a stronger test than observing a correlation: if changing an internal representation changes the model’s actions, it helps establish what that representation does.

Station recovered 75.8% of the evaluated findings on this task. The remaining gap included the reference paper’s explanation of how the model plans both forward from its current situation and backward from a goal. Detecting an internal plan was easier than explaining how the plan was formed.

The language-model task showed another step from observation to explanation. The agents found that a model’s predictions after one passage of text could be approximated by combining its predictions after other passages. They went on to show that the same relationships held across different possible continuations. The investigation thus moved beyond detecting a pattern to testing whether that pattern could be reused. Station recovered 70.4% of the evaluated findings, although developing useful prediction and generation algorithms from those relationships remained harder.

Staying with a difficult question

Open exploration also creates a practical problem: an agent can leave a difficult investigation for an easier one before it has learned much. Or it can keep collecting measurements that have little bearing on the original question.

We introduced two mechanisms to help agents sustain worthwhile investigations. A Supervisor asks each researcher to propose a direction and commit to pursuing it for a period of time. The researcher still chooses the direction; the Supervisor checks that the work follows the proposal. Meta Reflection periodically asks agents to reconsider their work from the perspective of a human researcher: is the question important, do the experiments address it, and is it time to persist or change course?

One agent studying the Sokoban model developed a method for tracking its representations of walls and corridors. But explaining static features of the puzzle told it little about planning. During reflection, the agent recognized this gap and applied the same method to movable boxes, asking how their changing positions affected the model’s plans. It could continue using what it had built while bringing the investigation back to its central question.

Removing both mechanisms reduced rediscovery on the planning task from 75.8% to 57.9%. Agents changed direction more frequently and were less likely to build on earlier observations through experiments that tested their effect on behavior. The comparison supports the value of encouraging persistence alongside freedom to explore.

A preference hidden in number sequences

Rediscovery let us compare the agents’ work with established findings. We next asked what they would discover when no reference paper defined the desired outcome.

The first question concerned subliminal learning. A teacher model with a preference for cats can pass that preference to a student through training data consisting only of number sequences. The numbers do not mention cats, yet the student becomes more likely to express the teacher’s preference. We asked Station to investigate how this transfer happens.

The agents used LoRA, a method for fine-tuning a model by training small additions to its existing weights. They varied the capacity of those additions, known as their rank. In the settings they tested, increasing rank first strengthened the transfer and then weakened it. Giving the student more capacity to adapt did not necessarily make it learn more of the teacher’s preference.

To understand this, the agents examined the changes made during training. They found that a few learned directions in the early layers carried most of the transferred preference. Keeping those directions preserved much of the effect, even after removing most of the other changes.

This suggested a further experiment: train only the early layers and leave the later layers unchanged. At a high rank where training across the network had failed to transfer the preference, restricting training to the early layers restored it.

The rank dependence and concentration in a few directions matched independent human research reported after the models’ knowledge cutoffs. The early-layer training result went further, turning the observation into a way to restore failed transfer. We subsequently tested it across five animal preferences, at two high ranks and with three random seeds per condition. Early-layer training consistently improved transfer over training across all layers.

Helping a model trust what it sees

Our second exploratory question concerned visual hallucinations. A model can describe an image incorrectly because it lacks relevant knowledge, but it can also know the objects involved and still misread the image. We asked the agents to investigate which parts of a vision-language model contribute to these errors.

The agents proposed that the model’s expectations from language could override the visual evidence. Looking inside the network, they focused on the final two feed-forward layers, which transform the model’s internal representations near the end of its computation.

They found a simple intervention: multiply those layers’ outputs by 0.5 while the model processes the image and question. On the 40 visual questions supplied to the agents, correct answers increased from 28 to 35.

Both the intervention and the proposed explanation aligned with concurrent human research. We then tested the intervention on a separate, larger evaluation using 1,000 examples from AMBER, a benchmark for visual hallucinations. It improved results across additional models; for example, accuracy on Qwen3-VL-8B rose from 58.53% to 71.23%. The agents’ small initial experiment had identified an intervention worth investigating more broadly.

Closing the gap with human researchers

The study also helps us identify where AI research still falls short of human research. In the memory task, agents often tried to improve how much information a network could retain. That produced measurable progress, but it drew attention away from the harder question of how the network forgets. Station recovered 41.9% of the evaluated findings, with limited progress on explaining abrupt forgetting in nonlinear networks.

We also asked an author of the planning reference paper to assess 24 agent proposals that went beyond that paper’s findings. Three quarters received scores of 1 or 2 out of 5: many focused on minor details that would contribute little to understanding the main problem. Running a sound experiment and choosing an important question remain distinct abilities.

These gaps give us concrete directions for improving Station. We will work on helping agents choose scientifically important questions, develop deeper explanations from their observations and recognize when a promising finding deserves further investigation. The agents can already recover substantial scientific findings and develop new ones through sustained, collaborative research. Our next goal is to strengthen the scientific judgment that guides that work.

We are releasing the paper, code and research records, so others can follow the investigations and build on what the agents found.