Exploring open-ended scientific discovery with Station
Today we are sharing a new study of Station: can AI make scientific discoveries when there is no predefined answer or score to optimize? Working on open questions in AI interpretability, AI within Station independently reproduced substantial findings from three published studies and developed new insights into how language models learn and behave.
Our earlier studies showed that Station could make discoveries on tasks with clear numerical targets. But much of science begins without a score to optimize. The theories of relativity and evolution emerged from attempts to explain the world, requiring new questions and new ways of thinking about familiar observations. To contribute more broadly to science, AI must also learn to pursue questions whose most important findings cannot be specified in advance.
To test whether AI could pursue research without a predefined score, we chose three recent ICLR oral papers on AI interpretability. These studies seek to explain how AI works, rather than optimize a single performance metric. We asked whether Station could independently rediscover their findings, providing the research questions and experimental resources while withholding the papers’ conclusions and disabling web access. Across these tasks, Station recovered 62.7% of the evaluated findings, compared with 15.4% for Codex Multiagent-v2 and up to 20.6% for AI Scientist-v2 under matched experimental time budgets. These findings required substantive experimental or theoretical support, allowing us to assess whether AI had reconstructed the science behind the original studies.
We then investigated two questions without a reference paper defining what AI should discover. One concerned how a model’s preferences can transfer to another model through training on seemingly unrelated number sequences. Station found that restricting adaptation to early layers could restore this transfer where adaptation across the network failed. Another investigation examined why vision-language models sometimes follow language expectations instead of visual evidence. AI identified an intervention that reduced these hallucinations, independently arriving at an approach closely related to concurrent human research. Follow-up experiments confirmed improvements across additional models and data.
Together, these investigations extend Station from problems with measurable construction targets to research where deciding what to investigate is itself part of the challenge. We are releasing the code and research records alongside the paper, including the experiments, discussions and manuscripts through which the findings developed.