Learnijoy
ResearchAI ResearchAI Engineering & DevTools

Planning Emerges in RL with Relational Hidden States

Armin Sommer· July 22, 2026 View original

Summary

This paper presents evidence that planning behavior can emerge in model-free reinforcement learning (RL) agents when they possess a network of relational hidden states. These states, anchored to environment states and exchanging messages along learned relations, effectively recover the environment's transition structure and improve policy through planning over a learned graph.

New research suggests that planning capabilities can spontaneously arise in model-free reinforcement learning (RL) systems, challenging the traditional division between model-based and model-free approaches. The study identifies that the specific structure of a neural network's hidden states is a crucial factor enabling this emergent planning. The findings indicate that when an RL agent utilizes a network of relational hidden states, where each state is tied to an environment state and communicates via learned relationships, it develops a planning mechanism. These relational hidden states are shown to reconstruct the environment's transition dynamics through their learned connections, allowing the agent to enhance its decision-making by planning over this internal, learned graph. In contrast, control agents lacking this specific binding mechanism between hidden states and environment states did not exhibit similar emergent planning. This research proposes that such architectural priors could explain the phenomenon of emergent planning in model-free RL and raises questions about its broader prevalence, potentially offering insights into how planning might arise in biological brains.

Why it matters

AI researchers and engineers can gain a deeper understanding of how to design neural architectures that foster emergent planning in RL agents, potentially leading to more intelligent and adaptable AI systems without explicit model learning.

How to implement this in your domain

  1. 1Explore neural network architectures that incorporate relational hidden states or graph-based representations for environment states.
  2. 2Design and experiment with model-free RL agents that leverage these relational structures in their hidden layers.
  3. 3Analyze the internal representations of trained agents to identify if and how planning mechanisms emerge.
  4. 4Apply these architectural insights to develop RL agents for complex, dynamic environments where planning is beneficial.
  5. 5Contribute to the theoretical understanding of emergent behaviors in AI, bridging the gap between model-based and model-free learning.

Who benefits

AI ResearchRoboticsAutonomous SystemsGamingLogistics

Key takeaways

  • Planning can emerge in model-free reinforcement learning through specific neural network architectures.
  • Relational hidden states, anchored to environment states, are key to this emergent planning.
  • These states recover environment transition structures and enable planning over a learned graph.
  • The findings suggest architectural priors are crucial for emergent planning in AI.

Original post by Armin Sommer

"arXiv:2607.18589v1 Announce Type: new Abstract: Reinforcement learning is conventionally divided into model-based and model-free methods. In this taxonomy, model-based methods perform lookahead planning over a learned world model, whereas model-free methods learn a reactive state…"

View on X

Originally posted by Armin Sommer on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Research

AI Engineering & DevToolsAI Research

New Tool Generates Contamination-Resistant, Labeled Code Datasets for LLMs

Spaghetti Architect is a new open-source tool that generates controlled, multi-language code datasets, addressing issues of contamination and lack of semantic control in existing code corpora. It creates correct-by-construction programs with adjustable "messiness" and difficulty labels, making it ideal for training and evaluating code-generating LLMs.

Yuxiang JiJul 22, 2026
AI ResearchAI Engineering & DevTools

New Method Safely Gates Hazardous LLM Knowledge Without Deletion

Researchers introduce Token Inoculation, a method that allows large language models to retain sensitive "dual-use" knowledge while selectively refusing hazardous queries. This approach uses a special token to condition the model's behavior, improving safety without sacrificing benign domain performance.

Seunghyun Lee, Dongyoon Han, Sangdoo YunJul 22, 2026
AI ResearchAI Engineering & DevTools

GNNAS-TSP Selects Optimal Algorithms for Traveling Salesman Problem

Researchers introduce GNNAS-TSP, a Graph Neural Network (GNN)-based framework for automated algorithm selection (AS) for the Traveling Salesman Problem (TSP). GNNAS-TSP learns TSP instance representations directly from raw graph data, avoiding manual feature engineering, and formulates AS as a joint cost-prediction and ranking task to select the best solver from a portfolio under fixed computational budgets.

Zhaoxuan Li, Jiale Yang, Yifei Lu, Mustafa MisirJul 22, 2026