Feudal Reinforcement Learning (FRL) System
Jump to navigation
Jump to search
A Feudal Reinforcement Learning (FRL) System is a Hierarchical Reinforcement Learning System based on hierarchical structure of a feudal fiefdom.
- Context:
- It was first introduced by Dayan & Hinton (1992).
- It can be used to divide a single task into sub-tasks at multiple management levels.
- It can be used to assign tasks hierarchically to different modules:
- Manager decides upon an action based on a performance metric and instructs sub-managers to perform a sub-task.
- Sub-Manager learns how to satisfy all the sub-tasks set by its manager.
- …
- Example(s):
- Counter-Example(s)
- See: FeUdal Network (FuN), Policy Gradient training System, Feudalism, FeUdal Network Worker Module, FeUdal Network Manager Module, Q-Learning System, Maze Task.
References
1992
- (Dayan & Hinton, 1992) ⇒ Peter Dayan, and Geoffrey E. Hinton. (1992). “Feudal Reinforcement Learning.” In: Proceedings of Advances in Neural Information Processing Systems 5 (NIPS 1992).
- QUOTE: Consider a control hierarchy in which managers have sub-managers, who work for them, and super-managers, for whom they work. If the hierarchy is strict in the sense that managers control exactly the sub-managers at the level below them and only the very lowest level managers can actually act in the world, then intermediate level managers have essentially two instruments of control over their sub-managers at any time - they can choose amongst them and they can set them sub-tasks. (...)
We sought to build a system that mirrored the hierarchical aspects of a feudal fiefdom, since this is one extreme for models of control. Managers are given absolute power over their sub-managers - they can set them tasks and reward and punish them entirely as they see fit.
- QUOTE: Consider a control hierarchy in which managers have sub-managers, who work for them, and super-managers, for whom they work. If the hierarchy is strict in the sense that managers control exactly the sub-managers at the level below them and only the very lowest level managers can actually act in the world, then intermediate level managers have essentially two instruments of control over their sub-managers at any time - they can choose amongst them and they can set them sub-tasks.