A Global Workspace Theory Transformer for Deep Reinforcement Learning

Abstract

Reinforcement Learning (RL) has emerged as a powerful paradigm for enabling agents to autonomously solve complex problems by interacting with the environment they are situated in. Deep Reinforcement Learning (DRL) extends this concept by employing Deep Neural Networks (DNNs) that parameterise the agent to enable effective learning and scalability in complex environment dynamics. Thereby empowering agents to tackle tasks spanning robotics, autonomous vehicles, Artificial General Intelligence (AGI), and beyond. However, DRL still faces three core challenges, namely, effectively handling longterm temporal dependencies, avoiding catastrophic forgetting, and performing complex reasoning over environment observations and dynamics. This thesis introduces a novel approach to bolster DRL by amalgamating a transformer with a shared global workspace inspired by Global Workspace Theory (GWT), constituting a GWT transformer, into state-of-the-art DRL algorithms. The primary aim of this thesis is to tackle the three core issues, consequently improving performance metrics such as long-term average rewards DRL agents accumulate across various environments. The proposed approach rectifies the drawbacks of the conventional DNNs, such as Multilayer Perceptrons (MLPs) and Recurrent Neural Networks (RNNs), typically used in DRL to handle episodic and sequential RL data. This thesis and research direction is unique, given recent DRL work focuses more closely on the algorithms than the underpinning DNNs. However, this thesis hypothesises that the DNNs themselves play a serious and causal role in such issues. Hence, tackling this issue. The postulated improvements and hypotheses are validated through experiments and ablation studies conducted within discrete environments and continuous robotic control tasks, encompassing fully-observable and partially-observable scenarios. These approaches and findings not only outperform prior benchmarks and baselines but also mark substantial strides toward forging a comprehensive DRL architecture capable of surmounting key hurdles and delivering superior performance across various domains.