Episodic Memory and Deep Q-Networks in machine learning explained

Introduction

In recent years, deep neural networks (DNN) have made significant progress in reinforcement learning algorithms. In order to achieve desirable results, these algorithms, however, suffer from sample inefficiency. A promising approach to tackling this challenge is episodic memory-based reinforcement learning, which enables agents to grasp optimal actions rapidly. Using episodic memory to enhance agent training, Episodic Memory Deep Q-Networks (EMDQN) are a biologically inspired RL algorithm. Research shows that EMDQN significantly improves sample efficiency, thereby improving the chances of discovering effective policies. It surpasses both regular DQN and other episodic memory-based RL algorithms by achieving state-of-the-art performance on Atari games with just 1/5 of the interactions required by traditional methods.

Understanding the Challenge of Sample Inefficiency

In RL research, deep neural networks have revolutionized research by combining convolutional neural networks with Q-learning to achieve human-level performance in Atari games. It is still a challenge for RL algorithms to overcome sample inefficiency, despite these achievements. In order for DQN, for example, to learn and generalize a strong policy, millions of encounters with the environment are required. In DQN, learning is done at a slow speed to ensure stability, but it results in a slower learning rate.

Episodic Control: A Data-Efficient Approach

Research has proposed episodic control (EC) as a data-efficient solution for decision-making problems. In EC, the most rewarding episodes are memorized during training and replayed during evaluation. For storing and updating episodic memories, EC relies on a lookup table, unlike parametric value functions. Comparatively to DNN-based RL approaches, table-based episodic control suffers from limited generalization and memory scalability issues.

Introducing Episodic Memory Deep Q-Networks (EMDQN)

The purpose of this paper is to present EMDQN, a novel RL algorithm that leverages episodic memory to enhance agent training. Human brains make decisions and control motion through multiple learning systems that interact and compete to formulate optimal strategies. A combination of DQN generalization and episodic control is called EMDQN. EMDQN achieves superior learning efficiency by distilling episodic memory information into a parametric model. As compared with existing methods, our algorithm learns robust policies more quickly and with fewer training data. Further, EMDQN addresses the issue of overestimation of Q-values in Q-learning-based agents.

In EMDQN, the striatum serves as an inference target, while the hippocampus serves as a memory target. These targets serve as learning objectives for the agent.

The loss function used in EMDQN is defined as follows −

L = α(Qθ - S)^2 + β(Qθ - H)^2

Here, Qθ is the value function parameterized by θ, which represents the estimated value of taking an action in a given state.

The inference target S is computed as follows −

S(st, at) = rt + γ max(Qθ(st+1, a')), for all possible actions a'

Here, rt is the immediate reward received after taking action at in state st, γ is the discount factor, and max(Qθ(st+1, a')) represents the maximum estimated value of all possible actions in the next state st+1.

The memory target H is defined as the best memorized return −

H(st, at) = max(Ri(st, at)), for i ∈ {1, 2, ..., E}

Here, Ri(st, at) represents the future return obtained when taking action at in state st during the i-th episode. E represents the total number of episodes experienced by the agent.

The loss function combines the squared differences between the value function Qθ and the inference target S, as well as between Qθ and the memory target H. The weights α and β control the relative importance of each target in the overall loss function.

By minimizing this loss function, the agent aims to improve the estimation of the value function Qθ based on both the immediate rewards and the best memorized returns. This allows the agent to rapidly latch onto high-rewarding policies while still benefiting from the slow optimization of the neural network for state generalization.

Advantages of EMDQN

Typically, episodic memory is used for direct control, but we aim to make DQN more efficient by leveraging it. There are several key aspects of DQN that can benefit from episodic memory.

• Slow reward propagation − Traditional value-bootstrapping methods like Q-learning provide updates based on one-step or nearby multi-step rewards, resulting in limited data efficiency. To overcome this, we propose using Monte Carlo (MC) return as the learning target. MC return offers better reward propagation, but it introduces higher variance. Our challenge is to effectively utilize MC return without compromising stability due to high variance.

• Single learning model − Most RL algorithms rely on a single learning model. Scalable deep RL methods, such as DQN and A3C, simulate the striatum in the human brain and learn neural decision systems. On the other hand, table-based methods like MFEC and NEC simulate the hippocampus and store experiences in a memory system. In this paper, we argue that incorporating both approaches during training can better replicate the working mechanism of the human brain.

• Sample inefficiency − Interacting with the real environment can be costly in terms of time and resources. Conventional DQN algorithms require millions of interactions with the simulated environment to converge. While techniques like prioritized experience replay and model-based RL can alleviate sampling costs to some extent, there is a need for more efficient ways to utilize samples and enhance learning.

To address these challenges, we propose Episodic Memory Deep Q-Networks (EMDQN), which leverages table-based episodic memory to accelerate the training of agents. By integrating episodic memory into the learning process, our agent can quickly latch onto valuable experiences and leverage them for more efficient learning.

In summary, our research focuses on utilizing episodic memory to enhance DQN in terms of reward propagation, learning model architecture, and sample efficiency. By leveraging episodic memory, EMDQN offers the potential to accelerate the training process and improve the overall performance of RL agents.

Conclusion

Episodic Memory Deep Q-Networks (EMDQN) introduces a biologically inspired RL algorithm that leverages episodic memory to improve agent training. By combining the strengths of DQN and episodic control, EMDQN offers enhanced sample efficiency and outperforms existing methods in terms of training time and accuracy. This algorithm holds immense potential for making RL more applicable in real-world scenarios. With its remarkable performance on Atari games, EMDQN paves the way for more efficient and effective reinforcement learning algorithms.