Model-Based Offline Adaptive Policy Optimization with Episodic Memory

Offline reinforcement learning (RL) is a promising direction to apply RL to real-world by avoiding online expensive and dangerous exploration. However, offline RL is challenging due to extrapolation errors caused by the distribution shift between offline datasets and states visited by behavior policy. Existing model-based offline RL methods set pessimistic constraints of the learned model within the support region of the offline data to avoid extrapolation errors, but these approaches limit the generalization potential of the policy in out-of-distribution (OOD) region. The artificial fixed uncertainty calculation and the sparse reward problem of low-quality datasets in existing methods have weak adaptability to different learning tasks. Hence, a model-based offline adaptive policy optimization with episodic memory is proposed in this work to improve generalization of the policy. Inspired by active learning, constraint strength is proposed to trade off the return and risk adaptively to balance the robustness and generalization ability of the policy. Further, episodic memory is applied to capture successful experience to improve adaptability. Extensive experiments on D4RL datasets demonstrate that the proposed method outperforms existing state-of-the-art methods and achieves superior performance on challenging tasks requiring OOD generalization.