MSc Thesis Presentation - Ruiyu Gou

Name: Ruiyu Gou

Date & Time: July 23rd 3 - 4 pm

Location: ICCS 238

Supervisor: Michiel van de Panne

Title: Learning Temporal Action Chunking for Motor Control

Abstract:

Deep reinforcement learning has had significant success at learning motor control tasks. Typically, these policies are fully closed loop or `state indexed', implying a control policy that is queried at every control time step with the current state in order to estimate the best current action corresponding to that state. However, this approach ignores the inherent predictability of many systems, wherein the future states and actions are often quite predictable and can thus be controlled in an open-loop or `time indexed' fashion.

Chunking of action sequences is a well-established mechanism in cognitive system to enhance memory and efficiency during task learning and execution. By modelling actions in temporal chunks, one reduces the computational and perceptual demands required for control. Learning this type of temporal action abstraction remains an under-explored direction.

We present a method that learns a chunk-based state-and-time-indexed policy from any existing state-indexed reinforcement learning policy, with minimal added complexity. We show that with a straightforward multi-layer perception, the chunk-based policy can decrease the required control frequency significantly.  In particular, we show a reduction from 60Hz to 10Hz for the control of a fully 3D humanoid capable of robust and realistic movement across varying terrain. We further propose an adaptive runtime algorithm that can leverage long action chunks while reverting to single-step actions as needed in order to achieve robust behavior.