PhD thesis defence - Daniele Reda

Name: Daniele Reda
Date: February 5, 2025
Time: 2pm-5pm
Location: ICICS 238 / Zoom link will be attached when available
Supervisor: Michiel van de Panne
Title: Physics-based Character Controllers with Reinforcement Learning

Abstract:

Physics-based character control has advanced significantly in recent years, with reinforcement learning (RL) emerging as a powerful method for producing general, versatile controllers. However, applying RL to humanoid control in animation and robotics poses fundamental challenges, including brittle policies, difficulties in exploring high-dimensional spaces, and a reliance on high-quality motion capture data. This thesis addresses these challenges, offering insights and methods in physics-based character control.
We begin with a survey of learning methods for humanoid control, identifying the exploration problem as a fundamental limitation for developing general controllers. The absence of expert data and the complexity of state-action spaces hinder RL’s effectiveness. We review existing solutions to these challenges, highlighting their strengths and limitations.
Next, we show the crucial role of environment design in RL. Poor design choices in key components, such as state representations, reward structures, or action spaces, can result in brittle and inefficient learning. Our analysis highlights how thoughtful environment enhances controller robustness and generalization.

A key challenge in RL is the heavy reliance on extensive, high-quality motion capture datasets to guide learning. This raises the question: what happens if we lack data for specific motions, either because it is unavailable or unfeasible to collect? To address this, we introduce two strategies that reduce the dependency on motion capture data.
The first approach uses a simplified physical model to provide motion priors, enabling controllers to learn complex behaviors without reference data. Through learning a brachiating controller, we show how the simplified model guides the center-of-mass trajectory and grasp timing, allowing the full model to efficiently learn swinging behaviors.

The second strategy focuses on sparse motion retargeting, adapting data from unrelated characters to new morphologies. Our framework retargets sparse sensor data to physically simulated characters with different skeletal structures. Using physics as a prior to refine the kinematic motion and overcoming the exploration problem, our method produces robust real-time controllers for applications in virtual reality (VR) and robotics.
Together, these contributions enable the development of robust, adaptable controllers, broadening RL’s applicability to physics-based animations and advancing the state of the art in learning character control.