Research

Presented at the 1st Workshop on Generalization in Autonomous Driving: Paradigms, Practice, and Public Road Demonstrations — ICRA 2026.

Abstract

Autonomous Racing has seen remarkable progress through deep Reinforcement Learning (RL), primarily for four-wheeled vehicles. However, motorbikes introduce substantially greater complexity due to the need to manage balance and lean angle, in addition to more reactive steering and throttle control, and a smaller weight. In this work, we present a framework for training an autonomous agent to race a superbike in VRider SBK, a physics-accurate Unity-based motorbike simulator. Our approach integrates Soft Actor-Critic (SAC) with Self-Paced curriculum Deep reinforcement Learning (SPDL), which dynamically generates progressively more challenging tasks based on the agent’s performance — without requiring manual curriculum design. The agent’s state space comprises proprioceptive features extended with lean-angle history, along with global track features via course points. The reward signal is shaped to encourage progress along the track while penalizing instability-inducing behaviors specific to two-wheeled dynamics. Preliminary experimental results demonstrate that SPDL outperforms SAC alone in training efficiency, lap time, and driving stability across multiple tracks and motorbike models, establishing a first baseline for RL-based autonomous motorbike racing.

Motivation

Autonomous racing requires an agent to perceive its environment, plan an optimal trajectory, and execute precise control at the physical limits of the vehicle. While end-to-end deep RL has shown super-human performance for cars in high-fidelity simulators, motorbike racing remains largely unexplored. A racing motorbike introduces fundamentally different dynamics: besides steering and throttle/brake control, the agent must actively manage the lean angle to counteract centrifugal forces when cornering. Failure to control the lean angle leads to loss of balance and falling — a failure mode absent in four-wheeled vehicles. Racing bikes are also subject to stronger braking and acceleration dynamics due to a higher power-to-weight ratio.

System and Method

We train the agent in VRider SBK, a Unity-based superbike simulator by Funny Tales featuring 14 real-world Superbike championship tracks and five bike models. The integration with RL is realised through Unity ML-Agents with a Gym wrapper and Stable Baselines 3.

State space. The observation vector combines proprioceptive features (linear and angular velocity, current/previous steering and lean angles, lean-angle history, tyre slip ratios, previous actions, and orientation relative to the trajectory) with global features (60 equally-spaced 3D course points sampled along the trajectory via Catmull-Rom spline interpolation at the agent’s current speed). Including lean-angle history is key: the agent must anticipate and suppress oscillatory lean behaviour that can lead to loss of control.

Action space. Two continuous actions in (\left[-1, 1\right]): (i) steering, which also drives the lean angle, and (ii) a combined throttle/brake signal.

Reward design

The reward is dense and guides the agent to minimise lap time while managing the instability-prone dynamics of a two-wheeled vehicle. It combines a course-progress term (displacement along the trajectory, with a wrong-way multiplier), an off-track penalty and a kinetic-energy wall penalty, and a family of stability penalties — on steering, leaning, and angular-velocity changes between steps, plus sigmoid-based history penalties that discourage oscillatory steering/lean, a max-lean penalty, and tyre-slip, standstill and backward-motion penalties. An episode resets if any penalty timer expires or the bike falls.

Self-Paced curriculum

Self-Paced Deep RL (SPDL) automatically generates task distributions of increasing difficulty based on the agent’s current performance, optimizing:

$$ \max_{\nu} \; \mathbb{E}_{p(c|\nu)}\bigl[J(\pi, c)\bigr] - \alpha \, D_{\mathrm{KL}}\!\bigl(p(c|\nu) \,\|\, \mu(c)\bigr) $$

The key insight is that the trajectory the agent follows is the difficulty axis: training starts from the track median (centerline) and progresses toward the ideal racing line (the optimal in-out-in line for minimum lap time). Three context variables define the current task: (c_1) (distance from the median, higher = harder), (c_2) (distance from the ideal line, lower = harder), and (c_3) (maximum speed limit). They select the interpolated spline the agent follows via:

$$ \alpha = \frac{1}{2}\left(\frac{c_1}{c_1 + c_2} + \frac{c_3}{v_{\max}}\right) $$

As performance exceeds a threshold (\delta), SPDL shifts the distribution toward the target context (ideal line at full speed), while a KL constraint prevents abrupt transitions.

Experiments

Experiments use SAC (Stable Baselines 3) with 10 parallel environments, comparing SAC (baseline, trained on the ideal line throughout) against SPDL (SAC + Self-Paced Curriculum). The primary track is Barcelona, with additional evaluation on Cremona and Phillip Island, using the Ducati Panigale V4 R in Hot Lap (time-trial) mode.

Deployment results on Barcelona (4 sessions per model; ”—” = no lap completed):

Metric	SPDL (iter. 1150)	SAC (iter. 1150)	SPDL (iter. 1900)	SAC (iter. 1900)
Avg. lap time	1:32.85	—	1:30.75	1:31.32
Dist. to ideal (m)	2.99	4.59	3.15	2.89
Off-track time (s)	1.74	9.22	1.17	2.15
Falls	0	3	0	0
Collisions	0	0	0	0

At iteration 1150, SAC fails to complete a single lap while SPDL completes all laps with zero falls. By iteration 1900, SPDL reaches 1:30.75 vs. SAC’s 1:31.32.

Track generalization (SPDL after 1150 iterations):

Track	Avg. lap	Dist. to ideal (m)	Off-track (s)	Avg. lap (SAC)
Barcelona	1:32.85	2.99	1.74	—
Cremona	1:28.22	3.73	1.31	1:29.14
Phillip Island	1:27.49	3.10	4.35	—

Cross-bike transfer — best single lap times (mm:ss.sss), transferring the Ducati-trained policy without retraining:

Motorbike	Barcelona	Cremona	Phil. Island
Ducati Panigale V4 R (trained)	1:30.24	1:27.80	1:26.01
Kawasaki Ninja ZX-10RR	1:31.10	1:27.98	1:28.75
Honda CBR1000RR	1:32.34	1:28.81	1:24.62
Yamaha R1	1:32.53	1:29.52	1:24.29
BMW M1000 RR	1:33.13	1:30.32	—

The learned policy transfers across bikes with different weight, power and handling, confirming it captures generalizable motorbike control priors rather than bike-specific dynamics.

Videos

SPDL — Ducati at Barcelona

SAC — Ducati at Barcelona

All the recorded runs are available in the full YouTube playlist.

Qualitative results

A notable limitation is that the agent currently prioritizes speed over trajectory precision, overshooting some corners. Future work should incorporate explicit velocity-planning signals or trajectory-adherence rewards during high-curvature segments, extend to multi-track generalization, and explore asymmetric actor-critic architectures and vision-based inputs.

BibTeX

@inproceedings{ghisi2026vrider,
  title     = {Self-Paced Curriculum Reinforcement Learning for Autonomous Superbike Racing in Simulation},
  author    = {Ghisi, Luca and Essenziale, Jacopo and D'Eramo, Carlo and Luperto, Matteo},
  booktitle = {1st Workshop on Generalization in Autonomous Driving (ICRA 2026)},
  year      = {2026},
  eprint    = {2606.09236},
  archivePrefix = {arXiv}
}