Reinforcement Learning: Rewards, Senses and PPO

The maths behind the collapsing-canyon drone, one formula per failure: returns to go, why a reward that is negative everywhere makes crashing the best move, potential-based shaping and the geodesic distance it should be measured on, the lidar fan and the stacked scans that turn it into motion, running observation normalisation, then the TD error, GAE, explained variance, the PPO clipped objective and the policy entropy that keeps it exploring.

Learning Objectives

• →Compute returns to go by backward recursion and read what discounting does to a distant reward
• →Show why a per-step penalty can make ending the episode the highest-scoring action, and why a discounted death penalty can make hesitating one
• →Apply potential-based shaping, which cannot change the optimal policy, and measure the potential on a geodesic rather than a straight line
• →Build the agent's observation: cast a lidar fan, stack past scans so motion is visible, and normalise the result with a running estimate
• →Compute TD errors and GAE advantages with correct handling of episode boundaries
• →Evaluate a value head with explained variance, and implement the PPO clipped objective and Gaussian policy entropy

Coding Problems (12)