user@devops:~$ cat README.md
Deep RL — Policy Gradients: REINFORCE vs PPO
# Description
Two policy-gradient algorithm families implemented from scratch in pure PyTorch (no stable-baselines3, no rllib) and compared on Gymnasium's CartPole-v1. REINFORCE with baseline (Williams 1992) updates the policy with the gradient of log π(a|s) · (G_t − V(s_t)) using full-episode returns and a learned value function as baseline to reduce variance. PPO (Schulman et al. 2017), the modern RL standard, estimates advantages with GAE(λ=0.95), clips the policy ratio to [1−ε, 1+ε] via its clipped objective, adds an entropy bonus for exploration, trains 4 minibatch epochs per rollout and collects data with 4 parallel environments. Both share the same MLP 64-64 network (softmax policy + value heads) and the same solve threshold (mean reward ≥ 475): REINFORCE solves it in 85,129 env steps (1,450 episodes, 104.5s) while PPO takes 51,200 env steps (iteration 25, 24.3s) — PPO is 4.3x faster wall-clock and 1.66x more sample-efficient. Key findings: REINFORCE's high variance causes periodic performance collapses (eval drops from 235 to 50 between episodes 1200-1250) that PPO avoids thanks to clipping; PPO's clip fraction peaks at 39.8% in the first iterations and falls as training stabilizes. 9 visualizations: learning curves (raw + EMA), final evaluation, policy entropy, PPO internals (policy loss, value loss, clip fraction), action probabilities in one episode (before vs after training), stochastic return distribution, GAE advantages (iteration 1 vs trained policy) and a summary table.
# Key features
$ REINFORCE with baseline from scratch: policy gradient of log π(a|s) · (G_t − V(s)) over full-episode returns
$ PPO from scratch: clipped objective [1−ε, 1+ε], GAE(λ=0.95), entropy bonus, minibatch epochs and 4 parallel environments
$ Shared network (MLP 64-64 with softmax policy + value heads) and base hyperparameters for a fair comparison
$ Both solve CartPole-v1 (reward ≥ 475): REINFORCE in 85,129 steps / 104.5s, PPO in 51,200 steps / 24.3s
$ PPO is 4.3x faster wall-clock and 1.66x more sample-efficient than REINFORCE
$ Final evaluation ~500 for both (greedy); stochastic return distribution shown in histogram
$ 9 visualizations: learning curves, entropy, PPO losses + clip fraction, action probabilities, GAE advantages, summary table
$ No stable-baselines3 or rllib: the whole pipeline in pure PyTorch on Gymnasium
# Gallery
# Technologies used
