user@devops:~$ cat README.md
World Models / Model-Based RL -- planning with the model from scratch
# Description
World Models / Model-Based RL implemented from scratch in PyTorch (Dreamer-lite style). Part 1: the agent collects 4,000 transitions (s, a, s') with a random policy over an 8-bit binary key (256 states); reward 1 if the state ends in 1 (random mean reward 0.4943). Part 2: train the world model -- MLP 9->64->64->8 predicting the 8 bits of the next state with multi-label BCE (not a 256-way class); 5,320 params, val_acc 1.0000 by ~100 epochs, 2 s CPU. Part 3: plan with the model using CEM: sample 80 Bernoulli action sequences, roll them out imaginarily, score with discounted reward, keep the 20% elite, re-fit mu, repeat 12x; execute the 1st action of the best plan in the real env. Part 4: closed-loop MPC at H=3/6/9 -- mean reward 1.0000 at every horizon, reaching the optimal policy with no extra env steps. Part 5: model-free Q-Learning tabular baseline (20,000 steps, eps-decay) reaches success 0.9302. Finding: the model-based advantage is sample efficiency -- 4,000 transitions build a perfect model and plan the optimum, while model-free spends 5x more interaction to fall short (this is what made Dreamer/PlaNet competitive). Trade-off: model error accumulates over long horizons and stochastic envs. Seed 42, CPU ~73 s, deterministic checkpoint. 7 visualizations.
# Key features
$ World model from scratch: MLP 9-64-64-8 predicts s' as 8 bits (multi-label BCE), 5,320 params, val_acc 1.0
$ Random rollout dataset: 4,000 transitions (s, a, s') over an 8-bit binary key
$ Model-based planning with CEM (Cross-Entropy Method): 80 samples, 20% elite, 12 iters, H=3/6/9
$ MPC imagines the future with the model and executes the 1st action: mean reward 1.0000 with no trial-and-error
$ Model-free Q-Learning tabular baseline (20,000 steps): success rate 0.9302
$ vs random policy 0.4943: planning with the model reaches the optimal policy
$ Finding: sample efficiency -- 4,000 transitions suffice to plan the optimum
$ 7 visualizations: coverage, curves, horizon, trajectories, comparison, efficiency, summary
# Gallery
# Technologies used
