SERIES: Learning RL and JAX in Public - from zero to DeepMind. Day 6 ended with REINFORCE...
Article explains Actor-Critic: a Critic network estimates state value, providing the Actor an advantage signal (return - estimate) that dramatically reduces REINFORCE noise. This architecture underpins PPO (ChatGPT), GRPO (DeepSeek-R1), A3C (DeepMind); essential for understanding modern LLM and RL training.
SERIES: Learning RL and JAX in Public - from zero to DeepMind.
Day 6 ended with REINFORCE working. The agent learned to navigate the gridworld using probabilities instead of Q-values. Clean idea, clean code.
But it was noisy. Training was slow. Some episodes the agent got lucky, some unlucky, and the network kept getting confused by the difference. The signal was there but it was buried in too much variance.
Day 7 is the fix.
The basketball analogy that made it click
SERIES: Learning RL and JAX in Public - from zero to DeepMind :) On Day 1 I covered what JAX...
Been heads-down lately, but I finally had a couple of days to catch up on some recent papers. Sharing...
I have a small cooperative multi-agent reinforcement-learning setup: eight agents on a 100×100 grid,...
Introduction This is a follow-up to my previous article. If you haven't read it yet, check...
It was 3 AM, and I was staring at a terminal window filled with telemetry data from a simulated Mars rover. The reinforcement…
Why trajectory-level reward is a terrible teacher for multi-step agents - and how a 2026 paper called SDAR proposes to fix it.…
