RL post-training: alternate generating and learning

Picture it

Picture a debate team practicing. First, they generate many possible arguments, then a judge scores them, and finally, they update their strategy. Generating arguments requires quick thinking, while updating strategy requires deep reflection. If the judge is too slow, the debaters sit idle; if the strategy updates too slowly, they keep practicing bad habits.

What to understand

• Rollout engines, reward/verifier workers, learner, and weight transfer
• Synchronous versus asynchronous updates, policy lag, and capacity sharing

Follow the references

1. 1 · Course + videos

   Stanford · CS336 Language Modeling from Scratch (2025) ↗

   Use the reinforcement learning lectures to identify rollout and update work.

2. 2 · Documentation

   SGLang · Serving documentation ↗

   Follow its reinforcement-learning integrations and locate the inference engine’s role.

Make it concrete

Sketch resource pools for rollout, reward, and learning. Measure or estimate each stage’s service rate and identify where a queue will grow.