Monday Meeting

The paper formulates self-play as a teacher-student pair (generator and learner). The learner is pretrained via next-token prediction to model lessons (sequences output from teacher-written randomized programs). The teacher is trained via RL to align the parameter gradients induced for a learner by a lesson with the global parameter movement from the most recent half of the learner’s overall trajectory. This is a shockingly good formalization of one-on-one tutoring in RL terms.