Document Details

Clip: The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization Shengyi Huang Michael NoukhovitchArian Hosseini Kashif Rasul Weixun Wang Lewis TunstallHugging FaceMila, Université de Montréal Fuxi AI Lab, NetEase costa@huggingface.co Abstract This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF)scaling behaviorsreported in OpenAI’s seminal TL;DR summa- rization work (Stiennon et al., 2020). We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in re- sponse quality that scale with model size with our 2.8B, 6.9B models outperforming OpenAI’s released 1.3B checkpoint. We publicly release the trained model check-

Filename: 2403.17031v1.pdf

Notes: