Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References 3 Background 3.1 Reward model training Reward models are trained with human preference data most often consisting of a task given to the model prompt, i.e a request or instruction, and ratings of the completion, or answer. The feedback can consist of selecting the best from groups of responses (Ziegler et al., 2019), scores and rankings of a group of candidate responses (Ouyang et al., 2022), a choice between a pair of responses (Bai et al., 2022) (choose best response between two options), and even finer grained data (Z. Wu et al., 2023). The workers employed are generally given detailed instructions on which styles, occurrences, or values to prioritize in their labels. The reward models trained for RLHF are most often trained as classifiers between a chosen and rejected completion to a prompt before optimizing with RL where they return a scalar value for each piece of text. Given two options for a completion y from a prompt x, and the scores they obtain a scalar output r from a initially untrained value head on an LLM or value model entirely, the loss for the reward model follows (Askell et al., 2021; Ouyang et al., 2022) 3.2 Reinforcement Learning on Language With LLMs, the generating model is referred to the as the policy model. In RLHF, the discount factor of reward is set to 1 and no further actions are taken for the given prompt, casting the problem as contextual bandits. An example of the RLHF loop is shown in Fig. 2b in comparison to a standard RL loop shown in Fig. 2a. This paper is under CC 4.0 license. available on arxiv

The Mechanics of Reward Models in RLHF

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Understanding Objective Mismatch

AI Benchmarks: Why Useless, Personalized Agents Prevail

The New Standard for Program Synthesis: How Tree-Diffusion Outperforms CSGNet and REPL Flow

GPT-4 vs GPT-3.5 Performance in Game Simulations

Testing GPT-4 on Game State Predictions

Understanding Objective Mismatch

AI Benchmarks: Why Useless, Personalized Agents Prevail

The New Standard for Program Synthesis: How Tree-Diffusion Outperforms CSGNet and REPL Flow

GPT-4 vs GPT-3.5 Performance in Game Simulations

Testing GPT-4 on Game State Predictions

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps