Reward Modeling: Training AI from Human Preferences

Seen as the clearest introduction to the idea behind RLHF, and also borrowed by commenters as a metaphor for human motivation.

Robert Miles explains reward modelling, in which an AI learns what humans want from their preference comparisons instead of a hand-written reward function. He walks through the 2017 paper on deep reinforcement learning from human preferences.