Evaluating and Improving Generative Models with Feedback

Abstract

Generative models can produce many candidate outputs for a request, but deciding which output is good, and what a model should learn next, requires feedback, a judgment of how well an output serves its task. This thesis studies how to obtain such feedback and how to use it, for evaluating generated outputs and for improving the models that produce them. The two uses place different demands on feedback. For evaluation, feedback has to agree with human judgment and explain its judgment. For training, it has to stay reliable while a model is optimized against it.

The first part studies feedback for evaluation. For text, I show that feedback can be learned without a reference output, by training an evaluator on error analyses so that it locates and explains each error in a response, and its scores agree with human ratings nearly as well as GPT-4 used as a judge. For images and videos, I collect pairwise human votes on an open platform, use them to rank generative models, and release them as a benchmark on which the best multimodal judge tested agrees with people about half the time, so human preferences remain the reference for this content.

The second part studies feedback for improving models, where agreement with people on fixed outputs is not enough. A policy trained against the learned text evaluator learned to refuse rather than answer, raising its reward while its accuracy fell. For code, I synthesize tests for tens of thousands of programming problems and turn execution results into preferences and rewards, which improve both Best-of-N selection and reinforcement learning. For agents that use tools over several turns, I build a reinforcement-learning framework that runs tools inside the training loop, train agents in six domains, and examine two choices in the feedback they receive. Rewarding tool use raises tool calls without necessarily raising task performance, and how much of a tool's output the policy reads can change the result.

Across these studies, feedback that evaluates fixed outputs well can behave differently once a model is trained against it, so a training reward needs a measure it does not see, tracked while the model is optimized.