Fine-Grained Reward Method

From GM-RKB
(Redirected from Fine-Grained Rewarding)
Jump to navigation Jump to search

A Fine-Grained Reward Method is a model training methodology that utilizes detailed feedback to refine the learning process of machine learning models, particularly LLMs.

  • Context:
    • It can (typically) target individual components of a task (offering precise guidance to improve performance).
    • It can (often) involves analyzing outputs at a granular level, such as sentence or paragraph, to provide context-specific feedback.
    • It can be applied to various domains such as natural language processing, computer vision, and reinforcement learning to enhance model accuracy and reliability.
    • It can improve the quality of model-generated content, including the generation of relevant citations and reducing hallucinations.
    • It can requires the development of sophisticated reward functions that can accurately measure the quality of complex outputs.
    • It can lead to models that better understand and replicate human-like decision-making and content creation processes.
    • ...

See: Reinforcement Learning, Natural Language Processing, Machine Learning Model Evaluation.



References

2024

2024

  • (Wu et al., 2024) ⇒ Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. (2024). “Fine-grained Human Feedback Gives Better Rewards for Language Model Training.” Advances in Neural Information Processing Systems 36
    • ABSTRACT: Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF)---where human preference judgments on LM outputs are transformed into a learning signal---has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with this reward function leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.