2022 EfficientFewShotLearningWithout
- (Tunstall et al., 2022) ⇒ Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. (2022). “Efficient Few-Shot Learning Without Prompts.” In: Proceedings of The second version of the Efficient Natural Language and Speech Processing (ENLSP-II) workshop .
Subject Headings: setfit, Neural Text Classification Algorithm, Parameter-Efficient Fine-Tuning (PEFT), Pattern Exploiting Training (PET).
Notes
- It proposes SetFit: a new framework for few-shot text classification that does not require prompts or large pretrained language models.
- It works in two steps: first fine-tuning a Sentence Transformer (ST) on a small number of text pairs in a Siamese manner, then training a classification head on embeddings from the fine-tuned ST.
- Experiments show SetFit achieves comparable performance to state-of-the-art few-shot methods like PEFT and PET, while being an order of magnitude faster and not needing prompts.
- It tests against these text classification tasks:
- Sentiment analysis, including SST-2SST-2, IMDBIMDB, Amazon ReviewsAmazon Reviews, and Customer ReviewsCustomer Reviews datasets, involves classifying text into positive or negative sentiment (2 classes).
- Topic classification, including AG NewsAG's news dataset (4 classes) and BBC NewsBBC News (dataset) (5 classes) datasets, aims to categorize documents into topics.
- Spam detection, using the Enron SpamEnron spam data set dataset (2 classes), detects whether emails are spam or not.
- Emotion detection, with the Emotion Twitter datasetEmotion (dataset) (6 classes), categorizes emotional content in text.
- Counterfactual statement detection, using the Amazon CounterfactualAmazon Counterfactual Corpus dataset (2 classes), identifies whether a statement describes something counterfactual.
- Question subject classification, using the Student Question CategoriesStudent Question Dataset dataset (4 classes), categorizes questions into school subjects.
- Toxicity detection, with the Toxic ConversationsToxic Conversations Dataset dataset (2 classes), aims to identify toxic or non-toxic online content.
Cited By
Quotes
Abstract
Recent few-shot methods, such as parameter-efficient fine-tuning (PEFT) and pattern exploiting training (PET), have achieved impressive results in label-scarce settings. However, they are difficult to employ since they are subject to high variability from manually crafted prompts, and typically require billion-parameter language models to achieve high accuracy. To address these shortcomings, we propose SetFit (Sentence Transformer Fine-tuning), an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers (ST). SetFit works by first fine-tuning a pretrained ST on a small number of text pairs, in a contrastive Siamese manner. The resulting model is then used to generate rich text embeddings, which are used to train a classification head. This simple framework requires no prompts or verbalizers, and achieves high accuracy with orders of magnitude less parameters than existing techniques. Our experiments show that SetFit obtains comparable results with PEFT and PET techniques, while being an order of magnitude faster to train. We also show that SetFit can be applied in multilingual settings by simply switching the ST body. Our code is available at this https URL and our datasets at this https URL.
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2022 EfficientFewShotLearningWithout | Nils Reimers Lewis Tunstall Unso Eun Seo Jo Luke Bates Daniel Korat Moshe Wasserblat Oren Pereg | Efficient Few-Shot Learning Without Prompts | 2022 |