Difference between revisions of "OpenAI GPT-2 System"

From GM-RKB
Jump to: navigation, search
(Created page with "An OpenAI GPT-2 System is an transformer-based language modeling system developed by OpenAI. * <B>See:</B> OpenAI GPT. ---- ---- == References == === 2019 ==...")
 
(No difference)

Latest revision as of 05:00, 14 August 2019

An OpenAI GPT-2 System is an transformer-based language modeling system developed by OpenAI.



References

2019

  • https://openai.com/blog/better-language-models/
    • QUOTE: Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper.

      GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset of 8 million web pages (the dataset which emphasizes diversity of content, by scraping content from the Internet. In order to preserve document quality, we used only pages which have been curated/filtered by humans—specifically, we used outbound links from Reddit which received at least 3 karma. This can be thought of as a heuristic indicator for whether other users found the link interesting (whether educational or funny), leading to higher data quality than other similar datasets, such as CommonCrawl.). ... GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than 10X the amount of data.