Turing-NLG Deep NNet-based Language Model
(Redirected from Turing-NLG)
Jump to navigation
Jump to search
A Turing-NLG Deep NNet-based Language Model is a deep NNet LM that ...
- See: Large Deep NNet, ZeRO Library.
References
2019
- https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/
- QUOTE: ... DeepSpeed is compatible with PyTorch. One piece of that library, called ZeRO, is a new parallelized optimizer that greatly reduces the resources needed for model and data parallelism while massively increasing the number of parameters that can be trained. Researchers have used these breakthroughs to create Turing Natural Language Generation (Turing-NLG), the largest publicly known language model at 17 billion parameters, which you can learn more about in this accompanying blog post. ...