Large Deep Neural Network
(Redirected from Large Deep NNet)
Jump to navigation
Jump to search
A Large Deep Neural Network is a deep neural network with over one billion NNet parameters.
- Example(s):
- a Large Deep Neural Language Model, such as Turing-NLG.
- See: Small DNN.
References
2019
- (Rajbhandari et al., 2019) ⇒ Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. (2019). “ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.” In: arXiv preprint arXiv:1910.02054.
- QUOTE: ... Training large DL models with billions and potentially trillions of parameters is challenging. Existing solutions exhibit fundamental limitations to obtain both memory and scaling (computation / communication) efficiency together. Data parallelism does not help reduce memory footprint per device: a model with 1.5 billion parameters or more runs out of memory. …