Large Neural Language Model (LLM)

A Large Neural Language Model (LLM) is a neural language model that is a large neural model.

• Context:
  • Model Input: LLM Input.
  • Model Output: LLM Output.
  • It can (typically) have over 10M LLM parameters.
  • It can (typically) reference an LLM Training Dataset.
  • It can (typically) reference an LLM Architecture, such as a GPT architecture.
  • It can (often) be a Transformer-based LLM (such as a decode-only LLM).
  • It can range from being a Base Pretrained LLM to being a Finetuned LLM.
  • It can range from being a All-Domain LLM to being a Domain-Specific LLM.
  • It can range from being a Short-Context LLM (<=16K) to being a Long-Context LLM (>16K), depending on its LM context length.
  • It can range from being a Closed-Source LLM to being an Open-Source LLM.
  • It can range from being a Unilingual LLM to being a Multilingual LLM.
  • It can range from being a Decoder-based LLM to being a Encoder-based LLM to being a Decoder-Encoder-based LLM.
  • It can range from being a Historical LLM (such as GPT-2) to being a Current LLM (such as GPT-4) to being a Future LLM.
  • It can be an input to an LLM Inference Task.
  • It can be used by an LLM-based System (solving an LLM-based task).
  • …
• Example(s):
  • a Made-in-USA LLM, such as:
    • an OpenAI LLM, such as: GPT-2, GPT-3, GPT-3.5, GPT-4.
    • an Anthropic Claude LLM (2022): Made-in-USA LLM by Anthropic. Understands conversation context.
    • a Google LLM, such as: PaLM LLM, LaMDA LLM, Gemini LLM.
  • GPT-J LLM (2021): Made-in-UK LLM by EleutherAI with 6 billion parameters. For fine-tuning.
  • Wu Dao 2.0 LLM (2022): Made-in-China LLM by Beijing Academy of AI. 1.75 trillion parameter multimodal model.
  • BLOOM LLM (????): Made-in-Israel LLM by Bar-Ilan University. Understands conversation context.
  • HyperClova LLM (????): Made-in-South Korea LLM by Naver Corp. Understands conversation.
  • Fugaku LLM (~2023): Made-in-Japan LLM by RIKEN. 1.4 trillion parameters. For fine-tuning.
  • PaLM-E.
  • HuggingFace LLM, such as: BLOOM, T5 LLM, GPT-Neo, BART, LLaMA.
  • Google LLM, such as: LaMDA. PaLM, Gemini LLM.
  • Meta LLM, such as: LLaMA.
  • Gopher LM.
  • RoBERTa.
  • GPT-J: GPT-J-6B.
  • BERT Model.
  • a Software Code LLM, such as Codex LLM.
  • ...
• Counter-Example(s):
  • a Shallow Neural Language Model.
• See: Multi-Lingual Neural Network-based Language Model (NLM), LLM-based Task.

---

---

References

2023

• (Wikipedia, 2023) ⇒ https://en.wikipedia.org/wiki/Large_language_model#List Retrieved:2023-3-19.

List

Name	Release dateTemplate:Efn	Developer	Number of parametersTemplate:Efn	Corpus size	Training cost (petaFLOP-day)	LicenseTemplate:Efn	Notes
BERT	Template:Dts	Google	Template:Sort	Template:Sort words	Template:Sort	Apache 2.0	An early and influential language model, but encoder-only and thus not built to be prompted or generative
XLNet	Template:Dts	Google	Template:Sort	Template:Sort words			An alternative to BERT; designed as encoder-only
GPT-2	Template:Dts	OpenAI	Template:Sort	40GB (~Template:Sort tokens)		MIT	general-purpose model based on transformer architecture
GPT-3	Template:Dts	OpenAI	Template:Sort	Template:Sort tokens	3640	proprietary	A fine-tuned variant of GPT-3, termed GPT-3.5, was made available to the public through a web interface called ChatGPT in 2022.
GPT-Neo	Template:Dts	EleutherAI	Template:Sort	825 GiB		MIT	The first of a series of free GPT-3 alternatives released by EleutherAI. GPT-Neo outperformed an equivalent-size GPT-3 model on some benchmarks, but was significantly worse than the largest GPT-3.
GPT-J	Template:Dts	EleutherAI	Template:Sort	825 GiB	200	Apache 2.0	GPT-3-style language model
Megatron-Turing NLG	Template:Dts	Microsoft and Nvidia	Template:Sort	Template:Sort tokens		Restricted web access	Standard architecture but trained on a supercomputing cluster.
Ernie 3.0 Titan	Template:Dts	Baidu	Template:Sort	4 Tb		Proprietary	Chinese-language LLM. Ernie Bot is based on this model.
Claude	Template:Dts	Anthropic	Template:Sort	Template:Sort tokens		Template:Partial success	Fine-tuned for desirable behavior in conversations.
GLaM (Generalist Language Model)	Template:Dts	Google	Template:Sort	Template:Sort tokens	5600	Proprietary	Sparse mixture-of-experts model, making it more expensive to train but cheaper to run inference compared to GPT-3.
Gopher	Template:Dts	DeepMind	Template:Sort	Template:Sort tokens	5833	Proprietary
LaMDA (Language Models for Dialog Applications)	Template:Dts	Google	Template:Sort	1.56T words, Template:Sort tokens	4110	Proprietary	Specialized for response generation in conversations.
GPT-NeoX	Template:Dts	EleutherAI	Template:Sort	825 GiB	740	Apache 2.0	based on the Megatron architecture
Chinchilla	Template:Dts	DeepMind	Template:Sort	Template:Sort tokens	6805	Proprietary	Reduced-parameter model trained on more data. Used in the Sparrow bot.
PaLM (Pathways Language Model)	Template:Dts	Google	Template:Sort	Template:Sort tokens	29250	Proprietary	aimed to reach the practical limits of model scale
OPT (Open Pretrained Transformer)	Template:Dts	Meta	Template:Sort	Template:Sort tokens	310	Template:Partial successTemplate:Efn	GPT-3 architecture with some adaptations from Megatron
YaLM 100B	Template:Dts	Yandex	Template:Sort	1.7TB		Apache 2.0	English-Russian model based on Microsoft's Megatron-LM.
Minerva	Template:Dts	Google	Template:Sort	38.5B tokens from webpages filtered for mathematical content and from papers submitted to the arXiv preprint server		Proprietary	LLM trained for solving "mathematical and scientific questions using step-by-step reasoning". Minerva is based on PaLM model, further trained on mathematical and scientific data.
BLOOM	Template:Dts	Large collaboration led by Hugging Face	Template:Sort	Template:Sort tokens (1.6TB)		Responsible AI	Essentially GPT-3 but trained on a multi-lingual corpus (30% English excluding programming languages)
Galactica	Template:Dts	Meta	Template:Sort	Template:Sort tokens	unknown	Template:Partial success	Trained on scientific text and modalities.
AlexaTM (Teacher Models)	Template:Dts	Amazon	Template:Sort	Template:Sort		proprietary	bidirectional sequence-to-sequence architecture
LLaMA (Large Language Model Meta AI)	Template:Dts	Meta	Template:Sort	Template:Sort	6300	Template:Partial successTemplate:Efn	Trained on a large 20-language corpus to aim for better performance with fewer parameters. Researchers from Stanford University trained a fine-tuned model based on LLaMA weights, called Alpaca.
GPT-4	Template:Dts	OpenAI	Exact number unknownTemplate:Efn	Unknown	Unknown	proprietary	Available for ChatGPT Plus users and used in several products.
Cerebras-GPT	Template:Dts	Cerebras	Template:Sort		270	Apache 2.0	Trained with Chinchilla formula.
Falcon	Template:Dts	Technology Innovation Institute	Template:Sort	1 trillion tokens, from RefinedWeb (filtered web text corpus) plus some "curated corpora".	2800	Apache 2.0	Training cost around 2700 petaFLOP-days, 75% that of GPT-3.
BloombergGPT	Template:Dts	Bloomberg L.P.	Template:Sort	363 billion token dataset based on Bloomberg's data sources, plus 345 billion tokens from general purpose datasets		Proprietary	LLM trained on financial data from proprietary sources, that "outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks"
PanGu-Σ	Template:Dts	Huawei	Template:Sort	329 billion tokens		Proprietary
OpenAssistant	Template:Dts	LAION	Template:Sort	1.5 trillion tokens		Apache 2.0	Trained on crowdsourced open data
Jurassic-2	Template:Dts	AI21 Labs	Exact size unknown	Unknown		Proprietary	Multilingual
PaLM 2 (Pathways Language Model 2)	Template:Dts	Google	Template:Sort	Template:Sort tokens	85000	Proprietary	Used in Bard chatbot.
Llama 2	Template:Dts	Meta	Template:Sort	Template:Sort tokens		Template:Partial success	Successor of LLaMA.

2022

• (Zhou et al., 2022) ⇒ Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. (2022). “Large Language Models Are Human-level Prompt Engineers.” In: arXiv preprint arXiv:2211.01910.
  • QUOTE: ... By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. ...

2020

• (Liu et al., 2020) ⇒ Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. (2020). “Adversarial Training for Large Neural Language Models.” arXiv preprint arXiv:2004.08994
  • QUOTE: … Pre-training a large neural language model such as BERT has proven effective to improve generalization performance in task-specific fine-tuning (Devlin et al.…

---