BERT Embedding System

A BERT Embedding System is a Deep Contextualized Word Representation System that is based on a BERT System.

• Example(s):
  • SensEmBERT System,
  • …
• Counter-Example(s):
  • ARES System,
  • ELMo System (Peters et al., 2018),
  • fastText System (Bojanowski et al., 2017),
  • GloVe System (Pennington et al., 2014),
  • MIMICK System (Pinter et al., 2017),
  • Polyglot System (Al-Rfou et al., 2013),
  • SENNA System (Collobert & Weston, 2008),
  • Word2Vec System (Mikolov et al., 2014).
• See: NLP System, Subword Embedding System, OOV Word, Deep Bidirectional Language Model, Bidirectional LSTM, CRF Training Task, Bidirectional RNN, Word Embedding, 1 Billion Word Benchmark.

---

---

References

2019

• (Devlin et al., 2019) ⇒ Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. (2019). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), Volume 1 (Long and Short Papers). DOI:10.18653/v1/N19-1423. arXiv:1810.04805
  • QUOTE: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers (...)

Figure 3: Differences in pre-training model architectures. BERT uses a bidirectional Transformer. OpenAI GPT uses a left-to-right Transformer. ELMo uses the concatenation of independently trained left-to-right and rightto-left LSTM to generate features for downstream tasks. Among three, only BERT representations are jointly conditioned on both left and right context in all layers. In addition to the architecture differences, BERT and OpenAI GPT are fine-tuning approaches, while ELMo is a feature-based approach..

---