Deep Contextual Word Representation System

From GM-RKB
(Redirected from embeddings based on context)
Jump to navigation Jump to search

A Deep Contextual Word Representation System is a Deep Learning System that is a Word Embedding System that can produce contextual word vector.



References

2021

2019

2018 BERTPreTrainingofDeepBidirectio Fig1.png
Figure 3: Differences in pre-training model architectures. BERT uses a bidirectional Transformer. OpenAI GPT uses a left-to-right Transformer. ELMo uses the concatenation of independently trained left-to-right and rightto-left LSTM to generate features for downstream tasks. Among three, only BERT representations are jointly conditioned on both left and right context in all layers. In addition to the architecture differences, BERT and OpenAI GPT are fine-tuning approaches, while ELMo is a feature-based approach..