2016 CharacterAwareNeuralLanguageMod

• (Kim et al., 2016) ⇒ Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. (2016). “Character-Aware Neural Language Models.” In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-2016).

Subject Headings: Neural Language Modeling, Recurrent Neural Network Language Model, Character-level Convolutional Neural Network, Highway Network.

Cited By

• Google Scholar: ~ 1,060 Citations.

2020

• (Melli et al., 2020) ⇒ Gabor Melli, Abdelrhman Eldallal, Bassim Lazem, and Olga Moreira. (2020). “GM-RKB WikiText Error Correction Task and Baselines.”. In: Proceedings of LREC 2020 (LREC-2020).

2017

• (Bojanowski et al., 2017) ⇒ Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. (2017). “Enriching Word Vectors with Subword Information.” In: Transactions of the Association for Computational Linguistics 5
  • QUOTE: ... Another family of models are convolutional neural networks trained on characters, which were applied to part-of-speech tagging (dos Santos and Zadrozny, 2014), sentiment analysis (dos Santos and Gatti, 2014), text classification (Zhang et al., 2015) and language modeling (Kim et al., 2016). …

Quotes

Abstract

We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with [[rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level / morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2016 CharacterAwareNeuralLanguageMod	David Sontag Alexander M. Rush Yoon Kim Yacine Jernite			Character-Aware Neural Language Models