2015 ImprovedSemanticRepresentations

Subject Headings: Long Short-Term Memory (LSTM) Network, Tree-LSTM, Tree-RNNs.

Notes

(Kiros et al., 2015) ⇒ Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. (2015). “Skip-thought Vectors.” In: Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS-2015).
- QUOTE: … Given the difficulty of this task, many existing systems employ a large amount of feature engineering and additional resources. Thus, we test how well our learned representations fair against heavily engineered pipelines. Recently, (Tai et al., 2015) showed that learning representations with LSTM or Tree-LSTM for the task at hand is able to outperform these existing systems. We take this one step further and see how well our vectors learned from a completely different task are able to capture semantic relatedness when only a linear model is used on top to predict scores. To represent a sentence pair, we use two features. Given two skip-thought vectors u and v, we compute their component-wise product u � v and their absolute difference ju 􀀀 vj and concatenate them together. These two features were also used by (Tai et al., 2015). To predict a score, we use the same setup as (Tai et al., 2015). …

Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2015 ImprovedSemanticRepresentations	Christopher D. Manning Raquel Urtasun Richard Socher Ruslan Salakhutdinov Kai Sheng Tai Richard S. Zemel Ryan Kiros Yukun Zhu Antonio Torralba Sanja Fidler			Improved Semantic Representations from Tree-structured Long Short-term Memory Networks