2019 MultimodalTransformerforUnalign

(Redirected from Tsai et al., 2019)
Jump to navigation Jump to search

Subject Headings: Multimodal Model.


Cited By



Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.



Figure 1: Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of crossmodal attention weights between text (“spectacle”) and vision/audio.



 AuthorvolumeDate ValuetitletypejournaltitleUrldoinoteyear
2019 MultimodalTransformerforUnalignLouis-Philippe Morency
Ruslan Salakhutdinov
Yao-Hung Hubert Tsai
Shaojie Bai
Paul Pu Liang
J Zico Kolter
Multimodal Transformer for Unaligned Multimodal Language Sequences10.18653/v1/p19-1656