2019 MultimodalTransformerforUnalign

From GM-RKB
Jump to navigation Jump to search

Subject Headings: Multimodal Model.

Notes

Cited By

Quotes

Abstract

Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise cross-modal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.

Introduction

...

Figure 1: Example video clip from movie reviews. [Top]: Illustration of word-level alignment where video and audio features are averaged across the time interval of each spoken word. [Bottom] Illustration of crossmodal attention weights between text (“spectacle”) and vision/audio.

References

;

 AuthorvolumeDate ValuetitletypejournaltitleUrldoinoteyear
2019 MultimodalTransformerforUnalignLouis-Philippe Morency
Ruslan Salakhutdinov
Yao-Hung Hubert Tsai
Shaojie Bai
Paul Pu Liang
J Zico Kolter
Multimodal Transformer for Unaligned Multimodal Language Sequences10.18653/v1/p19-16562019