Advancing multimodal emotion analysis: a hybrid deep learning approach with intermediate fusion and multi-task learning
Küçük Resim Yok
Tarih
2026
Dergi Başlığı
Dergi ISSN
Cilt Başlığı
Yayıncı
Springer
Erişim Hakkı
info:eu-repo/semantics/openAccess
Özet
Emotion analysis is a critical research domain focused on detecting the emotional states of individuals or communities across multiple data modalities, including text, images, and audio. While substantial progress has been made in unimodal (text-based) sentiment analysis, real-world scenarios often involve multimodal data, making integrated approaches essential for capturing contextual richness and improving predictive accuracy. This study introduces a hybrid deep learning model that combines text and visual features through an intermediate fusion mechanism and multi-task learning framework. Textual inputs are processed using RoBERTa and BiGRU layers, while visual inputs are analyzed through ViT and ResNet50 architectures enhanced by the Convolutional Block Attention Module (CBAM). The fused multimodal representations enable simultaneous and more robust emotion classification. Experimental results on the MVSA dataset demonstrate the superior performance of the proposed model, achieving 96.02% accuracy, 95.51% precision, 94.07% recall, and 94.73% F1-score, outperforming several state-of-the-art multimodal benchmarks. These findings underscore the model's methodological contributions and its strong potential for advancing the field of multimodal emotion analysis in both academic research and real-world applications.
Açıklama
Anahtar Kelimeler
Multimodal Emotion Analysis, Intermediate Fusion, Multi-Task Learning, Vision Transformer, Sentiment Classification
Kaynak
Visual Computer
WoS Q Değeri
Q2
Scopus Q Değeri
Q1
Cilt
42
Sayı
6
