Emotion-aware human computer interaction is the natural and flexible Human Computer Interaction (HCI) to perceive and consider human emotion responses in the computation. Few recent successes in deep learning have resulted in an increased emotion recognition accuracy but, mostly the emotion-aware HCI systems are limited to poor robustness in practical scenarios and lack of interpretability at the decision level, thus limiting their adoption for practical applications. This work defines the problem of robust and transparent emotion recognition by designing an explainable modality-adaptive multimodal deep learning framework, capable of fusing multiple input modalities such as facial expressions, speech and physiological responses. The emotion perception framework employs a set of deep neural models to perform spatial–temporal feature extraction and introduces a set of neural architecture designs, namely, Modality-Adaptive Attention Fusion Module (MAAFM) mechanism to adaptively fuse the modalities based on modality qualities and contextual information. Further, for interpretability, the knowledge at the modality level and at the fusion level is extracted through various shallow modules at the feature- and the fusion-level respectively, which provides transparency towards how each modality affects the inference. The proposed framework is evaluated over benchmark datasets for visual, audio and physiological modalities and reports an overall emotion recognition accuracy of 92.4 %, F1-score of 0.91, and area under the receiver operating characteristic curve of 0.97, while preserving a low inference latency for interactive applications. The approach provides improved robustness, transparency and applicability. The key innovation of this work is the focus on the attention not only as a performance tool but also as an explicit source of interpretation of the modality contributions at the fusion level that advances existing emotion recognition approaches.