Artificially Intelligent Interviewer – A Multimodal Approach

This paper presents an innovative system designed to automate the analysis of candidate interviews by inte-grating multiple analytical techniques into a single multimodal framework. This system combines text sentiment analysis, audio sentiment analysis, keyword extraction, and Mel-Frequency Cepstral Coefficients (MFCC) feature extraction to evaluate candidate performance holistically. This system employs text sentiment analysis using VADER and transformer-based sentiment features (probability-based outputs), audio sentiment analysis with an SVM model trained on both IEMOCAP and MELD datasets, keyword extraction via KeyBERT, and audio feature extraction including MFCCs, delta MFCCs, pitch, and energy to evaluate candidate performance holistically. A novel weighted scoring mechanism, incorporating personality traits such as Neuroticism, dis-tinguishes this project from existing single-modality systems, offering a comprehensive candidate assessment. Results highlight its potential for Human Resources (HR) applications, with future improvements planned to enhance Negative sentiment detection through hybrid approaches and advanced emotion recognition tech-niques.