Hybrid CNN-LSTM Framework-based Facial Emotion Recognition for Retail Decision Analytics

Abstract

This facial emotion recognition (FER) framework, which is the basic system that integrates technologies and processes related to facial emotion recognition, will be used in a relevant research project—we will explore whether facial emotion recognition, the technology that judges current emotions based on facial expressions, can become a component of retail decision analysis tools that perceive emotions. If customers' emotions can be accurately identified, future retail decision support applications will gain an additional piece of reference information reflecting customers' behaviours. Most of the previous CNN-based FER systems focus on static frames for classification without considering the temporal evolution of emotions. In this paper, three deep learning models—CNN, Mini-Xception, and CNN–LSTM are compared in the context of facial emotion recognition for retail decision analytics. CNN and Mini-Xception are spatial methods, while CNN–LSTM combines spatial and temporal information. CNN models are pre-trained on the FER-2013 database (35,887 images) and CNN-LSTM is tested on the CK+ dataset (981 sequences). In the experimental environment of CK+, the CNN-LSTM model processes temporally ordered expression sequences and achieves an accuracy rate of 92.42%. This result proves that under this experimental setup of CK+, the method of modelling sequential time-series data is effective. As for the application of this framework to support retail decision analysis, it can only be regarded as a potential expansion direction at present—all relevant research on it is completed based on existing, reference benchmark facial expression datasets, and has never used transaction-related data generated in real retail scenarios, so it cannot be implemented in actual retail business for the time being.