 Original Article
 Open Access
 Published:
Robust facial expression recognition system in higher poses
Visual Computing for Industry, Biomedicine, and Art volume 5, Article number: 14 (2022)
Abstract
Facial expression recognition (FER) has numerous applications in computer security, neuroscience, psychology, and engineering. Owing to its nonintrusiveness, it is considered a useful technology for combating crime. However, FER is plagued with several challenges, the most serious of which is its poor prediction accuracy in severe head poses. The aim of this study, therefore, is to improve the recognition accuracy in severe head poses by proposing a robust 3D headtracking algorithm based on an ellipsoidal model, advanced ensemble of AdaBoost, and saturated vector machine (SVM). The FER features are tracked from one frame to the next using the ellipsoidal tracking model, and the visible expressive facial key points are extracted using Gabor filters. The ensemble algorithm (AdaAdaSVM) is then used for feature selection and classification. The proposed technique is evaluated using the Bosphorus, BU3DFE, MMI, CK + , and BP4DSpontaneous facial expression databases. The overall performance is outstanding.
Introduction
Applications
Facial expression recognition (FER) is the automatic detection of the emotional state of a human face using computerbased technology. The field of study is currently a hotspot of research because it has increasing applications in several domains, such as psychology, sociology, health science, transportation, gaming, communication, security, and business. According to Panksepp [1], facial expressions and emotions guide the lives of people in a variety of ways, and emotions are key aspects that enlighten us in how we should act, from elementary processes to the most intricate acts [2, 3].
The sporadic advancements in the use of facial expressions in neuropsychiatric complications have shown more positive results [4], and current studies are focusing on human behavior and the detection of mental illnesses [5, 6].
FER can also affect data collection in specific research projects. For example, Shergill et al. [7] proposed an intelligent assistant FER framework that could be implemented in ecommerce to determine the product preferences of customers. The system captures the facial data as they browse the eshop for products to acquire. Based on the facial expression, the systems can automatically suggest more products of possible interest.
Certain physiological features of people have been discovered to be useful as intelligent data in the search for criminals [8, 9]. This theory is based on the tendency for someone with ego to commit a highprofile crime, such as terrorism, exhibits specific emotions such as anger and fear. Consequently, the accurate recognition of these expressions could lead to further security measures in apprehending criminals.
FER can also be valuable during the testing phase of video games. Target groups are frequently invited to play a game for a set amount of time, and their behaviors and emotions are observed as they play. Game developers may acquire more insights and valuable deductions about the emotions recorded during gameplay using FER technology, and incorporate the feedback into production.
Technical issues on the use of twodimensional facial data
Twodimensional (2D) FER systems are extremely sensitive to head orientation. Therefore, to achieve good results, the subject must be constantly in a frontoparallel orientation. The problem resulting from this is that the throughput of most siteaccess systems is significantly reduced. This implies that subjects are frequently required to perform several verifications to attain an ideal facial orientation. Consequently, surveillance systems operate on luck, hoping the subject faces the camera.
Another problem that arises from the use of 2D technology is the illumination conditions of the surrounding environment. If the subject is in a setting with varying lighting conditions, FER reduces in accuracy because the FER processes are sensitive to the direction of lighting and the ensuing shading pattern. Consequently, cast shadows may obstruct recognition by concealing informative features.
Threedimensional (3D) FER systems have a higher detection rate than 2D systems because of their higher intensity modality, and they also have more object description geometry information [10, 11]. This demonstrates the importance of pushing FER into higher face orientations to improve its realism and practicality.
Related work
The primary focus of this study is to improve FER accuracy in higher facial orientations.
Yadav and Singha [12] adopted the ViolaJones descriptor [13] to detect faces and used a combination of local binary patterns (LBP) and the histogram of gradients (HOG) as a feature extraction tool. Subsequently, traditional SVM with the kmeans method was employed as a training algorithm. LPB feature extraction techniques, such as Gabor, are orientationselective, and thus, highly robust in tracking key facial features. However, the ViolaJones descriptor is computationally demanding and has a low detection accuracy. Furthermore, the conventional SVM described in the study is slow to classify. Consequently, the overall architecture used in the study was computationally expensive. Yao et al. [14] proposed a linear SVM method that used AUs to recognize seven facial expression prototypes in the CK database. The ViolaJones descriptor was used as the facedetection technique again. Although the goal of the study was to minimize computational complexity and enhance recognition accuracy, the resulting average recognition accuracy of 94.07% for females and 90.77% for males was too low for a viable implementation. Ashir et al. [15] also proposed an SVMbased multiclass classification for detecting seven facial expressions across four prominent databases. The Nyquist–Shannon sampling method [16] was used to compress the extracted facial feature samples. Although the sampling method reduces data loss, it is prone to aliasing issues, particularly when the bandwidth is extremely large. The NyquistShannon sampling technique is difficult to deploy because it assumes the sampled signal is completely bandrestricted. In realworld applications, this is a concern because no actual signal is genuinely and completely bandrestricted. The compressing sampling [17] paradigm could have been a better option because it is less restrictive. PerezGomez et al. [18] recently proposed a 2D–3D FER system that used principal component analysis (PCA) and a genetic algorithm for feature selection, and a knearest neighbor (KNN)multiclass SVM for learning. In this study, the synthetic minority oversampling technique (SMOTE) [19] was used to balance the instances. However, SMOTE creates an equal number of synthetic samples for each minority data sample and relies on the hypothesis performance to update the distribution function. The adaptive synthetic (ADASYN) [20] method tends to generate more synthetic data for minority class samples that are harder to learn than with SMOTE, which is easy to learn. In addition, PCA uses observations from all the extracted features in the projection to the subspace and only considers linear relationships, ignoring the input multivariate structures. Compared to other recent studies, the findings of this study were not positive.
Li et al. [21] proposed a robust 3D local coordinate technique for extracting poseinvariant facial features at key points. The descriptor in this method is a multitask sparse representation finegrained matching algorithm. The method was evaluated using the Bosphorus datasets, and an average recognition accuracy of 98.9% was obtained. The success of this study is largely owed to the accurate tracking of 3D key points. This recent study is a primary driving force behind our proposed study.
The following are the significant contributions of this work: (1) A robust headtracking algorithm that tracks facial features from one frame to the next, accounting for more features in the overall prediction process; (2) A unique ensemble approach that employs AdaBoost for feature selection, and a combination of AdaBoost and SVM for classification. AdaBoost is extremely fast, whereas SVM is extremely accurate. Consequently, the proposed technique becomes extremely fast while also improving the recognition accuracy.
The remainder of this paper is organized as follows. Methods section delves into the proposed strategy. Results and discussion section discusses the findings, debates, and analyses. Finally, Conclusions section concludes the study.
Methods
We robustly tracked the facial features from one frame to the next using 3D facial data. With 3D data, information, such as the size and shape of an object, can be correctly estimated in each frame without prior assumptions.
The first priority is to detect the focal points in each frame. The next step is to search for matching features or objects across all frames. This method addresses the changing behavior of a moving object and the preceding annotations of the scene. In this approach, the location of an object is projected by iteratively updating the object position from previous frames [22, 23].
Architectural framework
Figure 1 presents the framework of this study.
This procedure uploads images and robustly tracks the features across frames using the proposed ellipsoidal model. Subsequently, the Gabor featureextraction approach was used. Feature points extraction section explains the reason for using Gabor features in this study. Feature selection and classification were executed using the AdaAdaSVM.
Ellipsoidal feature tracking method
Accurate tracking of a human face from the forehead, to the left cheek, to the chin, to the right cheek, and back to the same spot on the forehead where the tracking began unmistakably demonstrates that the human face is best shaped like an ellipse. Thus, considering the 3D facial representation in Fig. 2 with N feature points tracked across frames, we denote:
where N represents the most relevant feature points. In this study, we assumed N to be 24. In addition, let \({f}_{j}(t)\in \alpha (t)\) denote a facial feature. As the features move from one frame to the next at time t + 1, the position of feature \({f}_{j}(t)\) becomes \({f}_{j}(t+1)\). Therefore, \({f}_{j}(t+1)\in \alpha (t+1)\). Assuming that \({Y}_{j}\) is the position of \({\alpha }_{j}\) on the 3D facial model and \({\alpha }_{j,p}[\varnothing \left(t+1\right)]\) represents its back projection on the image plane, the 3D facial orientation at t + 1 is the vector \(\varnothing \left(t+1\right)\) that minimizes \({\sum }_{j=1}^{N}{S}_{j}^{2}\), where:
This is a multiview system based on the assumption that cameras are positioned around the subject to capture various rotation movements. Consequently, the facial image can be captured with a high degree of precision in any orientation. We extracted the features in the same manner as for 2D images. The right and left eyes, lips, and muscles around the cheeks are important parts of the face to consider. Slight disruptions primarily and severely distort the muscles in these places. The Gabor technique is then used to extract the features of the captured face.
The algorithm models a procedure that chooses a set of features and robustly tracks them from one frame to the next while discarding all other features that are no longer required for tracking. The ellipsoidal 3D face was modelled, as shown in Fig. 3.
Adopting homogeneous coordinates for an ellipsoid of the semiaxis, a, b, and c, states that a point \({X}_{0}=\left({x}_{0}, {y}_{0}, {z}_{0}, 1\right)\) belongs to the surface of the ellipsoid if \({X}_{0}^{T}{E}_{0}{X}_{0}=0\).
The algorithm tracks the facial features that are more noticeable by slight deformation from one frame to the next using the brightness change constraint [24]. These muscles are usually near the eyes, mouth, cheeks, and edges, as shown in Fig. 4 and contour τ in Fig. 3.
Given that pixel (x, y) with luminance \(I{\left(x,y\right)}^{T}\) moves from position (x, y)^{T} at frame t to position \({\left(x+u, y+v\right)}^{T}\) at frame t + 1 in high frame rates. In this instance, we can deduce that
By applying Taylor’s series, and considering I_{x} and I_{y} as gradients and that I_{t} is a temporal deviation of the image, we can infer that
If a whole window \({\omega }_{k}\) is considered instead of a single pixel, we deduce that
The solution of Eq. (6) is an optimization problem. By introducing the cost function, it follows that
The optimal displacement vector that determines the new position of face \({\omega }_{k}\) is given by:
where, (u_{k}, v_{k}) represents the image at a new position. By computing the derivative of J with respect to u and v and equating them to zero, we obtain:
where \({C}_{k}=\left(\begin{array}{cc}\sum_{{\omega }_{k}}{I}_{x}^{2}& \sum_{{\omega }_{k}}{I}_{x}{I}_{y}\\ \sum_{{\omega }_{k}}{I}_{x}{I}_{y}& \sum_{{\omega }_{k}}{I}_{y}^{2}\end{array}\right)\), and \({D}_{k}=\left(\begin{array}{c}\sum_{{\omega }_{k}}{I}_{x}{I}_{t}\\ \sum_{{\omega }_{k}}{I}_{x}{I}_{t}\end{array}\right)\). Assuming that \(I: \left[1, m\right]\times \left[1, n\right]\subseteq\)\({\mathbb{N}}^{2}\to \left[0, 1\right]\) is the matrix of the 3D face, then the j^{th} level of the pyramid description of the face image is expressed by the recursion:
The displacement vector in Eq. (9) can also be rewritten as:
The displacement vector in Eq. (10) is computed at the deepest pyramid level \({j}_{max}\) (in the Newton–Raphson fashion), and the result of the computation is propagated to the upper level \({j}_{max}1\) by the expression:
Equation (12) was used as the initial estimate for the evaluation of the displacement vector of the 3D face. The final displacement vector is given by the expression
The visible features of the face can be extracted from any location on the face, similar to any other 2D dimensional face. The extracted features are candidates for predicting the overall expression of the face. The Gabor extraction technique is critical for extracting the maximum amount of information required for the classifier.
Feature points extraction
The 2D Gabor filters are spatial sinusoids localized by the Gaussian window, and because they are orientation, localization, and frequencyselective, they are useful in this study. Demonstrate images using Gabor wavelets provides flexibility because the details about their spatial relations are preserved in the process. The general form of the Gabor function is given by:
where \({R}_{1}=uxcos\theta\) and \({R}_{2}=uysin\theta\), u is the spatial frequency of the band pass, θ is the spatial orientation, \(\sigma\) is the standard deviation that the 2D Gaussian envelops, and (x, y) is the position of the light impulse in the visual field. To allow for more robustness in illumination, we set the filter to zero direct current. The Gabor wavelet is then given by:
where \(\left(x,y, \theta ,u,\sigma \right)\) are parameters with (i, j) being the new position of the 2D input point, \(\theta\) is the scale, u is the orientation of the Gabor kernel, \(\sigma\) is the standard deviation of the Gaussian window in the kernel, n is the maximum size of the face peak, and q is the size of the filter given by \(q={\left(2n+1\right)}^{2}\). In this study, we used 8 orientations given by \(\left\{ {0,\frac{\pi }{8},\frac{\pi }{4},\frac{3\pi }{8},\frac{\pi }{2},\frac{5\pi }{8},\frac{3\pi }{4},\frac{7\pi }{8}} \right\}\) and 5 scales given by \(\left\{ {4,4\sqrt 2 ,8,8\sqrt 2 ,16} \right\}\). The sample points of the filtered image are coded into two bits \(\left({x}_{1}, { x}_{2}\right)\) such that:
where I is a subimage of the expressional face; \(\mathfrak{R}\) and \(\mathfrak{I}\) are the real and imaginary parts of each Gabor kernel, respectively; and the star (*) is the convolution operator. The final magnitude response, representing the feature vectors, was computed by determining the square root of the sum of the squares of G_{1} and G_{2}. Figure 5 shows the magnitude response of a template image.
Classification using AdaAdaSVM
For this optimization problem, an SVM with a radial basis function kernel was used as a weak classifier. This weak SVM classifier was trained to produce the optimum Gaussian value for the scale parameter \(\delta\) and regularization parameter \(\partial .\) Typically, the best parameters are \(\left\{ {^{\prime}\partial ^{\prime}:1.0,{ ^{\prime}}\delta {^{\prime}: 0}{\text{.1}}} \right\}\). The feature selection hypothesis is then computed from the expression \(sgn\left[{\sum }_{t1}^{T}{\omega }_{t}{h}_{t}^{1}\left({\varphi }_{t}^{1}\right)\right]\), where T is the final iteration, \({h}_{t}^{1}\) is the hypothesis with the most discriminating information, and \({\omega }_{t}\) is weights that weigh \({h}_{t}^{1}\) based on its classification performance. The learning process formulated in our recent study [25] is as follows:
Step 1: Input the training sets, \([\left({y}_{1}, {x}_{1}\right), \left({y}_{2}, {x}_{2}\right),\dots , \left({y}_{N}, {x}_{N}\right)]\), \(N=a+b\); where datasets a and b comprise \({y}_{i}=+1\) and \({y}_{i}=1\) datasets, respectively. Initially, \(\delta = {\delta }_{ini}, { \delta }_{min}, {\delta }_{step}\). The scale parameter \(\delta\), x, and y are the feature vectors selected by the AdaBoost algorithm.
Step 2: Initialize the training set weights, \({w}_{i}^{(1)}=1\left/ 2a\right., \forall \left({y}_{i}=+1\right)\) and \({w}_{i}^{(1)}=1\left/ 2a\right., \forall \left({y}_{i}=1\right)\).
Do while \(\delta >{\delta }_{min}\)
Step 3: Apply the RBFSVM kernel to train the weighted training datasets by applying the leaveonesubjectout cross validation (LOSOCV) approach and compute the training error for the weak classifier \({h}_{t}\) as
Step 4: At \({\xi }_{t}=1\left/ 2\right.\), reduce \(\delta\) by a factor of \({\delta }_{step}\) and then jump to Step 1.
Step 5: Place the weight of the constituent classifier \({h}_{t}\) such that
Step 6: Update the weights by computing:
where \({N}_{t}\) is a normalization constant and \({\sum }_{i=1}^{n}{w}_{i}^{t+1}=1\)
Step 7: The final classifier is given by
The LOSOCV approach is given by the expression: \(1\left/ 2n\right.=\sum_{t=1}^{n}\left{f}_{i}\left({x}_{i}\right){l}_{i}\right\), where n represents the total trained data.
Facial expression datasets
The algorithm was trained and tested on five popular datasets: Bosphorus, BU3DFE, MMI, CK + , and BP4DSpontaneous, and executed on a (4 CPUs), approximately 2.2 GHz processor with a memory capacity of 8192 MB RAM.
Results and discussion
Experiments on databases
Bosphorus contains 4666 images of 105 subjects [26] comprising 60 men and 5 women, with the majority being Caucasian; 27 of whom were professional actors, in various poses, expressions, and occlusion conditions. In addition to the 6 basic emotional expressions, various systematic head poses (13 yaw and pitch rotations) were present. The texture images have a resolution of 1600 × 1200 pixels, whereas the 3D faces comprise approximately 35,000 vertices [27]. Figure 6 presents sample datasets from Bosphorous. Occlusion images were discarded because they were not the focus of this study. The datasets used comprised 6 poses and 7 expressions. The images were partitioned into training and testing sets using the conventional LOSOCV approach. One specimen from each of the 6 groups of expressions was used as a test dataset during each training run, whereas the rest of the samples were used as a testing set. Table 1 summarizes the FER in Bosphorus.
The BU3DFE database was created at Binghamton University [28]. There were 100 respondents, ranging in age from 18 to 70 years old. Whites, Blacks, East Asians, Middle East Asians, Indians, and Hispanics are among the ethnic groups. Each participant displayed 7 expressions at 4 intensity levels, including neutral, and 6 archetypal facial expressions. Figure 7 shows sample datasets in the database. The images were separated into training and testing sets using the same LOSOCV method as that used for the Bosphorus datasets, and the average recognition accuracy was 94.56%.
The MMI database comprises over 2900 highresolution videos submitted by more than 20 students and research staff members, of which 44% are female, ranging in age from 19 to 62 years old. Seventyfive subjects were included in total, and Fig. 8 shows samples. The datasets are partitioned into training and testing sets using the LOSOCV technique. One sample from each of the 7 types of expressions was used as the test dataset during each training run. The remaining samples were used as training sets. For each training cycle, the samples were repeated with new test samples. The expressions included anger, disgust, fear, happiness, neutral, sadness, and surprise. The average recognition accuracy is 97.2%.
The CK + database is a version of the 210 adult CK database. Participants were 18 to 50 years old, with 69% female, 81% EuroAmerican, 13% AfroAmerican, and 6% from other ethnic groups. The expressions included anger, contempt, disgust, fear, happiness, sadness, and surprise. Figure 9 presents sample datasets. A tenfold crossvalidation procedure was used to partition the datasets into training and testing sets. The average recognition accuracy is 99.48%.
Finally, the BP4DSpontaneous dataset is a 3D video collection of spontaneous facial expressions from young individuals. The database comprises 41 subjects (23 women and 18 men) ranging in age from 18 to 29 years old, including 11 Asians, 6 AfricanAmericans, 4 Hispanics, and 20 EuroAmericans. Figure 10 shows sample images. We extracted expressions of anger, disgust, fear, pain, happiness, sadness, and surprise. The datasets were partitioned into training and testing sets using tenfold crossvalidation. The average recognition accuracy is 97.2%.
Figures 11 and 12 exhibit the respective confusion matrices for facial expressions and pose predictions in the Bosphorus database. Figures 13, 14, 15, and 16 show the rest of the confusion matrices for FERs in BU3DFE, MMI, CK + , and BP4DSpontaneous, respectively.
Comparison of methods
In Table 2, the proposed method was compared to some recent techniques. These results clearly demonstrated that the proposed method is promising. Figures 17, 18, and 19 show the performance of each of the 7 facial expressions. In the BU3DFE database, many authors failed to report the performance of neutral expressions; thus, the comparison was performed using the other 6. The performance shown in Fig. 17 was encouraging. Figure 18 shows the performance of the CK + database. Although the result, as shown in Fig. 18, depicts fierce rivalry between three current methods [29,30,31], the overall average recognition shows that the proposed technique is promising. In the Bosphorus database, the proposed method outperformed the most recent methods (Fig. 19). A comparison of the performances of the individual FER prototypes in the MMI and BP4DSpontaneous databases could not be executed because there were no reported data for comparison at the time of compilation. Statistical analysis using ANOVA shows the following performance results:
In the Bosphorus database, an analysis of variances demonstrated statistically significant differences between the proposed technique and the following: Hariri et al. [36] (p = 0.001), Azazi et al. [37] (p = 0.000), and Moeini A and Moeini H [40] (p = 0.013). In addition, the outcome is the same as in the BU3DFE: the variance analysis shows that a statistically significant difference (p < 0.05) exists between the proposed method and all other methods. However, in the CK + FER database, the statistical analysis shows that, except ref. [41], where a statistically significant difference (p < 0.05) exists, the remaining datasets show no statistically significant differences (p > 0.05). The proposed method compared to yields from An and Liu [29] (p = 0.847), Ch [30] (p = 0.909), and Liao et al. [31] (p = 0.991). Although the analysis appears to reveal a balanced performance between the proposed methodology and the last three techniques, the average recognition accuracy of the proposed method against any of them, as shown in Fig. 18, indicates that the proposed method is superior.
Conclusions
This study improves the FER performance in higher poses. 2D pose conversion schemes have been established to handle poseinvariant FER problems successfully, within a smallscale pose variation. However, they often flop for largescale, indepth face variations because of the disjointedness of the image. Human face geometry is ellipsoidal; therefore, the feature points are robustly tracked from one frame to next using an ellipsoidal model. We use the Gabor feature extraction technique for the salient visible features, mostly around the cheeks, eyes, mouth, and nose ridges. The Gabor feature extraction algorithm is useful for this study because it is selective toward orientation, localization, and frequency. We then used an ensemble classification technique, which combines SVM and AdaBoost, for feature selection and classification. The proposed technique outperforms the most recent and popular methods. In the future, we intend to investigate this problem using other feature extraction methods such as LBP and LBP + HOG.
Availability of data and materials
All data used for this data are publicly available on request from the original authors.
Abbreviations
 FER:

Facial expression recognition
 SVM:

Saturated vector machine
 LBP:

Local binary patterns
 HOG:

Histogram of gradients
 PCA:

Principal component analysis
 KNN:

Knearest neighbor
 SMOTE:

Synthetic minority oversampling technique
 2D:

Twodimensional
 3D:

Threedimensional
 LOSOCV:

Leaveonesubjectout cross validation
References
Panksepp J (2005) Affective consciousness: Core emotional feelings in animals and humans. Conscious Cogn 14(1):3080. https://doi.org/10.1016/j.concog.2004.10.004
Plutchik R (2001) The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Amer Scient 89(4):344350. https://doi.org/10.1511/2001.4.344
Zautra AJ (2003) Emotions, stress, and health. Oxford University Press, Oxford.
Kohler CG, Martin EA, Stolar N, Barrett FS, Verma R, Brensinger C et al (2008) Static posed and evoked facial expressions of emotions in schizophrenia. Schizophr Res 105(13):4960. https://doi.org/10.1016/j.schres.2008.05.010
Ambron E, Foroni F (2015) The attraction of emotions: irrelevant emotional information modulates motor actions. Psychon Bull Rev 22(4):11171123. https://doi.org/10.3758/s134230140779y
Kumari J, Rajesh R, Kumar A (2016) Fusion of features for the effective facial expression recognition. Paper presented at the international conference on communication and signal processing, IEEE, Melmaruvathur, 6–8 June 2016. https://doi.org/10.1109/ICCSP.2016.7754178
Shergill GS, Sarrafzadeh A, Diegel O, Shekar A (2008) Computerized sales assistants: the application of computer technology to measure consumer interesta conceptual framework. J Electron Commer Res 9(2):176191.
Tierney M (2017) Using behavioral analysis to prevent violent extremism: Assessing the cases of Michael ZehafBibeau and Aaron Driver. J Threat Assessm Manag 4(2):98110. https://doi.org/10.1037/tam0000082
Nonis F, Dagnes N, Marcolin F, Vezzetti E (2019) 3D approaches and challenges in facial expression recognition algorithms  A literature review. Appl Sci 9(18):3904. https://doi.org/10.3390/app9183904
Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2011) A dynamic approach to the recognition of 3D facial expressions and their temporal models. Paper presented at the ninth IEEE international conference on automatic face and gesture recognition, IEEE, Santa Barbara, 21–25 March 2011. https://doi.org/10.1109/FG.2011.5771434
Vieriu RL, Tulyakov S, Semeniuta S, Sangineto E, Sebe N (2015) Facial expression recognition under a wide range of head poses. Paper presented at the 11th IEEE international conference and workshops on automatic face and gesture recognition, IEEE, Ljubljana, May 4–8, 2015. https://doi.org/10.1109/FG.2015.7163098
Yadav KS, Singha J (2020) Facial expression recognition using modified ViolaJohn’s algorithm and KNN classifier. Multimed Tools Appl 79(19):1308913107. https://doi.org/10.1007/s1104201908443x
Jones M, Viola P (2003) Fast multiview face detection. Mitsubishi Electric Research Laboratories, Cambridge.
Yao L, Wan Y, Ni HJ, Xu BG (2021) Action unit classification for facial expression recognition using active learning and SVM. Multimed Tools Appl 80(16):2428724301. https://doi.org/10.1007/s1104202110836w
Ashir AM, Eleyan A, Akdemir B (2020) Facial expression recognition with dynamic cascaded classifier. Neural Comput Appl 32(10):62956309. https://doi.org/10.1007/s00521019041384
Farrow CL, Shaw M, Kim H, Juhás P, Billinge SJL (2011) NyquistShannon sampling theorem applied to refinements of the atomic pair distribution function. Phys Rev B 84(13):134105. https://doi.org/10.1103/PhysRevB.84.134105
Li F, Cornwell TJ, de Hoog F (2011) The application of compressive sampling to radio astronomy. I. Deconvolution. Astron Astrophys 528:A31. https://doi.org/10.1051/00046361/201015045
PerezGomez V, RiosFigueroa HV, RechyRamirez EJ, MezuraMontes E, MarinHernandez A (2020) Feature selection on 2D and 3D geometric features to improve facial expression recognition. Sensors 20(17):4847. https://doi.org/10.3390/s20174847
Duan J (2019) Financial system modeling using deep neural networks (DNNs) for effective risk assessment and prediction. J Franklin Inst 356(8):47164731. https://doi.org/10.1016/j.jfranklin.2019.01.046
Kurniawati YE, Permanasari AE, Fauziati S (2018) Adaptive syntheticnominal (ADASYNN) and adaptive syntheticKNN (ADASYNKNN) for multiclass imbalance learning on laboratory test data. Paper presented at the 4th international conference on science and technology, IEEE, Yogyakarta, 7–8 August 2018. https://doi.org/10.1109/ICSTC.2018.8528679
Li HB, Huang D, Morvan JM, Wang YH, Chen LM (2015) Towards 3D face recognition in the real: a registrationfree approach using finegrained matching of 3D keypoint descriptors. Int J Comput Vis 113(2):128142. https://doi.org/10.1007/s1126301407856
Comaniciu D, Ramesh V, Meer P (2003) Kernelbased object tracking. IEEE Trans Pattern Anal Mach Intell 25(5):564577. https://doi.org/10.1109/TPAMI.2003.1195991
Hao GT, Du XP, Chen H, Song JJ, Gao TF (2015) Scaleunambiguous relative pose estimation of space uncooperative targets based on the fusion of threedimensional timeofflight camera and monocular camera. Opt Eng 54(5):053112. https://doi.org/10.1117/1.OE.54.5.053112
Dibeklioglu H, Salah AA, Akarun L (2008) 3D facial landmarking under expression, pose, and occlusion variations. Paper presented at the IEEE second international conference on biometrics: theory, applications and systems, IEEE, Washington, 29 September1 October 2008. https://doi.org/10.1109/BTAS.2008.4699324
Owusu E, Wiafe I (2021) An advance ensemble classification for object recognition. Neural Comput Appl 33(18):1166111672. https://doi.org/10.1007/s00521021058813
Dharavath K, Laskar RH, Talukdar FA (2013) Qualitative study on 3D face databases: A review. Paper presented at the annual IEEE India conference, IEEE, Mumbai, 13–15 December 2013. https://doi.org/10.1109/INDCON.2013.6726093
Sandbach G, Zafeiriou S, Pantic M, Yin LJ (2012) Static and dynamic 3D facial expression recognition: A comprehensive survey. Image Vision Comput 30(10):683697. https://doi.org/10.1016/j.imavis.2012.06.005
Quan W, Matuszewski BJ, Shark LK, AitBoudaoud D (2009) Facial expression biometrics using statistical shape models. EURASIP J Adv Signal Process 2009:261542. https://doi.org/10.1155/2009/261542
An FP, Liu ZW (2020) Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM. Vis Comput 36:483498. https://doi.org/10.1007/s00371019016354
Ch S (2021) An efficient facial emotion recognition system using novel deep learning neural networkregression activation classifier. Multimed Tools Appl 80(12):1754317568. https://doi.org/10.1007/s11042021105472
Liao HB, Wang DH, Fan P, Ding L (2021) Deep learning enhanced attributes conditional random forest for robust facial expression recognition. Multimed Tools Appl 80(19):2862728645. https://doi.org/10.1007/s11042021109518
Kumar MP, Rajagopal MK (2019) Detecting facial emotions using normalized minimal feature vectors and semisupervised twin support vector machines classifier. Appl Intell 49(12):41504174. https://doi.org/10.1007/s1048901901500w
Li S, Deng WH (2019) Blended emotion inthewild: Multilabel facial expression recognition using crowdsourced annotations and deep locality feature learning. Int J Comput Vis 127(6):884906. https://doi.org/10.1007/s1126301811311
Danelakis A, Theoharis T, Pratikakis I, Perakis P (2016) An effective methodology for dynamic 3D facial expression retrieval. Pattern Recogn 52:174185. https://doi.org/10.1016/j.patcog.2015.10.012
Lei YJ, Guo YL, Hayat M, Bennamoun M, Zhou XZ (2016) A twophase weighted collaborative representation for 3D partial face recognition with single sample. Pattern Recogn 52:218237. https://doi.org/10.1016/j.patcog.2015.09.035
Hariri W, Tabia H, Farah N, Benouareth A, Declercq D (2017) 3D facial expression recognition using kernel methods on Riemannian manifold. Eng Appl Artif Intell 64:2532. https://doi.org/10.1016/j.engappai.2017.05.009
Azazi A, Lutfi SL, Venkat I, FernándezMartínez F (2015) Towards a robust affect recognition: Automatic facial expression recognition in 3D faces. Expert Syst Appl 42(6):30563066. https://doi.org/10.1016/j.eswa.2014.10.042
Chen ZX, Huang D, Wang YH, Chen LM (2018) Fast and light manifold CNN based 3D facial expression recognition across pose variations. Paper presented at the 26th ACM international conference on multimedia, ACM, Seoul, 22–26 October 2018. https://doi.org/10.1145/3240508.3240568
Huynh XP, Tran TD, Kim YG (2016) Convolutional neural network models for facial expression recognition using BU3DFE database. In: Kim K, Joukov N (eds) Information Science and Applications (ICISA) 2016. Lecture Notes in Electrical Engineering, vol 376. Springer, Singapore, pp 441–450. https://doi.org/10.1007/9789811005572_44
Moeini A, Moeini H (2015) Realworld and rapid face recognition toward pose and expression variations via feature library matrix. IEEE Trans Inform Forensics secur 10(5):969984. https://doi.org/10.1109/TIFS.2015.2393553
Meena HK, Sharma KK, Joshi SD (2020) Effective curveletbased facial expression recognition using graph signal processing. Signal Image Video Process 14(2):241247. https://doi.org/10.1007/s11760019015479
Acknowledgements
Not applicable.
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
All authors drafted this manuscript. Ideation was proposed by EO. EO and JKA developed the proposed solution. PO performed the experimentation. All authors finally discussed and analyzed the results from the experimentation. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
All authors declare that there is no known competing interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Owusu, E., Appati, J.K. & Okae, P. Robust facial expression recognition system in higher poses. Vis. Comput. Ind. Biomed. Art 5, 14 (2022). https://doi.org/10.1186/s42492022001090
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s42492022001090
Keywords
 Facial expressions
 Threedimensional head pose
 Ellipsoidal model
 Gabor filters
 AdaAdaSVM