On the impact of feature reduction on leave-one-subject-out cross-validation

Abstract

The high inter-individual variability of the electroencephalogram (EEG) is investigated in this contribution using leave-one-subject-out cross-validation (LOSO CV). The question of whether feature reduction can significantly increase the generalization ability of LOSO CV is addressed or whether feature reduction causes too high loss of relevant features, thus worsening the results. EEG recordings from three driving simulation studies are analyzed, in which microsleep (MS) and sustained attention (SA) were observed in 66 young drivers, some with very high fatigue. The gradient boosting machine LightGBM, was used as a classifier for discriminating MS and SA. The results show that the mean classification accuracies at validation sets have been found to be 90.8±0.8% for the standard CV and 86.4± 11.0% for the LOSO-CV. Through three different feature reduction criteria, a total of 57 different reductions were performed with different thresholds, but there were no significant improvements in the mean LOSO-CV accuracy. If the reduction was chosen too high, i.e. more than 96% of the features were not processed, then significant reductions of the mean classification accuracies down to 79.1% were obtained.