Abstract
Supervised automatic sleep scoring algorithms are usually trained using sleep stage labels manually annotated on 30 s epochs of PSG data. In this study, we investigate the impact of using shorter epochs with various PSG input signals for training and testing a Long Short Term Memory (LSTM) neural network. An LSTM model is evaluated on the provided 30 s epoch sleep stage labels from a publicly available dataset, as well as on 10 s subdivisions. Additionally, three independent scorers re-labeled a subset of the dataset on shorter time windows. The automatic sleep scoring experiments were repeated on the re-annotated subset.The highest performance is achieved on features extracted from 30 s epochs of a single channel frontal EEG. The resulting accuracy, precision and recall were of 92.22%, 67.58% and 66.00% respectively. When using a shorter epoch as input, the performance decreased by approximately 20%. Re-annotating a subset of the dataset on shorter time epochs did not improve the results and further altered the sleep stage detection performance. Our results show that our feature-based LSTM classification algorithm performs better on 30 s PSG epochs when compared to 10 s epochs used as input. Future work could be oriented to determining whether varying the epoch size improves classification outcomes for different types of classification algorithms.
Acknowledgments
The authors would like to thank Alexandra Jahaleanu, Marjolein Vliexs and Makrina Sekeri from Onera Health for their support in the re-annotation process.
-
Research funding: None declared.
-
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
-
Competing interests: Authors state no conflict of interest.
-
Informed consent: Not applicable.
-
Ethical approval: Not applicable.
References
1. Perry, G, Patil, S, Presley-Cantrell, L. Raising awareness of sleep as a healthy behavior how much sleep is needed and are we there ? why is sleep a public health issue? Call to action. Prev Chronic Dis 2013;10:130081. https://doi.org/10.5888/pcd10.130081.Search in Google Scholar PubMed PubMed Central
2. Killgore, WDS, Balkin, TJ, Yarnell, AM, Capaldi, VF. Sleep deprivation impairs recognition of specific emotions. Neurobiol Sleep Circadian Rhythms 2017;3:10–6. https://doi.org/10.1016/j.nbscr.2017.01.001.Search in Google Scholar PubMed PubMed Central
3. Schwarz, J, Axelsson, J, Gerhardsson, A, Tamm, S, Fischer, H, Kecklund, G, et al.. Mood impairment is stronger in young than in older adults after sleep deprivation. J Sleep Res 2019;28. https://doi.org/10.1111/jsr.12801.Search in Google Scholar PubMed PubMed Central
4. Nollet, M, Wisden, W, Franks, NP. Sleep deprivation and stress: a reciprocal relationship. Interface Focus 2020;10:3. https://doi.org/10.1098/rsfs.2019.0092.Search in Google Scholar PubMed PubMed Central
5. Hudson, AN, Van Dongen, HPA, Honn, KA. Sleep deprivation, vigilant attention, and brain function: a review. Neuropsychopharmacology 2020;45:21–30. https://doi.org/10.1038/s41386-019-0432-6.Search in Google Scholar PubMed PubMed Central
6. Wu, H, Dunnett, S, Ho, YS, Chang, RCC. The role of sleep deprivation and circadian rhythm disruption as risk factors of Alzheimer’s disease. Front Neuroendocrinol 2019;54:100764. https://doi.org/10.1016/j.yfrne.2019.100764.Search in Google Scholar PubMed
7. St Louis, EK, Boeve, AR, Boeve, BF. REM sleep behavior disorder in Parkinson’s disease and other synucleinopathies. Mov Disord 2017;32:645–58. https://doi.org/10.1002/mds.27018.Search in Google Scholar PubMed
8. Romanella, SM, Roe, D, Tatti, E, Cappon, D, Paciorek, R, Testani, E, et al.. The sleep side of aging and alzheimer’s disease. Sleep Med 2021;77:209–25. https://doi.org/10.1016/j.sleep.2020.05.029.Search in Google Scholar PubMed PubMed Central
9. Acquavella, J, Mehra, J, Bron, M, Suomi, JMH, Hess, GP. Prevalence of narcolepsy and other sleep disorders and frequency of diagnostic tests from 2013-2016 in insured patients actively seeking care. J Clin Sleep Med 2020;16:1255–63. https://doi.org/10.5664/jcsm.8482.Search in Google Scholar PubMed PubMed Central
10. Berry, RB, Brooks, R, Gamaldo, CE, Harding, SM, Lloyd, RM, Quan, SF, et al.. The AASM manual for the scoring of sleep and associated events. Version 2.4.Am. Acad. Sleep Med 2017.Search in Google Scholar
11. Loh, HW, Ooi, CP, Vicnesh, J, Oh, SL, Fuast, O, Gertych, A, et al.. Automated detection of sleep stages using deep learning techniques: a systematic review of the last decade (2010–2020). Appl Sci Switz 2020;10:1–25. https://doi.org/10.3390/app10248963.Search in Google Scholar
12. Chriskos, P, Frantzidis, CA, Nday, CM, Gkivogkli, PT, Bamidis, PD, Kourtidou-Papadeli, C. A review on current trends in automatic sleep staging through bio-signal recordings and future challenges. Sleep Med Rev 2021;55:101377. https://doi.org/10.1016/j.smrv.2020.101377.Search in Google Scholar PubMed
13. Fernandez-Varela, I, Hernandez-Pereira, E, Alvarez-Estevez, D, Moret-Bonillo, V. A convolutional network for sleep stages classification; 2019. arXiv:1902.05748.10.3390/proceedings2181174Search in Google Scholar
14. Tautan, AM, Rossi, AC, De Franciso, R, Ionescu, B. Automatic sleep stage detection using a single channel frontal EEG. Presented at the 7th IEEE international conference on e-health and bioengineering – EHB, Iasi Romania; 2019.10.1109/EHB47216.2019.8969973Search in Google Scholar
15. Biswal, S, Kulas, J, Sun, H, Goparaju, B, Westover, MB, Bianchi, MT, et al.. SLEEPNET: automated sleep staging system via deep learning; 2017. arXiv, 1–17.Search in Google Scholar
16. Hassan, AR, Bashar, SK, Bhuiyan, MIH. On the classification of sleep states by means of statistical and spectral features from single channel Electroencephalogram. In: 2015 International conference on advances in computing, communications and informatics (ICACCI), Kochi, India; 2015:2238–43 pp. https://doi.org/10.1109/ICACCI.2015.7275950.Search in Google Scholar
17. Alickovic, E, Subasi, A. Ensemble SVM method for automatic sleep stage classification. IEEE Trans Instrum Meas 2018;67:1258–65. https://doi.org/10.1109/TIM.2018.2799059.Search in Google Scholar
18. Hassan, AR, Bhuiyan, MIH. Automatic sleep scoring using statistical features in the EMD domain and ensemble methods. Biocybern Biomed Eng 2016;36:248–55. https://doi.org/10.1016/j.bbe.2015.11.001.Search in Google Scholar
19. Sundararajan, K, Georgievska, G, te Lindert, BHW, Gehrman, PR, Ramauter, J, Mazzotti, DR, et al.. Sleep classification from wrist-worn accelerometer data using random forests. Sci Rep 2021;11:1–10. https://doi.org/10.1038/s41598-020-79217-x.Search in Google Scholar PubMed PubMed Central
20. Cen, L, Yu, ZL, Tang, Y, Shi, W, Kluge, T, Ser, W. Deep learning method for sleep stage classification. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2017;796–802. https://doi.org/10.1007/978-3-319-70096-0_81.Search in Google Scholar
21. Phan, H, Ch, OY, Koch, P, Mertins, A, Vos, MD. Fusion of end-to-end deep learning models for sequence-to-sequence sleep staging 2019:1829–33. https://doi.org/10.1109/embc.2019.8857348.Search in Google Scholar
22. Muller, B, Gabelein, WD, Schulz, H. A taxonomic analysis of sleep stages. Sleep 2006;29:967–74. https://doi.org/10.1093/sleep/29.7.967.Search in Google Scholar PubMed
23. Cesari, M, Stefani, A, Penzel, T, Ibrahim, A, Hackner, H, Heidbreder, A, et al.. Inter-rater sleep stage scoring reliability between manual scoring from two European sleep centers and automatic scoring performed by the artificial intelligence-based Stanford-STAGES algorithm. J Clin Sleep Med 2021;17:1237–47. https://doi.org/10.5664/jcsm.9174.Search in Google Scholar PubMed PubMed Central
24. Koch, H, Jennum, P, Christensen, JAE. Automatic sleep classification using adaptive segmentation reveals an increased number of rapid eye movement sleep transitions. J Sleep Res 2019;28. https://doi.org/10.1111/jsr.12780.Search in Google Scholar PubMed
25. Álvarez-Estévez, D, Fernández-Pastoriza, JM, Hernández-Pereira, E, Moret-Bonillo, V. A method for the automatic analysis of the sleep macrostructure in continuum. Expert Syst Appl 2013;40:1796–803. https://doi.org/10.1016/j.eswa.2012.09.022.Search in Google Scholar
26. Supratak, A, Dong, H, Wu, C, Guo, Y, DeepSleepNet. A model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Trans Neural Syst Rehabil Eng 2017;25:1998–2008. https://doi.org/10.1109/tnsre.2017.2721116.Search in Google Scholar
27. Phan, H, Andreotti, F, Cooray, N, Chen, OY, de Vos, M. SeqSleepNet : end-to-end hierarchical recurrent neural network for sleep staging. IEEE Trans Neural Syst Rehabil Eng 2019;27:400–10. https://doi.org/10.1109/tnsre.2019.2896659.Search in Google Scholar
28. Rosenburg, R, van Hout, S. The American academy of sleep medicine interscorer reliability program: sleep stage scoring. J Clin Sleep Med 2013;9.10.5664/jcsm.2350Search in Google Scholar PubMed PubMed Central
29. Goldberger, AA, Glass, L, Hausdorff, JM, Ivanov, P, Ivanov, PC, Mark, RG, et al.. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 200AD;101:215–20. https://doi.org/10.1161/01.cir.101.23.e215.Search in Google Scholar PubMed
30. Ghassemi, MM, Moody, BE, Lehman, LW, Song, C, Li, Q, Sun, H, et al.. The PhysioNet/computing in cardiology challenge 2018. Computing in Cardiology Conference (CinC) 2018;20–3. https://doi.org/10.22489/CinC.2018.049.Search in Google Scholar PubMed PubMed Central
31. Iber, C, Ancoli-Israel, S, Chesson, A, Stuart, Q. The AASM manual for the scoring of sleep and associated events 2007.Search in Google Scholar
32. Gwet, K. Handbook of inter-rater reliability, 4th ed. Gaithersburg: Advanced Analytics LLC; 2014.Search in Google Scholar
33. Grigg-Damberger, MM. The AASM scoring manual four years later. J Clin Sleep Med 2012;8:323–32. https://doi.org/10.5664/jcsm.1928.Search in Google Scholar PubMed PubMed Central
34. Cohen, JA. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960:37–46.10.1177/001316446002000104Search in Google Scholar
35. Fleiss, JL. Measuring nominal scale agreement among many raters. Psychol Bull 1971;76:378–82. https://doi.org/10.1037/h0031619.Search in Google Scholar
36. Tautan, A, Rossi, AC, De Francisco, R, Ionescu, B. Automatic sleep stage detection: a study on the influence of various PSG input signals. Presented at the 42nd annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC) in conjunction with the 43rd annual conference of the Canadian Medical and Biological Engineering Society 2020.10.1109/EMBC44109.2020.9175628Search in Google Scholar PubMed
37. Tautan, AM, Rossi, AC, De Francisco, R, Ionescu, B. Dimensionality reduction for EEG-based sleep stage detection: comparison of autoencoders, principal component analysis and factor analysis. Biomed Tech 2020;66:125–36. https://doi.org/10.1515/bmt-2020-0139.Search in Google Scholar PubMed
38. Laza, R, Pavón, R, Reboiro-Jato, M, Fdez-Riverola, F. Evaluating the effect of unbalanced data in biomedical document classification. J Integr Bioinformat 2011;8:177. https://doi.org/10.1515/jib-2011-177.Search in Google Scholar
39. More, A. Survey of resampling techniques for improving classification performance in unbalanced datasets 2016;10000:1–7.Search in Google Scholar
40. Pozzolo, AD, Caelen, O, Bontempi, G. When is undersampling effective in unbalanced classification tasks? Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2015;9284:200–15.10.1007/978-3-319-23528-8_13Search in Google Scholar
41. Goodfellow, I, Bengio, Y, Courville, A. Deep learning. MIT Press; 2016. [Online]. Available from: http://www.deeplearningbook.org.Search in Google Scholar
42. Yu, Y, Si, X, Hu, C, Zhang, J. A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput 2019;31:1235–70. https://doi.org/10.1162/neco_a_01199.Search in Google Scholar PubMed
43. Hochreiter, S, Schmidhuber, J. Long short-term memory. Neural Comput 1997;9:1735–80. https://doi.org/10.1162/neco.1997.9.8.1735.Search in Google Scholar PubMed
44. Kingma, DP, Ba, JL. Adam: a method for stochastic optimization. In: 3rd int conf learn represent ICLR 2015 - conf track proc. San Diego, USA: ICLR; 2015:1–15 pp.Search in Google Scholar
45. Pedregosa, F, Varoquaux, G, Gramfort, A, Michel, V, Thirion, B, Grisel, O, et al.. Scikit-learn: machine learning in Python. J Mach Learn Res 2011;12:2825–30.Search in Google Scholar
46. Barnas, G, Mackenzie, CF, Skacel, M, Hempleman, SC, Wicke, KM, Skacel, CM, et al.. Amplitude dependency of regional chest wall resistance and elastance at normal breathing frequencies. Am Rev Respir Dis 1988;140. https://doi.org/10.1164/ajrccm/140.1.25.1.Search in Google Scholar PubMed
47. Meng, Q, Catchpoole, D, Skillicorn, D, Kennedy, PJ, Relational autoencoder for feature extraction, arXiv, 364–371, 2018.10.1109/IJCNN.2017.7965877Search in Google Scholar
48. Chen, Z, Yang, F. Intelligent terminal based on single channel EEG signal LSTM sleep staging. Comput Sci Appl 2019;9:1156–68. https://doi.org/10.12677/csa.2019.96131.Search in Google Scholar
49. Sun, Y, Wang, B, Jin, J, Wang, X. Deep convolutional network method for automatic sleep stage classification based on neurophysiological signals 2018:1–5.10.1109/CISP-BMEI.2018.8633058Search in Google Scholar
50. Hao, J, Luo, S, Pan, L A novel sleep staging algorithm based on hybrid neural network 2019;2019:1–4.10.1109/ICEIEC.2019.8784612Search in Google Scholar
51. Yang, Y, Zheng, X, Yuan, F. A study on automatic sleep stage classification based on CNN-LSTM. In: Int conf comput sci educ. Colombo, Sri Lanka: Int conf comput sci educ; 2018:1–5 pp.10.1145/3265689.3265693Search in Google Scholar
52. Michielli, N, Acharya, UR, Molinari, F. Cascaded LSTM recurrent neural network for automated sleep stage classification using single-channel EEG signals. Comput Biol Med 2019;106:71–81. https://doi.org/10.1016/j.compbiomed.2019.01.013.Search in Google Scholar PubMed
53. Kuo, C, Chen, G, Lin, N. Automatic sleep staging using deep long short-term memory: validation in large-scale datasets 2019;58–64.10.1145/3365966.3365980Search in Google Scholar
54. Huang, Y, Zhang, Y. A two-phase model for sleep staging using single channel EEG. 2018 Int. Conf. Big Data Artif. Intell. BDAI 2018;1:123–7.10.1109/BDAI.2018.8546683Search in Google Scholar
55. Zhang, L, Fabbri, D, Upender, R, Kent, D. Automated sleep stage scoring of the sleep heart health study using deep neural networks. Sleep 2019;42:zsz159. https://doi.org/10.1093/sleep/zsz159.Search in Google Scholar PubMed PubMed Central
56. Rahman, MM, Bhuiyan, MIH, Hassan, AR. Sleep stage classification using single-channel EOG. Comput Biol Med 2018;102:211–20. https://doi.org/10.1016/j.compbiomed.2018.08.022.Search in Google Scholar PubMed
57. Hassan, AR, Bhuiyan, MIH. A decision support system for automatic sleep staging from EEG signals using tunable Q-factor wavelet transform and spectral features. J Neurosci Methods 2016;271:107–18. https://doi.org/10.1016/j.jneumeth.2016.07.012.Search in Google Scholar PubMed
58. Hassan, AR, Subasi, A. A decision support system for automated identification of sleep stages from single-channel EEG signals. Knowl-Based Syst 2017;115;128–124. https://doi.org/10.1016/j.knosys.2017.05.005.Search in Google Scholar
59. Zaremba, W, Sutskever, I, Vinyals, O. Recurrent neural network regularization 2014;2013:1–8.Search in Google Scholar
© 2022 Walter de Gruyter GmbH, Berlin/Boston
