{"title":"Using Speech Emotion Recognition as a Longitudinal Biomarker for Alzheimer\u2019s Disease","authors":"Yishu Gong, Liangliang Yang, Jianyu Zhang, Zhengyu Chen, Sihong He, Xusheng Zhang, Wei Zhang","volume":203,"journal":"International Journal of Biomedical and Biological Engineering","pagesStart":267,"pagesEnd":273,"ISSN":"1307-6892","URL":"https:\/\/publications.waset.org\/pdf\/10013336","abstract":"<p>Alzheimer\u2019s disease (AD) is a progressive neurodegenerative disorder that affects millions of people worldwide and is characterized by cognitive decline and behavioral changes. People living with Alzheimer\u2019s disease often find it hard to complete routine tasks. However, there are limited objective assessments that aim to quantify the difficulty of certain tasks for AD patients compared to non-AD people. In this study, we propose to use speech emotion recognition (SER), especially the frustration level as a potential biomarker for quantifying the difficulty patients experience when describing a picture. We build an SER model using data from the IEMOCAP dataset and apply the model to the DementiaBank data to detect the AD\/non-AD group difference and perform longitudinal analysis to track the AD disease progression. Our results show that the frustration level detected from the SER model can possibly be used as a cost-effective tool for objective tracking of AD progression in addition to the Mini-Mental State Examination (MMSE) score.<\/p>","references":"[1] Allan M Landes, Susan D Sperry, and Milton E Strauss. Prevalence\r\nof apathy, dysphoria, and depression in relation to dementia severity\r\nin alzheimer\u2019s disease. The Journal of neuropsychiatry and clinical\r\nneurosciences, 17(3):342\u2013349, 2005.\r\n[2] Fariba Mirakhori, Mina Moafi, Maryam Milanifard, Hossein Tahernia,\r\net al. Diagnosis and treatment methods in alzheimer\u2019s patients based\r\non modern techniques: The orginal article. Journal of Pharmaceutical\r\nNegative Results, pages 1889\u20131907, 2022.\r\n[3] Michael Woodward. Aspects of communication in alzheimer\u2019s disease:\r\nclinical features and treatment options. International psychogeriatrics,\r\n25(6):877\u2013885, 2013.\r\n[4] In\u02c6es Vigo, Luis Coelho, and Sara Reis. Speech-and language-based\r\nclassification of alzheimer\u2019s disease: A systematic review.\r\nBioengineering, 9(1):27, 2022.\r\n[5] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily\r\nMower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S\r\nNarayanan. Iemocap: Interactive emotional dyadic motion capture\r\ndatabase. Language resources and evaluation, 42(4):335\u2013359, 2008.\r\n[6] Francois Boller and James Becker. Dementiabank database guide.\r\nUniversity of Pittsburgh, 2005.\r\n[7] Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain.\r\nConvolutional mkl based multimodal emotion recognition and sentiment\r\nanalysis. In 2016 IEEE 16th international conference on data mining\r\n(ICDM), pages 439\u2013448. IEEE, 2016.\r\n[8] Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval,\r\nFelix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan\r\nEmanuel, and Yinnon Haviv. Towards learning a universal non-semantic\r\nrepresentation of speech. arXiv preprint arXiv:2002.12764, 2020.\r\n[9] Michael Neumann and Ngoc Thang Vu. Improving speech emotion\r\nrecognition with unsupervised representation learning on unlabeled\r\nspeech. In ICASSP 2019-2019 IEEE International Conference on\r\nAcoustics, Speech and Signal Processing (ICASSP), pages 7390\u20137394.\r\nIEEE, 2019.\r\n[10] Zaijing Li, Fengxiao Tang, Ming Zhao, and Yusen Zhu. Emocaps:\r\nEmotion capsule based model for conversational emotion recognition.\r\narXiv preprint arXiv:2203.13504, 2022.\r\n[11] Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and\r\nYongbin Li. Unimse: Towards unified multimodal sentiment analysis\r\nand emotion recognition. arXiv preprint arXiv:2211.11256, 2022.\r\n[12] Taewoon Kim and Piek Vossen. Emoberta: Speaker-aware\r\nemotion recognition in conversation with roberta. arXiv preprint\r\narXiv:2108.12009, 2021.\r\n[13] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe,\r\nSamuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad,\r\nAbdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh,\r\nSzu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, Franc\u00b8ois Grondin,\r\nWilliam Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua\r\nBengio. SpeechBrain: A general-purpose speech toolkit, 2021.\r\narXiv:2106.04624.\r\n[14] Fasih Haider, Sofia de la Fuente, Pierre Albert, and Saturnino Luz.\r\nAffective speech for alzheimer\u2019s dementia recognition. LREC: Resources\r\nand ProcessIng of linguistic, para-linguistic and extra-linguistic Data\r\nfrom people with various forms of cognitive\/psychiatric\/developmental\r\nimpairments (RaPID), pages 67\u201373, 2020.\r\n[15] M Rupesh Kumar, Susmitha Vekkot, S Lalitha, Deepa Gupta,\r\nVarasiddhi Jayasuryaa Govindraj, Kamran Shaukat, Yousef Ajami\r\nAlotaibi, and Mohammed Zakariah. Dementia detection from speech\r\nusing machine learning and deep learning architectures. Sensors,\r\n22(23):9311, 2022.\r\n[16] Jody Corey-Bloom and Michael S Rafii. The natural history of\r\nalzheimer\u2019s disease. In Dementia, pages 473\u2013489. CRC Press, 2017.\r\n[17] Louise Cummings. Describing the cookie theft picture: Sources\r\nof breakdown in alzheimer\u2019s dementia. Pragmatics and Society,\r\n10(2):153\u2013176, 2019.\r\n[18] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar,\r\nEric Battenberg, and Oriol Nieto. librosa: Audio and music signal\r\nanalysis in python. In Proceedings of the 14th python in science\r\nconference, volume 8, pages 18\u201325, 2015.\r\n[19] Karen Simonyan and Andrew Zisserman. Very deep convolutional\r\nnetworks for large-scale image recognition. arXiv preprint\r\narXiv:1409.1556, 2014.\r\n[20] Alex Krizhevsky. One weird trick for parallelizing convolutional neural\r\nnetworks. arXiv preprint arXiv:1404.5997, 2014.\r\n[21] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip\r\nKegelmeyer. Smote: synthetic minority over-sampling technique.\r\nJournal of artificial intelligence research, 16:321\u2013357, 2002.\r\n[22] Juergen Dukart, Matthias L Schroeter, Karsten Mueller, and Alzheimer\u2019s\r\nDisease Neuroimaging Initiative. Age correction in dementia\u2013matching\r\nto a healthy brain. PloS one, 6(7):e22193, 2011.\r\n[23] Alberto Abadie and Guido W Imbens. Large sample properties\r\nof matching estimators for average treatment effects. econometrica,\r\n74(1):235\u2013267, 2006.\r\n[24] Madeline M Maier-Lorentz. Effective nursing intervention for the\r\nmanagement of alzheimer\u2019s disease. Journal of Neuroscience nursing,\r\n32(3):153, 2000.","publisher":"World Academy of Science, Engineering and Technology","index":"Open Science Index 203, 2023"}