Generate QR code
Open Access
Subscription Access
The method of real-time acoustic measurement of dynamical changes in the speaker’s emotional state
https://doi.org/10.32446/0368-1025it.2021-4-49-57
Abstract
In this paper we consider the issues in implementations of interactive voice response systems with remote access. Their efficiency can be improved by automatically analyzing changes in the user's emotional state during the dialogue. In order to measure the indicator of the dynamics of the emotional statein real time, it is proposed to use the effect of sound (phonetic) variability of the user's speech at short intervals (fractions of a minute). The novel method of acoustic measurements in conditions of small samples has been developed based on information-theoretic approach by using a scale-invariant gain-optimized dissimilarity measure of the speech signals in the frequency domain. An example of its practical implementation in soft real time is considered. It is shown that the delay in obtaining the measurement results does not exceed in this case 10–20 sec. The experimental results confirmed the high speed of the proposed method and its sensitivity to changes in the emotional state under the influence of external noise. The proposed method can be used for automated quality control of voice samples of users in unified biometric systems, as well as to improve safety by non-contact identification of potentially dangerous persons with short-term psycho-emotional disorders.
About the Authors
L. V. Savchenko
HSE University
Russian Federation
Russian Federation
Lyudmila V. Savchenko
Nizhniy Novgorod
A. V. Savchenko
HSE University
Russian Federation
Russian Federation
Andrey V. Savchenko
Nizhniy Novgorod
References
1. Davis S. K. et al., Personality and Individual Diff erences, 2020, vol. 160, no. 109938. https://doi.org/10.1016/j.paid.2020.109938.
2. Savchenko V. V., Savchenko А. V., Measurement Techniques, 2020, vol. 62, no. 5, рр. 458–465. https://doi.org/10.1007/s11018-020-01702-1
3. Savchenko V. V., Savchenko А. V., Measurement Techniques, 2019, vol. 62, no. 12, рр. 1071–1078. https://doi.org/10.1007/s11018-020-01736-w
4. Galyashina E. I., Aktual’nye problemy identifi kacii lits po fonogrammam telefonnyh peregovorov, Proceedings of the XXIII International Scientifi c and Practical Conference “Deyatel’nost’ pravoohranitel’nyh organov v sovremennyh usloviyah”, in 2 volumes, Irkutsk, Vostochno-Sibirskij institut Ministerstva vnutrennih del Rossijskoj Federacii Publ., 2018, pp. 141–146, available at: https://istina.msu.ru/publications/article/167326015/(accessed:14.08.2020).(In Russ.)
5. Falagiarda F., Collignon O., Cortex, 2019, vol. 119, рр. 184– 194. https://doi.org/10.1016/j.cortex.2019.04.017
6. Akbulut F. P., Perros H. G., Computer Methods and Programs in Biomedicine, 2020, vol. 195, no. 105571. https://doi.org/10.1016/j.cmpb.2020.105571
7. Shaqra F. A., Duwairi R., Al-Ayyoub M., Procedia Computer Science, 2019, vol. 151, рр. 37–44. https://doi.org/10.1016/j.procs.2019.04.009
8. Arana J. M. et al., Computers in Human Behavior, 2020, vol. 104, no. 106156. https://doi.org/10.1016/j.chb.2019.106156
9. Bourguignon M. et al., NeuroImage, 2020, vol. 216, no. 116788. https://doi.org/10.1016/j.neuroimage.2020.116788
10. Liu Z. et al., Brain and Language, 2020, vol. 203, no. 104755. https://doi.org/10.1016/j.bandl.2020.104755
11. Schuller B., Voice and Speech Analysis in Search of States and Traits, in: Salah A. A., Gevers T. (eds.) Computer Analisis of Human Behavior, Springer, Heidelberg, 2011, 227 p. https://doi.org/10.1007/978-0-85729-994-9_9
12. Cardona D. et al., Neurocomputing, 2017, vol. 265, рр. 78–90. https://doi.org/10.1016/j.neucom.2016.09.140
13. Yu D., Deng L., Automatic Speech Recognition: A Deep Learning Approach, Springer, 2014, 321 p. https://doi.org/10.1007/978-1-4471-5779-3
14. Schuster M., Lecture Notes in Computer Science, 2010, vol. 6230, рр. 8–10. https://doi.org/10.1007/978-3-642-15246-7_3
15. Rammohan R. et al., Journal of Allergy and Clinical Immunology, 2017, vol. 139, iss. 2, no. ab250. https://doi.org/10.1016/j.jaci.2016.12.804
16. Volodin N. A., Ermolenko T. V., Semenyuk V. V., Issledovanie eff ektivnosti primeneniya nejronnyh setej dlya raspoznavaniya emocij cheloveka po golosu, Proceedings of the Conference International Scientifi c Conference “Doneckie chteniya 2019: obrazovanie, nauka, innovacii, kul’tura i vyzovy sovremennosti”, 2019, pp. 221–223, available at: https://elibrary.ru/down load/elibrary_41422521_75290048.pdf(accessed:14.08.2020).(In Russ.)
17. Grachev A. M., Ignatov D. I., Savchenko A. V., Applied Soft Computing, 2019, vol. 79, рр. 354–362. https://doi.org/10.1016/j.asoc.2019.03.057
18. Ustinov R. A., Bezopasnost’ informacionnyh tekhnologij, 2017, vol. 24, no. 4. (In Russ.) https://doi.org/10.26583/bit.2017.4.08
19. Cui S., Li E., Kang X., 2020 IEEE International Conference on Multimedia and Expo (ICME), London, United Kingdom, 2020, рр. 1–6. https://doi.org/10.1109/ICME46284.2020.9102765
20. Savchenko V. V., Radioelectronics and Communications Systems, 2020, vol. 63, no 1, рр. 42–54. https://doi.org/10.3103/S0735272720010045
21. Savchenko V. V., Savchenko А. V., Journal of Communications Technology and Electronics, 2020, vol. 65, no. 11, рр. 1060– 1066. https://doi.org/10.31857/S0033849420110157
22. Hautamäki R.G. et al., Speech Communication, 2017, vol. 95, рр. 1–15. https://doi.org/10.1016/j.specom.2017.10.002
23. Lebedeva N. N., Karimova E. D., Uspekhi fi ziologicheskih nauk, 2014, vol. 45, no. 1, pp. 57–95, available at: http://naukarus.com/akusticheskie-harakteristiki-rechevogo-signala-kak-pokazatelfunktsionalnogo-sostoyaniya-cheloveka(accessed:14.08.2020).(In Russ.)
24. Savchenko V. V., Journal of Communications Technology and Electronics, 2018, vol. 63, no. 1, рр. 53–57. https://doi.org/10.1134/S1064226918010126
25. Savchenko A. V., Savchenko V. V., Journal of Communications Technology and Electronics, 2016, vol. 61, no. 4, рр. 430– 435. https://doi.org/10.1134/S1064226916040112
26. Savchenko V. V., Measurement Techniques, 2018, vol. 61, no. 1, pp. 79–84. https://doi.org/10.1007/s11018-018-1391-8
27. Savchenko V. V., Savchenko L. V., Measurement Techniques, 2019, vol. 62, no. 9, pp. 832–839. https://doi.org/10.1007/s11018-019-01702-1
28. Savchenko L. V., Savchenko A. V., Journal of Communications Technology and Electronics, 2019, vol. 64, no. 3, рр. 238– 244. https://doi.org/10.1134/S1064226919030173
29. Savchenko А. V., Savchenko V. V., Measurement Techniques, 2019, vol. 62, no. 3, pp. 282–288. https://doi.org/10.1007/s11018-019-01617-x
30. Savchenko A. V., Sequential Three-Way Decisions in Effi cient Classifi cation of Piecewise Stationary Speech Signals, in: Polkowski L. et al. (eds) Rough Sets. IJCRS 2017, Lecture Notes in Computer Science, 2017, vol. 10314, Springer, Cham. https://doi.org/10.1007/978-3-319-60840-2_19
31. Kullback S., Information Theory and Statistics, N. Y., Dover Publications, 1997, 432 p., available at: https://www.amazon.com//dp/0486696847(accessed:14.08.2020).
32. Gray R. M. et al., IEEE Transactions on Signal Processing, 1980, vol. 28, no. 4, рр. 367–377. https://doi.org/10.1109/TASSP.1980.1163421
33. Savchenko A. V., Savchenko V. V., Savchenko L. V., Optimization of Gain in Symmetrized Itakura-Saito Discrimination for Pronunciation Learning, in: Kononov A., Khachay M., Kalyagin V., Pardalos P. (eds), Mathematical Optimization Theory and Operations Research, MOTOR 2020, Lecture Notes in Computer Science, 2020, vol. 12095, Springer, Cham. https://doi.org/10.1007/978-3-030-49988-4_30
34. Vestman V. et al., Speech Communication, 2018, vol. 99, рр. 62–79. https://doi.org/10.1016/j.specom.2018.02.009
35. Candan Ç., Signal Processing, 2020, vol. 166, no. 107256. https://doi.org/10.1016/j.sigpro.2019.107256
36. Tuncel K. S., Baydogan M. G., Pattern Recognition, 2018, vol. 73, рр. 202–215. https://doi.org/10.1016/j.patcog.2017.08.016
37. Savchenko V. V., Savchenko А. V., Radioelectronics and Communications Systems, 2019, vol. 62, рр. 276–286. https://doi.org/10.3103/S0735272719050042
38. Marple S. L., Digital Spectral Analysis with Applications, 2nd ed. Mineola, New York, Dover Publications, 2019, 432 p., available at: https://www.goodreads.com/book/show/19484239 (accessed: 14.08.2020).
39.
Review
For citations:
Savchenko L.V., Savchenko A.V. The method of real-time acoustic measurement of dynamical changes in the speaker’s emotional state. Izmeritel`naya Tekhnika. 2021;(4):49-57. (In Russ.) https://doi.org/10.32446/0368-1025it.2021-4-49-57
Views:
120
Email this article 