Using BCI and EEG to process and analyze driver’s brain activity signals during VR simulation

Mirosław Nader; Ilona  Jacyna-Gołda; Stanisław Nader; Karol Nehring

doi:10.5604/01.3001.0015.6305

Authors

Mirosław Nader Warsaw University of Technology, Faculty of Transport, Warsaw Author https://orcid.org/0000-0002-5344-0195
Ilona Jacyna-Gołda Warsaw University of Technology, Faculty of Mechanical and Industrial Engineering, Warsaw Author https://orcid.org/0000-0001-8840-3127
Stanisław Nader Warsaw Management University, Warsaw Author
Karol Nehring Warsaw University of Technology, Faculty of Transport, Warsaw Author https://orcid.org/0000-0002-0682-8795

DOI:

https://doi.org/10.5604/01.3001.0015.6305

Keywords:

signal processing, EEG, BCI, emotion recognition, driver, virtual reality

Abstract

The use of popular brain–computer interfaces (BCI) to analyze signals and the behavior of brain activity is a very current problem that is often undertaken in various aspects by many researchers. This comparison turns out to be particularly useful when studying the flows of information and signals in the human-machine-environment system, especially in the field of transportation sciences. This article presents the results of a pilot study of driver behavior with the use of a pro-prietary simulator based on Virtual Reality technology. The study uses the technology of studying signals emitted by the human mind and its specific zones in response to given environmental factors. A solution based on virtual reality with the limitation of external stimuli emitted by the real world was proposed, and computational analysis of the obtained data was performed. The research focused on traffic situations and how they affect the subject. The test was attended by representatives of various age groups, both with and without a driving license. This study presents an original functional model of a research stand in VR technology that we designed and built. Testing in VR conditions allows to limit the influence of undesirable external stimuli that may distort the results of readings. At the same time, it increases the range of road events that can be simulated without generating any risk for the participant. In the presented studies, the BCI was used to assess the driver's behavior, which allows for the activity of selected brain waves of the examined person to be registered. Electro-encephalogram (EEG) was used to study the activity of brain and its response to stimuli coming from the Virtual Reality created environment. Electrical activity detection is possible thanks to the use of electrodes placed on the skin in selected areas of the skull. The structure of the proprietary test-stand for signal and information flow simulation tests, which allows for the selection of measured signals and the method of parameter recording, is presented. An important part of this study is the presentation of the results of pilot studies obtained in the course of real research on the behavior of a car driver.

References

AFANASIEVA, I., GALKIN, A., 2018. Assessing the information flows and established their effects on the results of driver’s activity. Archives of Transport, 45(1), 7-23. DOI: https://doi.org/10.5604/01.3001.0012.0938.

AGUDELO-VÉLEZ, L., SARMIENTO-OR-DOSGOITIA, I., CÓRDOBA-MAQUILÓN, J., 2021. Virtual reality as a new tool for transport data collection. Archives of Transport, 60(4), 23-38. DOI: https://doi.org/10.5604/01.3001.0015 .5392.

BIAN, D., WADE, J.W., ZHANG, L., BEKELE, E., SWANSON, A., CRITTENDON, J.A., SARKAR, M., WARREN, Z., SARKAR, N., 2013. A Novel Virtual Reality Driving Environment for Autism Intervention. In: Proceedings of the Universal Access in Human-Computer Interaction. User and Context Diversity, Stephanidis, C., Antona, M., Eds., Springer Berlin Heidelberg: Berlin, Heideberg, 2013, 474–483.

BOZKIR, E., GEISLER, D., KASNECI, E., 2019. Assessment of Driver Attention during a Safety Critical Situation in VR to Generate VR-Based Training. In: Proceedings of the ACM Symposium on Applied Perception 2019, Association for Computing Machinery: New York, NY, USA, 23: 1-5.

CHEN, T., YIN, H., YUAN, X., GU, Y., REN, F., SUN, X., 2021. Emotion recognition based on fusion of long short-term memory networks and SVMs. Digital Signal Processing, 117: 103153. DOI: 10.1016/J.DSP.2021.103153.

CHEN, Z., LI, Q., WU, L., CHENG, S., LIN, P., 2019. Optimal data collection of multi-radio multi-channel multi-power wireless sensor networks for structural monitoring applications: A simulation study. Structural Control and Health Monitoring, 26(7), e2328. DOI: https://doi.org/10.1002/stc.2328.

CHRZANOWICZ, T.P., MACKUN, T., 2017. Methodology for assessing the lighting of pedestrian crossings based on light intensity parameters. MATEC Web of Conferences, 122: 01008. DOI: 10.1051/matecconf/2017122010 08.

CUDLENCO, N., POPESCU, N., LEORDEANU, M., 2020. Reading into the mind’s eye: Boosting automatic visual recognition with EEG signals. Neurocomputing, 386: 281–292. DOI: 10.1016/j.neucom.2019.12.076.

DRAGOS, K., THEILER, M., MAGALHÃES, F., MOUTINHO, C., SMARSLY, K., 2018. Onboard data synchronization in wireless structural health monitoring systems based on phase locking. Structural Control and Health Monitoring, 25(11), e2248. DOI: https://doi.org/10.1002/stc.2248.

DUANN, J.-R., CHEN, P.-C., KO, L.-W., HUANG, R.-S., JUNG, T.-P., LIN, C.-T., 2009. Detecting Frontal EEG Activities with Fore-head Electrodes. In: Proceedings of the Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience, Schmorrow, D.D., Estabrooke, I. V, Grootjen, M., Eds., Springer Berlin Heidelberg: Berlin, Heidelberg 2009, 373-379.

EVANS, G.W., WALLACE, G.F., SUTHERLAND, G.L., 1967. Simulation Using Digital Computers. Prentice-Hall: Englewood Cliffs, N.J.

FAN, J., WADE, J.W., BIAN, D., KEY, A.P., WARREN, Z.E., MION, L.C., SARKAR, N., 2015. A Step towards EEG-based brain computer interface for autism intervention. In: Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 3767–3770.

HAJINOROOZI, M., MAO, Z., JUNG, T.P., LIN, C.T., HUANG, Y., 2016. EEG-based prediction of driver’s cognitive performance by deep convolutional neural network. Signal Processing: Image Communication, 47: 549-555. DOI: 10.1016/j.image.2016.05.018.

HASAN, M., PEREZ, D., SHEN, Y., YANG, H., 2021. Distributed Microscopic Traffic Simulation with Human-in-the-Loop Enabled by Virtual Reality Technologies. Advances in Engineering Software, 154(3): 102985. DOI: 10.1016/J.ADVENGSOFT.2021.102985.

HASSANIEN, A.E., AZAR, A., 2014. Brain Computer Interfaces: Current Trends and Applications. ISBN 978-3319109770.

HE, B., YUAN, H., MENG, J., GAO, S., 2020. Brain-Computer Interfaces. In Neural Engineering. He, B., Ed., Springer International Publishing: Cham., 131-183. ISBN 978-3-030-43395-6.

HERRMANN, C.S., DEBENER, S., 2008. Simultaneous recording of EEG and BOLD responses: a historical perspective. International Journal of Psychophysiology, 67(3): 161–168. DOI: 10.1016/j.ijpsycho.2007.06.006.

HILFERT, T., KÖNIG, M., 2015. Low-cost virtual reality environment for engineering and construction. Visual Engineering 4, 2015: 2(2016). DOI: 10.1186/s40327-015-0031-5.

HUANG, K.-C., HUANG, T.-Y., CHUANG, C.-H., KING, J.-T., WANG, Y.-K., LIN, C.-T., JUNG, T.-P., 2016. An EEG-Based Fatigue Detection and Mitigation System. International Journal of Neural Systems, 26(4): 1650018. DOI: 10.1142/S0129065716500180.

JACYNA, M, SEMENOV, I., 2020. Models of vehicle service system supply under information uncertainty. Eksploatacja i Niezawodnosc – Maintenance and Reliability, 22 (4): 694–704 DOI: http://dx.doi.org/10.17531/ein. 2020.4.13.

KIM, S., LEE, S., KANG, H., KIM, S., AHN, M., 2021. P300 Brain–Computer Interface-Based Drone Control in Virtual and Augmented Reality. Sensors, 21(17): 5765. DOI: 10.3390/s21175765.

KIM, Y.M., RHIU, I., 2021. A comparative study of navigation interfaces in virtual reality environments: A mixed-method approach. Applied Ergonomics, 96(3): 103482.

KRISHNAN, A., BAI, V.T., 2021. Investigation of brain computer interface for rich multi-media environment. International Journal of Communication Systems, 34(6): e4584. DOI: https://doi.org/10.1002/dac.4584.

LATUSZYNSKA, M., 2012. Computer simulation methods in economics and management. Actual Problems of Economics, 131: 170–179.

LAW, A.M., KELTON, D.W., 1991. Simulation modelling and analysis. 2nd ed, McGraw-Hill: New York.

LI, X., LING, J., SHEN, Y., LU, T., FENG, S., ZHU, H., 2021. The impact of CCT on driving safety in the normal and accident situation: A VR-based experimental study. Advanced Engineering Informatics, 50: 101379. DOI: 10.1016/J.AEI.2021.101379.

LIN, C.-T., HUANG, T.-Y., LIANG, W.-C., CHIU, T.-T., CHAO, C.-F., HSU, S.-H., KO, L.-W., 2009. Assessing Effectiveness of Various Auditory Warning Signals in Maintaining Drivers’ Attention in Virtual Reality-Based Driving Environments. Perceptual and Motor Skills, 108(3): 825–835. DOI: 10.2466/pms.108.3.825-835.

LIN, Y., LENG, H., CAI, H., 2006. A Study on Driver’s Physiological Non-Intrusive Measurement in a Virtual Environment. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(26): 2707–2711. DOI: 10.1177/154193120605002608.

MCFARLAND, D.J., WOLPAW, J.R., 2017. EEG-based brain–computer interfaces. Current Opinion in Biomedical Engineering, 4: 194–200. DOI: https://doi.org/10.1016/j.co bme.2017.11.004.

MERT, A., AKAN, A., 2018. Emotion recognition based on time–frequency distribution of EEG signals using multivariate synchrosqueezing transform. Digital Signal Processing, 81: 106–115. DOI: 10.1016/J.DSP.2018.07.003.

MITA, A., SATO, H., KAMEDA, H., 2010. Platform for structural health monitoring of buildings utilizing smart sensors and advanced diagnosis tools. Structural Control and Health Monitoring, 17(7): 795–807. DOI: https://doi.org/10.1002/stc.399.

NOOR, A.K., ARAS, R., 2015. Potential of multimodal and multiuser interaction with virtual holography. Advances in Engineering Software, 81: 1–6. DOI: 10.1016/J.ADVENGSOFT.2014.10.004.

PFURTSCHELLER, G., GRAIMANN, B., NEUPER, C., 2006. EEG-Based Brain-Computer Interface System. In: Wiley Encyclopedia of Biomedical Engineering, American Cancer Society. ISBN 9780471740360.

QUIRÓS, A., WILSON, S.P., DIEZ, R.M., SOLANA, A.B., HERNÁNDEZ TAMAMES, J.A., 2015. Brain activity detection by estimating the signal-to-noise ratio of FMRI time series using dynamic linear models. Digital Signal Processing, 47: 205–211. DOI: 10.1016/J.DSP.2015.06.008.

SCHROETER, R., GERBER, M.A., 2018. A Low-Cost VR-Based Automated Driving Simulator for Rapid Automotive UI Prototyping. In: Proceedings of the Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Association for Computing Machinery: New York, NY, USA, 248–251.

TAHERI, S.M., MATSUSHITA, K., SASAKI, M., 2017A. Development of a Driving Simulator with Analyzing Driver’s Characteristics Based on a Virtual Reality Head Mounted Display. Journal of Transportation Technologies, 7(3): 351–366. DOI: https://doi.org/10.4236/jtts. 2017.73023.

TAHERI, S.M., MATSUSHITA, K., SASAKI, M., 2017B. Virtual Reality Driving Simulation for Measuring Driver Behavior and Characteristics. Journal of Transportation Technologies, 7(2): 123–132. DOI: 10.4236/jtts.2017.72009.

WANG, Y., XU, G., ZHANG, S., LUO, A., LI, M., HAN, C., 2017. EEG signal co-channel interference suppression based on image dimensionality reduction and permutation entropy. Signal Processing, 134(C): 113–122. DOI: 10.1016/J.SIGPRO.2016.11.015.

WEI, C.-S., CHUANG, S.-W., WANG, W.-R., KO, L., JUNG, T., LIN, C.-T., 2011. Implementation of a motion sickness evaluation system based on EEG spectrum analysis. 2011 International Symposium on Circuits and Systems, 1081–1084.

WU, H., YANG, G., ZHU, K., LIU, S., GUO, W., JIANG, Z., LI, Z., 2021. Materials, Devices, and Systems of On-Skin Electrodes for Electrophysiological Monitoring and Human–Machine Interfaces. Advanced Science, 8(2): 2001938. DOI: https://doi.org/10.1002/advs. 202001938.

XIONGQING, P., HU, S., ZHIQIANG, W., YANG, Y., 2018. A Vehicle Driving Simulator Based on Virtual Reality. CICTP, 2087–2097.