Skip to content
BY 4.0 license Open Access Published by De Gruyter Open Access November 30, 2021

A novel strategy for driving car brain–computer interfaces: Discrimination of EEG-based visual-motor imagery

  • Zhouzhou Zhou , Anmin Gong , Qian Qian , Lei Su , Lei Zhao EMAIL logo and Yunfa Fu EMAIL logo

Abstract

A brain–computer interface (BCI) based on kinesthetic motor imagery has a potential of becoming a groundbreaking technology in a clinical setting. However, few studies focus on a visual-motor imagery (VMI) paradigm driving BCI. The VMI-BCI feature extraction methods are yet to be explored in depth. In this study, a novel VMI-BCI paradigm is proposed to execute four VMI tasks: imagining a car moving forward, reversing, turning left, and turning right. These mental strategies can naturally control a car or robot to move forward, backward, left, and right. Electroencephalogram (EEG) data from 25 subjects were collected. After the raw EEG signal baseline was corrected, the alpha band was extracted using bandpass filtering. The artifacts were removed by independent component analysis. Then, the EEG average instantaneous energy induced by VMI (VMI-EEG) was calculated using the Hilbert–Huang transform (HHT). The autoregressive model was extracted to construct a 12-dimensional feature vector to a support vector machine suitable for small sample classification. This was classified into two-class tasks: visual imagination of driving the car forward versus reversing, driving forward versus turning left, driving forward versus turning right, reversing versus turning left, reversing versus turning right, and turning left versus turning right. The results showed that the average classification accuracy of these two-class tasks was 62.68 ± 5.08%, and the highest classification accuracy was 73.66 ± 6.80%. The study showed that EEG features of O1 and O2 electrodes in the occipital region extracted by HHT were separable for these VMI tasks.

1 Introduction

Brain–computer interface (BCI) neural activity learned to generate user’s perception or cognitive activity patterns for controlling external devices [1]. BCI can improve the quality of life of patients suffering from severe motor disorders. It can also communicate and control modes of healthy people. BCI based on imaginative mental activity is an important brain–computer interaction. In BCI, the traditional imagery task is motor imagery (MI) [2,3], which requires subjects to feel or recall the movement of specific parts of their body, such as hands or feet [4,5,6,7,8,9,10]. However, learning and controlling MI mental activities are difficult and often require many hours of training. This could impose a burden on subjects, thereby reducing BCI users’ acceptance. The MI-BCI performance is associated with the amount of training received by the subjects. However, the classification performance of this BCI is unreliable [4,11] owing to a serious “BCI illiteracy” [12,13,14]. This can affect the performance and application of the BCI.

Many earlier studies have achieved remarkable results on the MI mental activity (the neurological principle of brain structure and function) [15,16,17] and on analyzing and decoding the algorithm of brain signals related to MI [15]. However, BCI is limited to laboratory demonstrations, and its practical application is difficult [16]. Therefore, MI may be unsuitable in controlling BCI, especially for patients with motor disorders [15]. Using the MI method, it will be difficult to mentally rehearse the patients’ movement process.

Compared with the above-mentioned MI mental activity, visual motor imagery (VMI) is a kind of mental activity that is easy to learn and control. It requires subjects to view a specific picture or scene in their brain from a third-person perspective [4,8,9,10]. The novel aspect of VMI is that it only requires fewer hours of training [15], and people will be proficient within a few hours since they often carry out such mental activities in their daily life. In addition, VMI is a natural mental activity similar to children viewing their mother’s face and adults viewing portraits. Thus, it is easy to reflect the image of an object in our brain and to mentally manipulate the object with various movements. We can also recall a movie or a social scene in our daily life. In particular, it is easy to reflect the movements of other people’s limbs in our brain. Thus, VMI activities contrast to MI activities, which require subjects to mentally rehearse movements of specific parts of their body but not the actual execution of a movement. That is, subjects must mentally rehearse an action and at the same time prevent the action from happening [4,5,12,13,14,16]. This is a contradictory confrontational process and an unnatural mental activity. This is difficult to implement and control as people rarely carry out such mental activities in their daily life. Compared with MI, VMI is a better mental task to control BCI [15]. Therefore, this study designs a new VMI-BCI experimental paradigm to identify the VMI tasks for BCI.

Compared with the traditional MI-BCI, few studies focus on VMI-BCI [15]. Neuper et al. [18] explored kinesthetic MI, VMI, movement execution, and movement observation. Frequency band and electrode position are extracted as features and classified using the distinction sensitive learning vector quantization algorithm. From the results, the average classification accuracy of movement execution and movement observation is approximately 80%, the average classification accuracy of kinesthetic MI is approximately 67%, and the average classification accuracy of VMI is approximately 56%. Azmy and Safri [19] used BCI based on visual imagery electroencephalogram (EEG) to control a robot. In extracting the power spectrum features, the F8 electrode located in the frontal region is the best position for detecting VMI EEG, which responds to the oscillation rhythm in the alpha band. However, the classification accuracy and classifier are absent. Sousa et al. [20] classified three visual imagery tasks of static points, dynamic points moving vertically in two directions, and dynamic points moving in four directions of up, down, left, and right. The power spectrum energy feature is extracted and classified by a support vector machine (SVM) with an average classification accuracy of 87.64%. Kosmyna et al. [15] classified the visual imagination of predefined flowers and hammers. The feature of the power spectrum is extracted and classified by spectral weighed common spatial patterns. From the results, the classification accuracy is 52% of the visual imagination flower versus the visual imagination hammer. These studies on VMI-BCI limit the extracted features to the power spectrum, and it is necessary to introduce a new feature extraction method.

Traditional EEG feature extraction methods are autoregressive (AR) model, adaptive model, wavelet transform, and common space model. The Fourier transform wavelet method uses sinusoidal components of different frequencies to fit the original signal. It is difficult for the wavelet method to achieve a high resolution in both the time domain and the frequency domain [21]. Compared with the wavelet transform, the Hilbert–Huang transform (HHT) can obtain high-resolution information simultaneously in the time domain and the frequency domain. HHT is a new signal processing method, proposed by Huang et al., suitable for nonlinear and nonstationary signal analysis [21]. EEG is a highly nonlinear and nonstationary signal. Sun et al. [22] show that EEG feature extraction for MI based on HHT is superior to the traditional wavelet transform and the feature extraction method without HHT processing. Thus, this study attempts to introduce HHT into VMI-BCI feature extraction to verify the effectiveness of the method.

The rest of the article is organized as follows. Section 2 introduces the materials and methods in designing a novel VMI-BCI experimental paradigm; Section 3 presents the results; Section 4 provides the discussions; and Section 5 concludes the article.

2 Materials and methods

2.1 Subjects

Using the kinesthetic and visual-motor imagery questionnaire (KVMIQ) [23,24,25,26,27,28,29,30,31,32,33,34,35], a total of 30 healthy subjects (numbered sub1–sub30) with strong visual imagination (question score ≥70) were recruited to participate in data collection. All subjects were male, right handed, and aged 25 ± 1 years. Their visual acuity was normal or corrected to normal, and they have no mental illness. None of the subjects had taken part in this type of experiment.

  1. Informed consent: Informed consent has been obtained from all individuals included in this study.

  2. Ethical approval: The research related to human use has been complied with all the relevant national regulations, institutional policies, and in accordance the tenets of the Helsinki Declaration and has been approved by the Medical Ethics Committee of Kunming University of Science and Technology.

2.2 Experimental paradigm, process, and settings

2.2.1 Visual imagination paradigm design

To control the car or robot to move forward, move backward, turn left, and turn right naturally by mental activity, four VMI tasks were designed: imagine the car moving forward, reversing, turning left, and turning right as shown in Figure 1. The test began with a fixed cross on the screen for 10 s, prompting the subjects to start the experiment. The fixed cross disappeared, and the screen showed the animation of the car moving forward for 5 s. The subjects must observe the animation (i.e., visual observation) and pay attention to the direction of the car moving forward. When the prompted animation disappeared, the screen was empty. For 5 s, the subjects must retain the visual mental imagery of the prompted animation from a third-person perspective (i.e., visual imagination). Afterward, the screen presents an animation of the car reversing for 5 s, and the subjects were asked to observe the animation and pay attention to the car’s backward direction. After the prompted animation disappeared, the screen was blank, and the subjects were asked to perform the animation visual imagination for 5 s. By analogy, the task of visual observation versus visual imagination is prompted to turn left and right. Finally, the vehicle is prompted to rest for 5 s and complete a block test. Each participant completed 50 blocks and 50 trials for visual observation versus VMI of the car moving forward, reversing, turning left, and turning right.

Figure 1 
                     Visual imagination experimental paradigm (animated in the experiment).
Figure 1

Visual imagination experimental paradigm (animated in the experiment).

2.2.2 Experimental process

Subjects were allowed to have enough sleep before the experiment to maintain a good mental state. Before the experiment, the main tester, the EEG acquisition equipment, was connected, and the relevant software was launched. Subjects were seated in comfortable chairs, about 70 cm away from the visual cue screen, with their hands flat on the table. The tester wears a brain cap and injects the brain electric cream into the subjects until the electrode impedance falls to a suitable range. In the beginning, the subjects completed the visual observation and visual imagery tasks according to the prompts of Figure 1. Subjects were asked to avoid movement and blinking during the VMI tasks.

2.2.3 Experimental settings

The experimental setup is shown in Figure 2. Earlier studies [15,18,20] revealed that the VMI was correlated with the right frontal cortex and parietal cortex. In this study, the designed schematic of the electrode layout is shown in Figure 2(a). Fp2, F7, F8, F3, F4, Fc3, Fc4, C3, C4, P3, P4, O1, O2, Fz, Fcz, Cz, Pz, and Oz are 18 recording channels, and the reference electrodes are A1 and A2. Figure 2(b) shows the experimental setup. The TCL 24 inch LCD is used for the VMI task, a Lenovo ThinkPad computer with a Windows 10 operating system is used for data acquisition and processing, and MATLAB is installed on the computer for data acquisition and data processing. The EEG amplifier is the NT9200 series of Beijing Zhongke Xintuo Instrument Co., Ltd., (Beijing, China) with a sampling rate of 1,000 Hz and a 45 Hz low-pass filter.

Figure 2 
                     Electrode layout and experimental setup: (a) electrode layout (the reference electrodes are A1 and A2) and (b) (experimental setup (a) is the subject, (b) is the brain electrical amplifier, (c) is the data acquisition and processing computer, and (d) is the tip for visual imagination tasks).
Figure 2

Electrode layout and experimental setup: (a) electrode layout (the reference electrodes are A1 and A2) and (b) (experimental setup (a) is the subject, (b) is the brain electrical amplifier, (c) is the data acquisition and processing computer, and (d) is the tip for visual imagination tasks).

2.3 Visual imagination EEG signal processing

2.3.1 Data preparation

The collected visually imagined EEG data were examined visually, and the data of five subjects with serious EEG contamination were removed. Then, the EEG data of each subject, collected under different tasks, were extracted. The duration of each task was 5 s. To prevent interference of other signals, only the data segments from 1 s (corresponding to 0 s in Figure 3(a)) to 4 s after the task are extracted and analyzed. Figure 3(a) shows the visually imagined original EEG signals of the O1 versus O2 channels during forward and reverse driving of the car.

Figure 3 
                     The original EEG signal of VMI, the total average waveform contrast of rest and VMI, the waveform after preprocessing, and the waveform after classification. (a) The original EEG signal of the O1 and O2 channels during visual imagery of the car forward versus reverse driving (a is the original EEG signal of the car driving forward and b is the original EEG signal of the car reverse). (b) A comparison of the total average waveform of the resting state versus the visual imagination. (c) (The waveforms of the O1 and O2 channels undergoing baseline drift correction, 8–13 Hz bandpass filtering, and ICA processing during visually imagined car forward versus reverse driving (a) is the waveform after processing when the car is driven forward and (b) is the waveform after processing when reversing)). (d) The average classification accuracy of the sub1 visually imagined car traveling forward versus reverse.
Figure 3

The original EEG signal of VMI, the total average waveform contrast of rest and VMI, the waveform after preprocessing, and the waveform after classification. (a) The original EEG signal of the O1 and O2 channels during visual imagery of the car forward versus reverse driving (a is the original EEG signal of the car driving forward and b is the original EEG signal of the car reverse). (b) A comparison of the total average waveform of the resting state versus the visual imagination. (c) (The waveforms of the O1 and O2 channels undergoing baseline drift correction, 8–13 Hz bandpass filtering, and ICA processing during visually imagined car forward versus reverse driving (a) is the waveform after processing when the car is driven forward and (b) is the waveform after processing when reversing)). (d) The average classification accuracy of the sub1 visually imagined car traveling forward versus reverse.

2.3.2 Preprocessing

First, the baseline drift of the extracted VMI EEG signal is corrected to eliminate the EEG signal deviation from the baseline [13]. According to Kosmyna et al. [15], the visual imaginary EEG signal is related to the alpha band and elliptic filters with 8–13 Hz bandpass. Then, independent component analysis (ICA) [36,37,38,39] removed the electro-ocular artifacts and electromyographic artifacts. Figure 3(c) shows the waveform after the above preprocessing.

2.3.3 HHT-based feature extraction and SVM classification

The HHT method includes empirical mode decomposition (EMD) and Hilbert spectrum analysis (HSA). The EMD obtains the intrinsic mode function (IMF) using the average of the upper and lower envelopes of the time series. The EMD process is as follows: first, all of the maximum and minimum points of the input signal are obtained. Then, the maximum value point and the minimum value point are fitted using a cubic spline. The curve of the upper and lower envelopes is obtained, and the mean value function is calculated to obtain the difference h between the analyzed signal and the mean value. Second, we examine whether h meets the IMF condition, and if it does, we treat h as the first IMF. Otherwise, the first two steps are obtained until the kth step satisfies the IMF condition. Then, we obtain the first IMF and calculate the difference r between the original signal and the IMF. Finally, the difference r is the signal decomposed until the remaining r is a monotonic signal or one pole [22]. From the above analysis, the expression of the original signal is shown in equation (1) [22]:

(1) s ( t ) = i = 1 N C i ( t ) + R n ( t ) ,

where s(t) is the original signal, C i (t) is the IMF component obtained by the ith screening, N is the number of screening times, and R n (t) is the final residual component.

All extractable IMFs are obtained after EMD, and HSA is performed. Equation (2) [22] is a Hilbert spectral transformation for each IMF component:

(2) Y i ( t ) = 1 π + C i ( φ ) t φ d φ .

The analytical signal of the original signal is as shown in equation (3) [22]:

(3) X i ( t ) = C i ( t ) + j Y i ( t ) = A i ( t ) e j p i ( t ) .

The instantaneous amplitude and the instantaneous phase are obtained from equations (4) and (5), respectively [22]:

(4) A i ( t ) = Y i ( t ) 2 + C i ( t ) 2 ,

(5) P i ( t ) = arctan Y i ( t ) C i ( t ) .

The average instantaneous energy is obtained from the instantaneous amplitudes according to equations (6) and (7) [22]:

(6) EC N = 1 N n = 1 N C n 2 , N < F s ,

(7) EC N = 1 N n = N F s + 1 N C n 2 , N > F s .

The left side of equations (6) and (7) is the instantaneous energy value.

After EMD decomposition of the EEG signals of VMI, the IMF of each order is shown in Figure 4. Related studies have shown that [22] the IMF first three orders contribute to classification. The IMF first three orders are combined, and the average instantaneous amplitude is obtained using a Hilbert transform. The sixth-order AR model coefficients AR1–AR6 are extracted using the Burg algorithm. The 12-dimensional feature vector comprises the AR model coefficients of the O1 and O2 channels: {O1AR1, …, O1AR6, O2AR1, …, O2AR6}. Then, the SVM suitable for small sample classification is used for feature classification [40]. The classification was carried out using the MATLAB SVM library function and selected with Gauss Kernel Function. A total of 90% of each subject’s data were used to train the SVM model. Thus, 10% of the dataset is used to test the SVM model. A 10-fold cross-validation was implemented to evaluate the experimental performance. Figure 3(d) shows the average classification accuracy of the sub1 visual imaginary car driving forward versus reversing.

Figure 4 
                     IMF of each order after EMD decomposition of the VMI EEG signal (the vertical axis represents the amplitude in microvolts).
Figure 4

IMF of each order after EMD decomposition of the VMI EEG signal (the vertical axis represents the amplitude in microvolts).

3 Results

Figure 3 shows the original EEG signal of VMI, the total average waveform contrast of rest and VMI, the waveform after preprocessing, and the waveform after classification. Figure 3(a) shows the original EEG signal of the O1 and O2 channels during visual imagery of forward driving versus reverse driving (a is the original EEG signal of the car driving forward and b is the original EEG signal of the car driving in reverse). Figure 3(b) shows a comparison of the total average waveform of the resting state versus visual imagination. Figure 3(c) shows the waveforms of the O1 and O2 channels undergoing baseline drift correction, 8–13 Hz bandpass filtering, and ICA processing during visual car forward versus reverse driving (where a is the waveform after the process when the car is driven forward and b is the waveform after the process when reversing). Figure 3(d) shows the average classification accuracy of the sub1 visually imagined car traveling forward versus reversing. From Figure 3(b), the EEG of VMI versus rest has two obvious voltage reversals of approximately 200 ms in the position of a and b.

Figure 4 shows the IMF of each order after EMD VMI EEG signal decomposition. As shown in Figure 4, when the waveform is decomposed to the eighth layer, there is only one pole in the waveform, and the decomposition stops. At the same time, the first three orders of IMF contain the required time-frequency information, and the fourth-order IMF contains almost no time-frequency information, consistent with the theoretical analysis in Section 2.3.

Table 1 shows the results of a two-way ANOVA analysis of EEG in each lead under four visual imagery tasks. The mean values of EEG of 25 subjects were calculated in four VMI tasks. The two factors were the type and lead of the visual imagery task. Table 1 shows the mean of visual imagery EEG, revealing that the main effects of electrodes Fp2, F7, P3, O1, Fcz, and Pz are significant (P < 0.05), whereas the main effects of other leads are insignificant.

Table 1

The results of the EEG analysis of each lead in four VMI tasks using two-way ANOVA (FDR correction)

Leads df Mean square F P
Fp2 3 2811.906 6.879 0.000***
F7 3 1153.996 5.078 0.009**
F8 3 700.248 2.784 0.097
F3 3 496.798 2.476 0.118
F4 3 260.193 1.142 0.413
Fc3 3 256.310 1.472 0.330
Fc4 3 390.191 1.932 0.200
C3 3 158.137 1.191 0.413
C4 3 22.740 0.184 0.907
P3 3 1376.669 5.823 0.006**
P4 3 658.243 2.725 0.097
O1 3 903.591 3.485 0.045*
O2 3 86.344 0.386 0.808
Fz 3 216.104 1.108 0.413
Fcz 3 735.923 3.626 0.043*
Cz 3 480.581 2.078 0.182
Pz 3 1518.675 6.385 0.000***
Oz 3 80.270 0.658 0.650

Notes: *P < 0.05, **P < 0.01, and ***P < 0.001.

To verify the VMI movement direction as a mental strategy for BCI users, we carried out the feature extraction and pattern classification of EEG collected during car forward driving and reverse driving of VMI. Table 2 presents the average classification accuracy under different combinations of electrodes. The average classification accuracies of P3 versus P4, O1 versus O2, F7 versus F8, C3 versus C4, F3 versus F4, and Fc3 versus Fc4 were 59.9666 ± 6.7385, 64.7673 ± 4.2748, 53.5205 ± 2.5087, 51.9838 ± 1.8770, 51.5868 ± 1.9974, and 57.6417 ± 5.7230%, respectively. We selected few electrodes for classification as our aim was to explore a less-channel VMI-BCI brain-controlled robotic system. Thus, we carried out only two classifications.

Table 2

Average classification accuracy under different electrode combinations

Subjects Combination of P3 versus P4 (%) Combination of O1 versus O2 (%) Combination of F7 versus F8 (%) Combination of C3 versus C4 (%) Combination of F3 versus F4 (%) Combination of Fc3 versus Fc4 (%)
sub1 62.1104 65.1284 55.0210 50.1001 57.0581 51.1177
sub2 53.1567 72.0869 57.0605 51.1284 55.0723 55.1079
sub3 54.1616 62.0781 55.1392 51.1729 50.0830 70.0967
sub4 70.0840 63.1265 51.1157 53.1538 50.0249 53.0591
sub5 69.0127 65.1157 53.0098 55.1880 51.0435 58.0181
sub6 56.0708 64.0430 51.1265 51.2153 50.0835 70.0620
sub7 58.1177 66.0664 52.1313 50.1846 50.1572 64.0684
sub8 63.0933 65.0854 53.1152 50.1709 51.1211 62.0034
sub9 53.1284 65.0898 52.0215 51.1553 50.1104 55.0767
sub10 52.1074 63.1245 63.0771 51.1055 52.0137 54.1226
sub11 53.1006 59.1787 55.1221 52.0249 56.0103 55.1484
sub12 65.0493 62.1885 52.1377 51.0049 52.0693 63.1265
sub13 54.0244 73.1714 53.0918 53.0391 51.0444 52.0244
sub14 54.1465 55.1118 54.1040 51.0049 50.0439 58.1138
sub15 68.1597 69.0474 51.1392 54.1064 50.0381 54.1489
sub16 69.1919 72.0327 51.0703 52.1079 50.0918 62.0884
sub17 58.0954 68.0562 53.0278 51.1113 51.1182 66.0752
sub18 52.0278 65.1812 52.0249 53.1587 53.1245 51.0562
sub19 69.0059 67.2466 52.1025 51.1665 54.1011 53.0142
sub20 58.0527 62.2261 53.1382 51.0933 51.1265 55.0322
sub21 57.0342 61.1816 52.1436 51.0044 50.0776 62.1289
sub22 62.0791 69.1982 54.0620 57.0522 50.0254 54.1582
sub23 52.0586 59.1909 55.0054 56.0874 51.0010 51.0806
sub24 72.0166 62.1611 54.0093 51.0527 52.0171 56.0625
sub25 64.0815 63.0669 53.0176 50.0073 51.0151 55.0532
Average classification rate 59.9666 64.7673 53.5205 51.9838 51.5868 57.6417
Variance 6.7385 4.2748 2.5087 1.8770 1.9974 5.7230

Table 3 shows the descriptive statistical results of EEG in each lead in four VMI tasks. Table 4 shows the results of principal effect analysis using a two-way ANOVA of different VMI tasks in Table 3. Table 4 shows the different leads between groups, which visually imagined the car moving forward, reversing, turning left, and turning right. Table 4 showed that the differences between groups of the Fp2, F7, P3, O1, Fcz, and Pz leads, revealing that they were significant (P < 0.05).

Table 3

The EEG descriptive statistical results in each lead during the four visual imagery tasks, with the standard deviation in parentheses and the mean value outside parentheses

Leads Visual imagination driving forward Visual imagination reversing Visual imagination turning left Visual imagination turning right
Fp2 0.150 (26.75) 0.789 (14.09) 0.529 (19.02) 0.804 (18.94)
Pz 0.529 (16.08) 0.729 (14.84) 0.338 (14.95) 0.850 (15.76)
P3 0.258 (17.80) 0.657 (12.61) 0.405 (15.37) 0.713 (15.27)
F7 0.463 (16.63) 0.738 (13.53) 0.456 (14.83) 0.844 (15.13)
Fcz 0.453 (15.18) 0.704 (13.38) 0.455 (13.94) 0.744 (14.40)
O1 0.460 (17.81) 0.739 (14.50) 0.563 (15.82) 0.847 (16.09)
Table 4

Results of the main effect analysis of four visual imagery tasks under different leads (FDR correction)

Leads Source Sum of squares Mean square F P
Fp2 Between groups 1813.333 5.965 4.818 0.000
Intergroup 148186.667 1.238
Total 150000.000
Pz Between groups 422.081 2.833 2.270 0.000
Intergroup 149577.919 1.248
Total 150000.000
P3 Between groups 800.732 3.906 3.136 0.006
Intergroup 149199.268 1.245
Total 150000.000
F7 Between groups 662.188 3.638 2.919 0.009
Intergroup 149337.812 1.246
Total 150000.000
Fcz Between groups 453.109 3.001 2.405 0.043
Intergroup 149546.891 1.248
Total 150000.000
O1 Between groups 671.761 3.499 2.807 0.045
Intergroup 149328.239 1.246
Total 150000.000

Table 5 presents an average classification of 25 subjects visually imagining the car’s forward versus reverse, forward versus left, forward versus right, reverse versus left, reverse versus right, and left versus right movements. The average classification accuracies are 64.76 ± 4.27, 56.49 ± 4.63, 55.78 ± 3.09, 61.11 ± 4.56, 64.30 ± 7.13, and 73.66 ± 6.80%, respectively. The electrode combination is O1 versus O2.

Table 5

Average classification accuracy of different visual imagery task pairs for 25 subjects under electrode combinations O1 and O2

Subjects Visual imagination driving forward versus reversing (%) Visual imagination driving forward versus turning left (%) Visual imagination driving forward versus turning right (%) Visual imagination reversing versus turning left (%) Visual imagination reversing versus turning right (%) Visual imagination left turn versus right turn (%)
sub1 65.1284 59.1079 60.0303 62.0361 63.0684 73.0737
sub2 72.0869 57.1860 55.0093 58.0269 58.1099 78.1523
sub3 62.0781 58.2417 52.0151 65.1533 74.0347 82.0259
sub4 63.1265 68.2012 54.0254 56.0220 56.0742 63.0039
sub5 65.1157 57.1509 53.0610 66.0933 72.0630 59.1274
sub6 64.0430 68.0151 55.0869 63.0981 65.1328 79.0356
sub7 66.0664 57.0659 56.0313 59.0737 73.0708 68.0308
sub8 65.0854 55.0449 58.0269 63.0815 66.0117 63.0020
sub9 65.0898 55.0352 58.1108 55.0010 64.0283 76.1270
sub10 63.1245 56.0083 61.0962 57.0933 71.0986 75.1089
sub11 59.1787 50.0376 55.1216 65.0200 51.0210 75.0151
sub12 62.1885 58.1094 60.0410 56.0229 50.0469 67.0474
sub13 73.1714 58.1250 53.0293 60.0015 59.0454 68.0166
sub14 55.1118 57.0879 54.0010 68.1064 73.0327 82.0073
sub15 69.0474 58.1299 56.1240 65.0444 65.0132 79.0547
sub16 72.0327 59.2061 53.0229 54.0161 66.0156 73.0576
sub17 68.0562 51.0029 55.0381 59.1025 72.0962 67.0610
sub18 65.1812 50.0518 56.1567 73.1294 59.0869 68.1665
sub19 67.2466 50.1187 52.1240 65.0840 55.0747 75.1328
sub20 62.2261 59.0928 59.0669 62.0684 67.1313 85.0654
sub21 61.1816 57.0010 52.0444 57.0947 59.0229 77.0098
sub22 69.1982 55.1304 61.0200 59.0205 73.0957 76.0820
sub23 59.1909 55.1714 56.0601 62.1230 61.1050 83.0894
sub24 62.1611 52.0532 59.1353 59.1401 71.1045 71.1309
sub25 63.0669 51.0503 50.1255 58.1221 63.0928 78.0713
Average classification rate 64.7673 56.4970 55.7841 61.1110 64.3070 73.6678
Variance 4.2748 4.6322 3.0904 4.5623 7.1310 6.8068

Figure 5 presents the correlation of 18 channels of EEG during VMI to examine the correlation between the signals of the channel in relevant brain regions during visual imagery. The thickness and color of the connection between the two nodes show the degree of their correlation. Figure 5 shows that O1 and F7 have the strongest connectivity during the car reversal period of the visual imagination. In the car left turn, the connection between O2 and F8 is the strongest. In the car right turns, the connection between O2 and Fp2 is the strongest. Thus, Cz and F7 have the strongest connectivity when the car is moving forward.

Figure 5 
               The correlation of 18 channels of EEG during VMI: (a) the correlation of the lead when the car moves forward in the visual imagination; (b) the correlation of the lead when the car reverses; (c) the correlation of the lead when the car turns left; and (d) the correlation of the lead when the car turns right.
Figure 5

The correlation of 18 channels of EEG during VMI: (a) the correlation of the lead when the car moves forward in the visual imagination; (b) the correlation of the lead when the car reverses; (c) the correlation of the lead when the car turns left; and (d) the correlation of the lead when the car turns right.

4 Discussion

Many earlier studies have made a remarkable research progress on MI-BCI [15,16,17]; however, research on VMI-BCI is slow. The design of BCI’s paradigm and feature extraction needs to be further explored. This article focuses on the VMI-BCI paradigm and feature extraction. At present, few studies have explored predefined VMI (guided by prompting visual observation). A research on this VMI is challenging, and it is necessary to design an appropriate experimental paradigm to evaluate the visual imagination ability of subjects and identify different visual imagination tasks. In this study, a new VMI-BCI paradigm was designed, in which subjects were asked to complete four VMI tasks: imagine a car moving forward, reversing, turning left, and turning right, and then, we use HHT to extract features and classify them.

The experimental paradigm is different from that of Nataliya et al. [15], which requires subjects to visually imagine static flowers versus hammers, whereas this study requires subjects to visually imagine dynamic cars. Sousa [20] showed that dynamic visual imagination induces stronger brain activation than static visual imagination. The classification results of the different visual imagery tasks designed in this study (the average classification accuracy of the left versus right turns of the visual imagination car is 73.66 ± 6.80%) indicate that the experimental paradigm has good separability. Table 4 indicates that the visually imaginary car can be distinguished from forward, backward, left, and right turns by selecting the appropriate lead to extract features. This shows that the experimental paradigm of visual imagination is feasible. Second, the experimental paradigm is different from other experimental paradigms [15,18,19,20] based on the visual imagination of the same object (car), but the direction of movement is different. This design explores the difference of visual imagination in controlling direction variables. However, this design poses a challenge to feature extraction and classification. Better methods are needed to obtain classification results. In contrast, different driving directions of the VMI car can naturally correspond with the forward, backward, left, and right turn of a control robot or a car, which may provide the practical application of visual-motor-imagery-based brain-controlled robot or car.

The EEG pattern induced by the VMI task is also closely related to the performance of subjects’ imaginative mental activities [4]. To this end, 25 healthy volunteers with strong VMI ability (questionnaire score ≥70) were recruited before the experiment according to the VMIQ [24,25,26,27,28,29,30,32] or the KVMIQ [23,31,33,34,35]. After the experiment, each participant was asked to fill in a questionnaire of the visual imagery car movement direction task. Compared with performing MI tasks, most subjects consider VMI tasks to be a mental task, which is easier to accomplish. Although all subjects did not perform specialized VMI training, they can complete the VMI task better.

For the feature extraction of VMI-BCI, the EEG power spectrum estimation method is mainly used in the existing research [15,18,19,20,41,42], whereas the HHT method, which can obtain high-resolution information in both the time domain and the frequency domain, is used to extract features in this article. Thus, SVM was used to classify the forward and reverse, forward and left turn, forward and right turn, reverse and left turn, reverse and right turn, and left and right turn. Among them, the highest average classification accuracy is 73.66 ± 6.80%, and the average classification accuracy is 62.68 ± 5.08%. The average classification accuracy of this study is 6.68% points higher than that of Neuper et al. [18], in which the VMI is 56%. Compared with Kosmyna et al. [15], the average classification accuracy of the visual imagination flower and the visual imagination hammer is 52%, improved by 10.68 percentage points. This indicates that the features of VMI-EEG extracted from HHT have certain separability for VMI tasks.

When extracting the EEG features of VMI using HHT, the selection and combination of electrodes have an impact on the classification accuracy of VMI tasks. Table 1 indicates that the EEG mean values of the electrodes Fp2, F7, P3, O1, Fcz, and Pz are different from the designed VMI task. Figure 5 illustrates the strongest connection between the electrode pairs O1 and F7, O2 and F8, O2 and Fp2, and Cz and F7 during visual imaginary car reversing, left turn, right turn, and forward travel. The classification results in Table 2 showed that the classification accuracy of the combination of O1 versus O2 electrodes was the highest. The two-way ANOVA in Tables 3 and 4 showed that the interaction between the O1 and O2 electrodes was significant during the VMI (P = 0.045, P < 0.05). The first reason for these results is that VMI is related to memory, whereas memory is related to the prefrontal cortex and the endothelial layer of the lateral interior of the parietal lobe (LIP area) [4,7]. The electrodes Fp2, F7, Fcz, P3, and Pz are located in these cortex areas. There are two visual information processing pathways: one is from the dorsal striate cortex to the parietal lobe, which mainly analyzes motion vision, and the second is from the ventral projection to the temporal lobe that mainly recognizes objects [4]. Therefore, the occipital region participates in the neuroprocessing of the VMI and can analyze the EEG features of the O1 versus O2 electrodes to identify the VMI task.

The EEG features of electrodes Fp2, F7, Fcz, P3, and Pz in VMI have been analyzed, and the correlation between channel EEG and VMI [4,5,6,7,8,9,10,15,19,20] has been calculated. However, the EEG features of O1 and O2 electrodes in VMI have not been analyzed and extracted. The EEG features of O1 versus O2 electrodes in VMI were analyzed and extracted using HHT. The classification results in Table 5 showed that the designed VMI tasks can be distinguished.

Compared with the traditional MI-BCI with more research, the task based on EEG recognition VMI is more challenging. The research is still insufficient, and the classification accuracy needs to be further improved. For this reason, feature selection and extraction and electrode combination optimization should be studied.

In our future work, we will attempt to extract the correlation between channel EEGs in VMI as a feature to optimize the combination of electrodes and to improve the classification accuracy of VMI tasks. We will construct an online real-time VMI-BCI brain-controlled robot system with few channel EEGs.

5 Conclusion

Contrary to the traditional MI-BCI paradigm, this study designed a new VMI-BCI paradigm: visual imagination of a car moving forward, reversing, turning left, and turning right. These four mental strategies can control the car or robot to move forward, backward, left, and right. EEG features are extracted from HHT with high temporal and spatial resolution, and the designed VMI task is identified using an SVM suitable for small sample classification. Studies have shown that in the designed experimental paradigm, the above EEG extracted features at the O1 versus O2 electrode positions in the occipital region can distinguish visual imagination tasks (average classification accuracy can reach 73.66 ± 6.80%). Visual imagination car has the strongest connection between the electrode pairs O1 and F7, O2 and F8, O2 and Fp2, and Cz and F7 in reversing, turning left, turning right, and driving forward. The visual imagination driving car based on EEG is expected to be a BCI strategy, and VMI-BCI can be used as a window to observe brain function. This article can provide ideas for online real-time VMI-BCI brain-controlled robot systems based on less-channel EEG.

  1. Funding information: This work was supported by the National Natural Science Foundation of China (Nos. 82172058, 81771926, 61763022, and 62006246) and the Science Research Foundation of Yunnan Provincial Education Department (No. 2020J0064).

  2. Conflict of interest: Authors state no conflict of interest.

  3. Data availability statement: The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

[1] Fouad MM, Amin KM, El-Bendary N, Hassanien AE. Brain computer interface: a review. In Hassanien A, Azar A, editors. Brain-Computer Interfaces. Intelligent Systems Reference Library. Vol. 74. Cham: Springer; 2015.10.1007/978-3-319-10978-7_1Search in Google Scholar

[2] Pfurtscheller G, Christa N. Motor imagery and direct brain–computer communication. Proc IEEE. 2001;89:1123–34.10.1109/5.939829Search in Google Scholar

[3] Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain–computer interfaces for communication and control. Clin Neurophysiol. 2002;113:767–91.10.1017/CBO9780511545061.036Search in Google Scholar

[4] Neuper C, Scherer R, Reiner M, Pfurtscheller G. Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Cogn Brain Res. 2005;25(3):668–77.10.1016/j.cogbrainres.2005.08.014Search in Google Scholar PubMed

[5] Hall CR, Martin K. Measuring movement imagery abilities: a revision of the movement imagery questionnaire. J Ment Imag. 1997;21:143–54.Search in Google Scholar

[6] Roberts R, Callow N, Hardy L, Markland D, Bringer J. Movement imagery ability: development and assessment of a revised version of the vividness of movement imagery questionnaire. J Sport Exerc Psychol. 2008;30:200.10.1123/jsep.30.2.200Search in Google Scholar PubMed

[7] Campos A, Pe´rez MJ. Vividness of movement imagery questionnaire: relations with other measures of mental imagery. Percept Mot Skills. 1988;67:607–10.10.2466/pms.1988.67.2.607Search in Google Scholar

[8] Malouin F, Richards CL, Jackson PL, Lafleur MF, Durand A, Doyon J. The kinesthetic and visual motor imagery questionnaire (KVMIQ) for assessing motor imagery in persons with physical disabilities: a reliability and construct validity study. J Neurol Phys Ther. 2007;31:20–9.10.1097/01.NPT.0000260567.24122.64Search in Google Scholar PubMed

[9] Naito E. Controllability of motor imagery and transformation of visual motor imagery. Percept Mot Skills. 1994;78:479–87.10.2466/pms.1994.78.2.479Search in Google Scholar PubMed

[10] Callow N, Hardy L. The relationship between the use of kinesthetic imagery and different visual motor imagery perspectives. J Sport Sci. 2004;22:167–77.10.1080/02640410310001641449Search in Google Scholar PubMed

[11] Fu Y, Xu B, Li Y, Wang Y, Yu Z, Li H. Single-trial decoding of imagined grip force parameters involving the right or left hand based on movement-related cortical potentials. Chin Sci Bull. 2014;59(16):1907–16.10.1007/s11434-014-0234-5Search in Google Scholar

[12] Thompson MC. Critiquing the concept of BCI illiteracy. Sci Eng Ethics. 2019;25(4):1217–33.10.1007/s11948-018-0061-1Search in Google Scholar PubMed

[13] Vidaurre C, Blankertz B. Towards a cure for BCI illiteracy. Brain Topograph. 2010;23(2):194–8.10.1007/s10548-009-0121-6Search in Google Scholar PubMed PubMed Central

[14] Blankertz B, Vidaurre C. Towards a cure for BCI illiteracy: machine learning based co-adaptive learning. BMC Neurosci. 2009;10(Suppl 1):1–2.10.1186/1471-2202-10-S1-P85Search in Google Scholar

[15] Kosmyna N, Lindgren JT, Lécuyer A. Attending to visual stimuli versus performing visual imagery as a control strategy for EEG-based brain-computer interfaces. Sci Rep. 2018;8:13222.10.1038/s41598-018-31472-9Search in Google Scholar PubMed PubMed Central

[16] https://en.wikipedia.org/wiki/Mental_image/, February 10,2019.Search in Google Scholar

[17] Yamanoi T. Elucidation of brain activities by electroencephalograms and its application to brain computer interface. In: IEEE International Symposium on Multiple-valued Logic. IEEE; 2016.10.1109/ISMVL.2016.45Search in Google Scholar

[18] Neuper C, Scherer R, Reiner M, Pfurtscheller G. Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Cognit Brain Res. 2006;25(3):668–77.10.1016/j.cogbrainres.2005.08.014Search in Google Scholar PubMed

[19] Azmy H, Safri NM. EEG based BCI using visual imagery task for robot control. J Teknologi (Sci Eng). 2013;61(Suppl 2):7–11.10.11113/jt.v61.1628Search in Google Scholar

[20] Sousa T, Amaral C, Andrade J, Pires G, Nunes UJ, Castelo-Branco M. Pure visual motor imagery as a potential approach to achieve three classes of control for implementation of BCI in non-motor disorders. J Neural Eng. 2017;14(4):046026.10.1088/1741-2552/aa70acSearch in Google Scholar PubMed

[21] Huang NE, Shen Z, Long SR, Wu MC, Shih HH, Zheng Q, et al. The empirical mode decomposition and the Hilbert spectrum for nonlinear and nonstationary time series analysis. Proc R Soc A: Math Phys Eng Sci. 1998;454((1971):903–95.10.1098/rspa.1998.0193Search in Google Scholar

[22] Sun HW, Fu YF, Xiong X, Yang J, Liu CW, Yu ZT. Identification of EEG induced by motor imagery based on Hillert-Huang transform. Acta Automat Sin. 2015;41(9):1686–92.Search in Google Scholar

[23] Liu H, Cheng Y, Yang LI, Zhang T, Rong X. Construct validity of Chinese version of kinesthetic and visual motor imagery questionnaire. Chin J Rehabil Theory Pract. 2017;23(5):580–3.Search in Google Scholar

[24] Chara PJ, Hamm DA. An inquiry into the construct validity of the vividness of visual imagery questionnaire. Percept Mot Skills. 1989;69(1):127–36.10.2466/pms.1989.69.1.127Search in Google Scholar PubMed

[25] Marks DF. Construct validity of the vmivmidness of vmisual imagery questionnaire. Percept Mot Skills. 1989;69(2):459–65.10.2466/pms.1989.69.2.459Search in Google Scholar PubMed

[26] Marks DF. Bibliography of research utilizing the vmivmidness of vmisual imagery questionnaire. Percept Mot Skills. 1989;69(3):707–18.10.1177/00315125890693-101Search in Google Scholar

[27] Mckelvie SJ. The vividness of visual motor imagery questionnaire: commentary on the Marks-Chara debate. Percept Mot Skills. 1990;70(2):551–60.10.2466/pms.1990.70.2.551Search in Google Scholar PubMed

[28] Richardson JTE. Gender differences in the vividness of visual motor imagery questionnaire: a meta-analysis. J Ment Imag. 1995;19(3–4):177–87.Search in Google Scholar

[29] Campos A, González MA, Amor A. The Spanish version of the Vividness of visual motor imagery questionnaire: factor structure and internal consistency reliability. Psychol Rep. 2002;90(2):503.10.2466/pr0.2002.90.2.503Search in Google Scholar PubMed

[30] Malouin F, Richards CL, Jackson PL, Lafleur MF, Durand A, Doyon J. The kinesthetic and visual motor imagery questionnaire (KVMIQ) for assessing motor imagery in persons with physical disabilities: a reliability and construct validity study. J Neurol Phys Ther. 2007;31:20–9.10.1097/01.NPT.0000260567.24122.64Search in Google Scholar PubMed

[31] Allbutt J, Shafiullah M, Ling J. The relationship between self-report imagery questionnaire scores and subtypes of socially desirable responding: visual and movement imagery. J Ment Imag. 2006;30(1–2):1–20.Search in Google Scholar

[32] Campos A, Pérez-Fabello MJ. Psychometric quality of a revised version Vividness of visual motor imagery questionnaire. Percept Mot Skills. 2009;108(3):798–802.10.2466/pms.108.3.798-802Search in Google Scholar PubMed

[33] Randhawa B, Harris S, Boyd LA. The kinesthetic and visual motor imagery questionnaire is a reliable tool for individuals with Parkinson disease. J Neurologic Phys Ther. 2010;34(3):161–7.10.1097/NPT.0b013e3181e1aa71Search in Google Scholar PubMed

[34] Demanboro A, Sterr A, Anjos SM, Conforto AB. A Brazilian–Portuguese version of the kinesthetic and visual motor imagery questionnaire. Arquivos de Neuro-Psiquiatria. 2018;76(1):26–31.10.1590/0004-282x20170181Search in Google Scholar

[35] Nakano H, Kodama T, Ukai K, Kawahara S, Horikawa S, Murata S. Reliability and validity of the Japanese version of the kinesthetic and visual motor imagery questionnaire (KVMIQ). Brain Sci. 2018;8(5):79–81.10.3390/brainsci8050079Search in Google Scholar PubMed PubMed Central

[36] Li BoA. Review of brain computer interface. Digital Commun. 2013;40(4):5–8.Search in Google Scholar

[37] Srivastava G, Crottaz-Herbette S, Lau KM, Glover GH, Menon V. ICA-based procedures for removing ballistocardiogram artifacts from EEG data acquired in the MRI scanner. NeuroImage. 2005;24(1):50–60.10.1016/j.neuroimage.2004.09.041Search in Google Scholar PubMed

[38] Subasi A, Gursoy MI. EEG signal classification using PCA, ICA, LDA and support vector machines. Expert Syst Appl. 2010;37(12):8659–66.10.1016/j.eswa.2010.06.065Search in Google Scholar

[39] Song-Yun Xie, Wei-Ping Zhang, Hui Pan. Study on extraction of α rhythm in EEG with ICA method. Computer Eng Application. 2008;44(29):217–9.Search in Google Scholar

[40] Chang CC, Lin CJ. LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol (TIST). 2011;2(3):1–27.10.1145/1961189.1961199Search in Google Scholar

[41] Bullier J, Girard P, Salin PA. The role of area 17 in the transfer of information to extrastriate visual cortex. Primary Visual Cortex in Primates. US: Springer; 1994.10.1007/978-1-4757-9628-5_7Search in Google Scholar

[42] Ishai A, Haxby JV, Ungerleider LG. Visual motor imagery of famous faces: effects of memory and attention revealed by fMRI. NeuroImage. 2002;17(4):1729–41.10.1006/nimg.2002.1330Search in Google Scholar PubMed

Received: 2021-07-01
Revised: 2021-10-18
Accepted: 2021-11-02
Published Online: 2021-11-30

© 2021 Zhou Zhouzhou et al., published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 21.2.2024 from https://www.degruyter.com/document/doi/10.1515/tnsci-2020-0199/html
Scroll to top button