Spatial Vector-based Approach to the ERP Analysis as Applied to an EEG-based Discrimination of Traffic Light Signals

The purpose of the study was to assess the utility of the spatial vector-based representation of multichannel electroencephalography (EEG; when each spatial vector denotes an “instantaneous” sample of cortical activation evolving over time) in the analysis of cortical responses to visual stimulation—as opposed to the traditional, temporal vector-based approach, when vectors are associated with distinct EEG channels. This representation was used in the analysis of EEG collected in the virtual traffic light environment with the attempt to determine the color of traffic light perceived by four participants. Kruskal-Wallis (K-W) analysis of variance was implemented for selected EEG electrodes. To utilize all available information, discrimination value was evaluated next for 32dimensional EEG spatial vectors followed by modified “k nearest neighbors” (knn) classification. K-W test indicated that EEG samples at selected electrodes are different between different colors of traffic light and when observed for specific latencies. The average accuracy of a modified three-class knn classifier was approaching 60% (the random assignment would yield approximately 33%) for the specific poststimuli latencies. The proposed technique allows analyzing stimulation-synchronized cortical activity with the temporal resolution generally determined by the sampling rate of the neuroimaging modality. The discrimination value appears instrumental for predicting the clusterability of data assessed. Stimulation-evoked cortical responses are often of interest in studies of human cognition. The proposed technique may overcome the low signal-to-noise limitation of the traditional evoked response potential (ERP) analysis and possibly provide means to assess such responses under the real-time constraint.


Introduction
Understanding cognitive processes involved in critical tasks may be essential for future scientific and technological advances. Virtual technologies and simulated environments utilized during the last decade allow studying cognitive mechanisms evoked under various perception scenarios in the controlled laboratory setting rather than in the real-life scenery. Analysis of electroencephalogram (EEG) is an established and potentially accurate technique to study human cognitive tasks. Event related potentials (ERPs)-distinctive electrophysiological responses to specific (usually external) stimuli-are, among other applications, used in some neurofeedback applications (Strehl et al., 2017) and in studying visual perception (Rutiku, Aru, & Bachmann, 2016;Yigal & Sekuler, 2007).
Considering letter applications, ERPs are usually registered over visual cortices. Analyzing visual evoked potentials (VEPs)-the EEG components related to perception of visual information-permits linking brain electrical activity to visual stimulation by studying changes in EEG that occur following the stimuli (Walsh, Kane, & Butler, 2005). In particular, evidences suggest that VEPs may be related to the color that a subject perceives.
For instance, Yeh, Lee, and Ko (2013) suggest that the images with high preference color combinations (blue on white or white on blue) may produce significantly faster response in EEG (peak latency of P100 component) with significantly greater amplitude of P300 component. Wang and Zhang (2010) concluded that images of a red car result in a larger amplitude of P300 component compared to blue car images.
However, evaluating ERPs or VEPs usually requires averaging of multiple EEG epochs recorded in response to identical stimulation events and thus use of VEPs in real time may be limited. In the present report, we evaluate an alternative approach to the traditional VEP analysis that may alleviate the underlined limitation. The proposed approach will be applied to the analysis of a driver's response to images of traffic lights.

Evaluating Drivers' Response to Traffic Lights: Literature Review
Analyzing drivers' cognitive responses may help developing an in-car brain computer interface (BCI) as possible means for improving road safety. Attempts were recently made to relate the perceived color of traffic light to the subject's EEG. Bayliss and Ballard (2000) have reported VEP-based discrimination between red and yellow traffic lights. Studying the P300 component of VEPs, authors suggest that this component "occurs at red and not yellow lights" (p. 189). While reporting an average accuracy of 85%, the work leaves the perception of the third, green traffic light, unaddressed. Lin et al. (2007) have further developed the yellow or red classification, reporting similar accuracy of 85%. The authors also extended their study by including all three traffic light signals (Liang, Lin, Wu, Huang, & Chao, 2005;Lin, et al., 2008). However, since the participants were instructed to act based on the traffic light color they perceive, we may hypothesize that the cognitive response to the driving environment may be contaminated by the action-related cortical response.
More recent studies by Khaliliardali, Chavarriaga, Gheorghe, and Millán (2012) utilized the contingent negative variation (CNV) potential for detection of the anticipated visual cue, such as the instruction to either "go" or "stop." A countdown appearing on the screen was used as the contingent warning stimuli designed to involve the subjects in the anticipating state. However, watching a countdown may not be a very realistic scenario. A required action (applying either the brake or the accelerator pedal) may produce a more complex cortical response than that evoked by viewing a traffic light itself.
In the present study, participants' perception will be limited to vision only. Multichannel EEG will represent participants' cognitive response to the virtual traffic light environment in the form of static images of traffic lights. We propose utilizing an alternative, spatial vector-based, approach in the analysis of cognitive responses in terms of VEPs, which comprises the novelty of this study. We expect that different traffic light colors may be linked to the specific VEP features and that it may be further possible to classify them. Not hypothesizing regarding the origins of perceptual changes, the primary goal of the present study is to validate the proposed technique on detecting differences between the drivers' cognitive responses (via EEG) evoked by the specific traffic light color.

Material and Methods
This study followed Lamar University Institutional Review Board guidelines and was approved by the LU IRB committee. The informed consent was obtained from all study participants prior the experimentation. No identifiable personal information was retained in the research data.

Participants, Experimental Setup, and EEG Acquisition
Continuous EEG was recorded from four subjects (three males and one female, aged from 25 to 35 years) with normal color vision, corrected to normal vision, at least one year of driving experience, no history of known neurological disorders, and without being tired or sleepy during data collection. The recordings were performed using the Advanced Neuro Technology (ANT, Netherlands) system equipped with 32 electrodes positioned according to the extended International 10/20 electrode placement map. EEG was prefiltered in the 0.3-50 Hz range, notch-filtered at 60 Hz, sampled at 256 Hz, and stored in a computer for an off-line processing. One to six EEG recordings were obtained per each participant under the identical conditions; the EEG with less visible artifacts was selected for the further analysis.
Visual stimulation consisted of images of traffic light signals presented via a multimedia projector on a screen at approximately 2 meters in front of the subject. The images were covering most of the subject's visual field and were displayed for 300 ms followed by a black screen for 1500 to 2100 ms. The images of traffic light signals of different colors were displayed in a randomized order. A sample image of the red traffic light is shown in Figure 1. Similar images of green and yellow traffic light signals were also used for visual stimulation.

EEG Analysis
EEG can be viewed as a stochastic, nonstationary in time, multichannel process. Due to its low magnitude (generally up to 100 V), EEG is often a subject to various artifacts that originate either from the surroundings, body functions, equipment (external artifacts), or from within the brain (internal artifacts). These artifacts are usually reduced to improve accuracy of the EEG analysis. As an example, DC offsets are normally removed from EEG before further processing. Additionally, spatial filters that reduce surface currents may improve the signal-to-noise ratio of EEG potentially improving the classification of mental tasks (Mourino et al., 2001). One of such filters, a common average reference (CAR) spatial filter, is particularly popular due to its robustness, good performance, and computational efficiency (Ludwig et al., 2009). Therefore, CAR spatial filter was implemented in this project after DC offsets were removed from EEG.
We have recently demonstrated that the discrete wavelet transform (DWT) decomposition of the VEPs can be instrumental in their classification of the perceived color of traffic light (Hoque & Tcheslavski, 2018). However, the reported approach requires the complete VEP being available for the processing and, therefore, may not be suitable for real-time applications. Selection of the most appropriate EEG channels/electrodes contributing to the more accurate classification is another challenge. Besides, the temporal dynamics of VEPs may be of interest. Therefore, we propose an alternative approach to analyze EEG data acquired while participants were performing cognitive tasks.

The VEP Analysis: Its Limitations and an Alternative Approach
Traditionally, VEPs can be viewed as the time-locked responses that are treated as stimulation-specific. VEPs can be extracted by averaging the EEG segments (epochs) synchronized with the specific repetitive stimuli-the color of traffic light images, for instance-and for the EEG electrodes associated with the visual cortex. Since differences in cognition may be of interest, the VEP components related to both cognition and memory may be assessed. One of the objectives of the present project was to determine whether the VEP components-such as P100, P300, or N400-are elicited in a virtual traffic light environment and can be used for the traffic light color classification.
From a physics standpoint, discrete-time M-channel EEG can be viewed as a sampled in both time and space version of a continuous spatial distribution of an electric potential that evolves over time; hence a spatiotemporal distribution. From the mathematical standpoint, a discrete EEG epoch synchronized with the external (visual, in our case) stimulation can be viewed as a collection of temporal vectors, where each vector corresponds to a specific EEG channel. It is safe to assume that only the selected channels will contribute to VEPs. Another, perhaps less traditional, approach will be to consider such an epoch as a single spatial vector evolving over time.
In this case, this vector at any time instance is an instantaneous sample of continuous electric potential represented by EEG that is sampled (in space) at the individual electrodes' locations. The cartoon diagram in Figure 2 illustrates this dual representation concept. The diagram in Figure 2 shows a traditional temporal EEG vector for the channel 3 that starts at the time instance n0+1 and is 9-sample long (blue dash-dot contour) and a spatial EEG vector for all M EEG channels and for the time instance n0+3 (green dash contour). We may argue that the traditional temporal vector concept can be utilized to estimate VEPs; however, the spatial vector concept may be more adequate for the analysis of the perception-related alterations in the temporal distribution of the cortical activity. We will be using the latter, spatial vector approach.

Selection of the Statistical Analysis Techniques
Problems involving assessment of statistical differences between clusters of multidimensional vectors emerge in many areas of science and engineering. Yet, no universally accepted approach seems to exist for their solution. One procedure to alleviate this problem is to reduce the assessed vectors to scalar quantities, such as vector norm or a single judiciously selected coordinate of the examined vector. Traditional statistical tools, such as ANOVA, can be applied then to scalars. Adopting the spatial vector representation of EEG, the latter would lead to evaluating statistical differences between EEG samples recorded from a specific electrode. This, however, will considerably reduce the information content available for the analysis. We will form the analysis statistics as single-channel EEG samples corresponding to the specific poststimulus latencies. Three groups of observations will be formed corresponding to red, yellow, and green colors of traffic light images; each group will consist of 140 observations (4 participants, 35 repetitions of each stimulus per participant). Kruskal-Wallis (K-W) one-way analysis of variance will be conducted to assess whether samples originate from different distributions.
On the other hand, addressing the outlined shortcoming, Krauss et al. (2018) have recently introduced the quantity referred to as discrimination value as means to analyze spatiotemporal cortical activations; more specifically, to assess their clusterability. In present study, EEG recordings were partitioned into epochs synchronized with stimuli presentation allowing pre-and poststimulation buffers of 100 ms and 200 ms, respectively. However, instead of the traditional temporal representation, epochs were partitioned into spatial vectors of the instantaneous EEG; that is, the vectors composed from the EEG recorded for a particular time instance and for all available 32 electrodes. Since each stimulus was repeated 35 times, three clusters of such spatial vectors were formed containing 35 vectors each. Intra-and intercluster Euclidean distances were evaluated between these vectors and the instantaneous discrimination value (IDV) was estimated for them as in (1). Please refer to the Appendix for the detailed description of the classifier design.
In the present project, the initial k was selected as 10 and the modified knn was implemented for three classes according to Figure A1. To evaluate the performance of the classifier, leave-one-out crossvalidation was applied.
Data analysis was implemented using MATLAB R2008a.

Results
After selecting the least contaminated EEG recordings, Figure 3 illustrates the VEPs obtained by averaging over 35 epochs corresponding to the red, green, and yellow images of traffic light evaluated for four subjects and for three different EEG channels. * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *  Each column in Figure 3 represents a specific participant; the graph color represents the color of the corresponding traffic light. In Figure 3, one can observe well-pronounced VEPs evaluated for different EEG electrodes. For instance, those evaluated for the first subject and for Oz channel (the upper left panel) resemble traditional flash VEPs with well-defined P200 (approximate latency of 150 ms), N300 (past 200 ms), and P300 (approximately 340 ms) components, while other participants produced somewhat less pronounced VEPs. We also observe clear responses to the stimulation at electrodes, such as Cz, not normally associated with the visual system. On the contrary, this channel, among other functions, is normally linked to the motor cortex. Similarly, the frontal channel, Fz, is usually associated with memory-related tasks and high-level information processing, rather than just vision. Perhaps, welldefined responses at these channels may indicate that the stimulation evokes a more complex response including motor-and memory-related components in addition to those from the vision-related cortical regions. Local extrema in these evoked responses evaluated for different EEG channels appear at similar latencies for each participant. Therefore, we may hypothesize that fusing information from different EEG channels could lead to more comprehensive description of perception phenomenon.
Close examination of Figure 3 shows that, although VEPs appear somewhat different for different stimuli (i.e., the color of traffic light), the observed alterations do not appear consistent between participants or even between different EEG channels recorded from the same individual. We conclude that neither the shape nor the latency of VEPs evoked by images of traffic lights of different colors appear contributing to the reliable discrimination between cognitive responses to the images. Therefore, we assess the utility of the spatial vectors approach next. EEG epochs corresponding to three images of traffic lights (i.e., red, yellow, and green) were formed for each participant. EEG was represented as collections of instantaneous spatial vectors. The dimensionality of these vectors is determined by the number of EEG channels (32 in our study).
The samples corresponding to the specific channels-Oz, Cz, and Fz-were used as observations in the Kruskal-Wallis one-way analysis of variance. The resulting Hstatistics is shown in Figure 4.  As seen in Figure 4, H-statistics exceeds 6 for the specific latencies and EEG electrodes. Since the distribution of H can usually be approximated by the chi-square distribution and assuming the significance level of 0.05, we may conclude that the null hypothesis of equal medians can be rejected for the EEG responses to different traffic light color at these latencies. On the other hand, using EEG data from a single electrode only may result in discarding important information. Additionally, spikes in Hstatistics after the stimulation off-set (i.e., for latencies exceeding 300 ms) may be difficult to explain. Nevertheless, we may argue that local maxima in Hstatistics may indicate significant differences in cortical activations observed in response to different stimuli. We implement the discrimination value utilizing all available EEG data followed by the classification of spatial vectors of instantaneous EEG. Figure 5 illustrates IDVs-according to (1)-and the average instantaneous classification accuracies (ACA) for the same EEG epochs that were used for evaluating VEPs in Figure 3. One observation that can be made while examining Figure 5 is that the IDV and ACA graphs are generally opposite to each other; that is, local maxima in ACA are seen at approximately the same latencies as local minima in the discrimination value. The latter is consistent with the expectation that more negative IDV indicates more distinct clusters and, therefore, a better classification performance can be expected. The ACA was evaluated as the average of the percentages of correct classifications for the three classes (i.e., three traffic light colors) considered. For such problems, a random assignment to classes would lead to ACA of approximately 100/3  33%. Therefore, ACA exceeding 50% may indicate that the underlying classification features (EEG spatial vectors) are indeed dissimilar between the different clusters.
We also observe that the classification accuracy generally reaches its local maximum at the latency between 100 and 200 ms for all participants assessed.
Similar results (i.e., the opposite appearance of IDV and ACA and the highest classification accuracy within the same latency range) were also observed for other EEG sets recorded from the same participants and under the identical experimental conditions (not included in the present report).

Discussion and Conclusions
Although VEPs elicited by traffic light images of different colors appear dissimilar, such dissimilarities do not seem to be consistent between the experiment subjects. We may thus conclude that the traditional VEP-based approach may be insufficient to reliably determine EEG alterations that may be related to the specific color of traffic light that the individual perceives.
On the other hand, the alternative technique handling EEG epochs as collections of instantaneous spatial vectors that are used as the classifier's feature vectors appears more promising for the EEG-based assessment of traffic light perception. Furthermore, this approach appears suitable for real-time processing (after the classifier is trained with the appropriate library of feature vectors). The well-established techniques assessing statistical differences, such as ANOVA or Kruskal-Wallis tests, can be applied to the instantaneous EEG samples evaluated for the individual electrodes prior attempting the classification of spatial vectors. Additionally, the discrimination value evaluated for the feature vectors-spatial EEG-appears instrumental for predicting the outcomes of such classification.
Positions of local ACA maxima shown in Figure 5 can be related to the results of previous studies on human cognition. For instance, we may conclude that no reliable detection of traffic light color arises at the poststimulus latencies prior 100 ms. The latter agrees with Thorpe, Fize, and Marlot (1996) who suggested that the recognition of familiar objects generally occurs within 150 ms after stimulus onset. Another local maximum in ACA that is evident for all assessed participants at the approximate latency of 300 ms, perhaps, can be attributed to the P300 component of VEP that was previously reported as contributing to the classification of traffic light color (Bayliss & Ballard, 2000;Liang et al., 2005;Lin et al., 2007).
We have observed that the classification performance was greatly affected by the implemented EEG preprocessing steps, especially by the baseline correction applied to epochs. Perhaps implementing more sophisticated baseline correction methods may lead to an improved performance. Another potential improvement could be an incorporation of EEG artifact suppression techniques. Replacing the knn classifier with more advanced methods may also contribute to a better performance.
Addressing the possible limitations of the present study, one such limitation is a relatively small participant pool, although it may be sufficient considering the pilot nature of the report. Additionally, participants with more diverse background and of broader age range could be of interest. It is unclear whether the observed discrimination between EEG vectors is due to the mere recognition of colors or to more complicated perceptual mechanisms involving the environment analysis. Perhaps using images of solid color or colored figures for visual stimulation or including participants with no driving experience could provide further insight to this question.
We hypothesize that the temporal resolution of the proposed technique is mostly determined by the sampling rate of the neuroimaging modality used (i.e., EEG in the present study). Therefore, increasing this rate may improve the temporal resolution of the stimulation-related analysis. It may also be of interest to apply the proposed technique for the interparticipants comparison of cortical activations evoked by the identical stimulation, although individual cognitive specifics may diminish the utility of such comparison.
Based on the reported results, we conclude that representing stimulation-synchronized EEG epochs as collections of spatial vectors may better reveal the temporal structure of event potentials and that the discrimination value is instrumental for the prediction of clusterability (i.e., whether the analyzed data set can be partitioned into distinct clusters) of eventrelated cortical activity.
Due to the very low signal-to-noise ratio in EEG, evaluating ERPs normally requires averaging over multiple EEG epochs collected in response to the same stimulus. The latter severely limits the use of evoked potentials in real-time applications including neurofeedback, in which the effectiveness of operant conditioning decreases dramatically with the number of seconds between the behavior and the reinforcer/punishment (Sherlin et al., 2011). Although the spatial vector-based analysis also assumes collection of the library of responses (EEG epochs in our study) to train the classifier, the classification stage may occur in real time. This advantage of the proposed technique may make it attractive for applications requiring instantaneous feedback. Finally, we further hypothesize that the spatial vectorbased approach may also be instrumental for other multichannel neuroimaging techniques, such as magnetoencephalography (MEG).
where the training (feature) sample xn belongs to the n th out of three possible classes-A, B, or C-and (xn) represent the class associations. The Euclidean distances between the test N-dimensional vector z and all training vectors constituting to each of three classes was evaluated as (Duda, Hart, & Stork, 2000): k training vectors were selected that were closest (i.e., having the shortest distances) to the test vector. Such training vectors are referred to as "nearest neighbors." The test vector z is assigned to the class, to which the majority of its k nearest neighbors belong. Here, k is a scalar value selected by a user. Regrettably, the performance of knn classifiers may be affected by tied votes. While tied votes can easily be avoided for two-class discriminations by selecting odd values of k (Phyu, 2009), no straight-forward solutions seem to exist for multiclass problems. We propose a simple iterative technique for tied votes' avoidance in knn classification. The implemented algorithm is summarized in Figure A1. Tied votes occur if, among the k nearest neighbors, more than one group produces the maximum number of neighbors. Therefore, we propose monitoring group membership among the k neighbors to detect tied votes and, if such votes occur, incrementing k by one, reselecting neighbors, and reevaluating their membership until no tie votes arise.