Study on Performance Metrics for Consideration of Efficiency of the Ocular Artifact Removal Algorithms for EEG Signals

A major challenge related to the removal of Ocular Artifacts (OA) from Electroencephalogram (EEG) signals is the lack of a consensus about one superior standard OA removal algorithm among all, due to unavailability of both natural artifactfree EEG signal and a standard performance evaluation criterion. The present paper discusses the various validated performance criteria often used in research papers for considering the efficiency of the OA removal algorithms. These metrics can be measured in the MATLAB software. In addition to natural EEG signals, artificial or simulated signals are also considered in this study. Some of these performance criteria are commonly used for both real and simulated signals, while others are appropriate only for one of them specifically. However, evaluation of a simulated signal can be easier than a real one because of the availability of artifact-free EEG signals in a simulation study, evaluation using a real signal is more trustworthy than simulated one. Hence, for more reliable and precise efficiency considering of an OA removal algorithm, employing both signals, is recommended. Further, because of the non-stationary nature of real EEG signals, the comparison of an algorithm with others, via implementing them on the same signals, will be meaningful and applicable. The present work will help researchers in recording the efficiency of their algorithms, as well as comparing the performance of their methods with others for reaching a consensus in OA removal.


Introduction
The electric fields around of eyes change due to rotation of the eyeball and eye blink which is called as Electrooculogram (EOG). The EOG signals are usually large amplitude and very low frequency signals in the range of 0 to 16 Hz 1 and being maximal at frequencies below 4Hz 2,3 . The EOG signals propagate over the scalp and superposition of these signals and brain signals are recorded in the head surface 1,4,5 using Electroencephalogram (EEG). These EOG signals which are mixed in the EEG signals are known as the Ocular Artifacts (OAs). Hence, electric fields over the scalp are distorted with the OAs. Because of the applications of the EEG signals in the diagnosis and treatment of different brain disorders as well as Brain Computer Interface (BCI) system 6 , correct interpretation of these signals is very important before any use. In recent decades, researchers have introduced various types of advanced digital signal processing techniques such as Independent Component Analysis (ICA) 7 , Adaptive Filter (AF) 8 , Wavelet Transform (WT) 9 , Artificial Neural Network (ANN) 10 , Principal Component Analysis (PCA) 11 , Hilbert-Huang Transform (HHT) 12 for the OAs removal in the EEG signals. A comprehensive review of these techniques is given by 13 .

Keywords: EEG, EOG, Ocular Artifact Removal, Performance Metrics
Since a natural artifact-free EEG signal as well as, a standard performance metric, do not exist, the performance of these algorithms cannot be reported and compared precisely. Hence, there is no consensus regarding which algorithm is superior to the others in term of performance 11 . Various metrics can be approximately used for the consideration of efficiency of these methods. But there is no consensus on which metrics are superior to the others partly 11 . For overcoming to these problems, some of the researchers have proposed artificial artifact-free EEG signals using simulations. Consequently, the performance of these algorithms has tested and compared for both real and simulated signals with different criteria. This paper is focused on different performance metrics which, can be used in different OAs removal algorithms using real and simulated EEG signals.

Performance Metrics for Real EEG Signals
Despite the fact that there is no standard waveform as artifact-free EEG, some validated criteria are available, for performance evaluation of the OAs removal algorithms for real or natural EEG signals. These are listed below.

Visual Inspection by Expert
This can be done by helping neurologists or bio-signal expert who is able to approximately distinguish the existence of the OAs in corrected EEG 4,7,14,15 . This qualitative method can be commonly used for any OAs removal algorithms.

Visual Comparison in Time Domain
In order to use this qualitative criterion for obtaining the performance of the proposed algorithms, the Electrooculogram (EOG) signal as OA source and EEG signal, must be recorded simultaneously. The approximate level of the removed OAs in the time domain, can be estimated by comparison of these waveforms and corrected EEG signal at various instants 1,10,[16][17][18][19][20] . This method has been used frequently to approximate the performance of most of the OAs removal algorithms.

Correlation Analysis in Time Domain
From previous research works, it is well accepted that because of volume conduction, the EOG signal are modulated on EEG signals in unknown path. Hence, OAs source signals and artifact-free EEG signals (pure cortical signals) are quietly un-correlated. On the other hand, the propagation of the OA voltage over the scalp is attenuated approximately with the square of the distance from eyes 13 . Hence, the frontal EEG channels are more contaminated than parietal and occipital channels. By using this fact, researchers have used three forms of correlation criteria in the time domain as a quantitative metric. According to this interpretation, if the OAs removal algorithm is ideal and cancels the artifact perfectly, the expectation is as bellow.
• High correlation value between measured EOG signal and estimated EOG artifact 21 . • Low correlation value between measured EOG signal and corrected EEG signal in comparison with a correlation value between measured EOG signal and contaminated EEG signal 10,22 . • High correlation value between corrected EEG and measured EEG in parietal (or occipital) lobe and on the other side, a low correlation value between corrected EEG and measured EEG in frontal lobe 7 .

R 2 Value or Artifact to Signal Ratio (ASR)
Since in the real experiment artifact-free EEG signal is unknown, an appropriate quantitative metric is defined in Equation (1) as the ASR or R 2 rate for determining of performance of the proposed algorithms in minimizing the OAs from contaminated EEG signal 23 .
Where d(k) is the primary or measured EEG signal, e(k) is the error signal or estimated EEG signal and N is the number of samples. The R 2 is the ratio of the power of the OA being removed from the measured (contaminated) EEG to the power in the estimated pure EEG obtained from the proposed algorithm 24 . From this equation, it can be seen that, the higher the value of R the better is the OA minimization 23 . For data that does not contain artifacts a higher ratio is not necessarily indicative of better artifact removal, but rather it means that the most useful part of EEG signal is removed 25 . Hence, this metric is applicable just for the contaminated zones of the EEG signal.

Mean Square Error (MSE)
This quantitative criterion is used for estimation that, how much of the OAs in measured EEG signal are cancelled 10,22,[26][27][28][29] . The MSE in AF algorithm can be measured using Equation (2) 10 . The least value of the MSE shows the best performance of the proposed algorithms.
Where, d(k) is the measured EEG signal, y(k) is estimated of the existing OA in the measured EEG signal which are obtained from the output signal of the filter, and n is the number of samples.
The root MSE (RMSE) can be also used as a standard performance measuring parameter 30 .
In the AF with recursive least square algorithm, MSE is also defined as Equation (3) 8 . Where n is the length of the window or number of samples, M is the length of FIR filters, λ is forgetting factor and e(n) is error signal.

Power Spectrum (PS)
This quantitative metric reflects the energy of the signal in the frequency domain. This metric can be calculated based on various models such as the PS based on Auto Regressive (AR) parametric which is the major part of the PS estimation 29 , the PS Magnitude (PSM) 10,17 and PS Density (PSD) 11,31 . By looking, calculating and comparing discrepancy of the PSD of measured EEG (uncorrected EEG) and corrected EEG waveforms, the performance of the OA removal algorithms can be evaluated 4 .

Time Consumption
The elapsed time for completing the artifact removal process can be a measurable indicator of the success rate of the artifact removal algorithms. Hence, the minimum of the time consumption with regard to other criteria in turn, can be as a symbol for priority of an algorithm 20,23 .

Classification Accuracy
To verify the effectiveness of OA removal algorithms, the comparison of classification accuracy with and without OA can be applied. It is expected that, the classification accuracy rate increases after applying a proper OA removal algorithm 21 .

Correlation Analysis in Frequency Domain
It is expected that after applying a perfect OAs removal algorithm, the contaminated and corrected EEG signals to be uncorrelated in lower frequency because of OA removal in this frequency range, and be highly correlated in the higher frequency range. This metric is applicable for both simulated and real signals and can be measured using Equation (4). x y xy C xx yy Where, x and y are the symbols of contaminated and corrected EEG respectively, w1 and w2 are the window limits, x  and y  are the Fourier coefficients of x and y, and x   and y   are the complex conjugate of x  and y  9 .

Performance Metrics for Artificial or Simulated Signals
Since, artifact-free EEG signal is not available in real experiments, the artificial artifact-free EEG signal can approximate, the performance of OAs removal algorithms. Different ways can be used for generation of artificial artifact-free EEG signal 32 which, some of these methods are listed in 25 . Often in these methods, artifacts are added to artificial artifact-free EEG signals. Hence, in these methods, the pure EEG signal is available and consequently the judgment of artifact removal algorithms is easier than real or natural signals. Here, some metrics for determining the efficiency of OAs removal algorithms using simulated EEG signals are listed.

Visual Comparison in Time Domain
To measure the effects of OA removal algorithms, a qualitative approach is the visual comparison of the amplitudes of corrected-EEG, artificial artifact-free-EEG and contaminated EEG signals at different moments in the time domain 11 .

Mean Square Error (MSE)
Since the artificial artifact-free EEG signal is available here, in the definition of MSE, one can benefit this, in the Equation (5).
Where e(n) and x(n) are the corrected EEG and artificial artifact-free EEG signals respectively, and N is the number of samples 32 .
Also root MSE (RMSE) which is defined in the Equation (6) is used 28 as a quantitative metric.
Where x(t) and ˆ( ) x t are the original and enhanced EEG signals respectively, and T is the number of time points 28 .

Artifact to Signal Ratio (ASR)
As it is mentioned in Equation (1), this criterion is defined as the ratio of the power of the remaining artifact to the power of the estimated pure EEG signal. Since in simulated data, the pure or artifact-free EEG signal is available, it is expected that, for complete OAs removal, the value of this quantity results to zero 25 .

Correlation Method in Time Domain and Frequency Domain
The correlation coefficient between clean EEG and corrected EEG signals in time and frequency domain 11 are used as a quantitative metric for algorithm evaluation. Also, comparison of correlation coefficients between EOG and EEG signals before and after applying algorithms, can be as a good criteria for determining of algorithm efficiency 18,31,33 .
Where P e and P x are the power spectral density functions of e(n) and x(n), wherein e(n) and x(n) represent the corrected EEG and artificial artifact-free EEG signals respectively, and i and j define the frequency range of a particular band 32 . This quantitative metric is used for comparison of the accuracy of different algorithms.

Signal to Noise Ratio (SNR)
The SNR can be used as a quantitative metric for performance evaluation of the OAs removal algorithms 11,23,30,31 . Generally a larger SNR is expected for corrected signals 31 . For calculating SNR, simulation data can be reconstructed from the collected data 11 and SNR can be compared before and after applying these algorithms.
For calculating output SNR (SNR out ) which is defined as Equation (8), the EOG channel is added to the desired signal at different Signal to Noise Ratios (SNRs) to construct the primary signal 23 .
Where s(n) is the desired signal (artificial artifact-free EEG signal), ˆ( ) s n is the estimate of the s(n) (corrected EEG signal), y(n) is the primary signal (noisy EEG signal) and N is the total number of samples.
Also, SNR improvement is defined as Equation (9) 27 . (Note that, this method is applied for ECG signal 27 , but same metrology can be used for artificial artifact-free EEG signals).
According to this equation, better de-noising algorithm is expected to have larger SNR Imp in dB at a particular SNR.

The Criterion of Angle Cosine
In machine learning, the cosine of the angle is applied to measure the difference between two samples 29 . While in the geometry, this criterion is used to approximate the similarity of two vectors. The criterion of angle cosine has higher resolution than the correlation coefficient 29 . Hence, this quantitative metric can be used for measuring similarity between artificial artifact-free EEG signals and corrected EEG signal or similarity between other signals. Obviously, larger value of this metric presents the higher similarity. The angle cosine of two sets of data as x i and y i , with N number of samples can be measured using Equation (10).

Signals to Error Ratio (SER)
The SER can be applied as a quantitative metric for measuring performance of proposed OAs removal algorithms. This metric is defined as Equation (11) Where x(t) and ˆ( ) x t are the original and enhanced EEG signals respectively.

Conclusion
In this paper, we have collected some evaluation metrics, which have used and validated in research papers for evaluation of the OA removal algorithms using real and simulated signals. It is found that, although evaluation of these algorithms using simulated signals has more simplicity, but evaluation using real signals is more trustworthy than simulated signals. Also, because of the nonstationary nature of real EEG signals, the comparison of an algorithm with other algorithms will be meaningful, just by applying these algorithms on the same EEG data.