The electrooculography control system

Damian Pakulski, Artur Gmerek

Institute of Automatic Control, Lodz University of Technology

1. Introduction

The electrooculographic signals appear when someone moves his or her eyes. EOG signals are usually used in diagnosis of eye diseases. Since these signals are determined, they may be used for manipulators control. This paper describes a system which, based on the information from EOG signals, is able to control external devices.

Proper processing of EOG signal and its usage in manipulator control is complicated. Signals have to be registered from the skin, then processed in the multilevel process, after that the characteristic features of signals are designated and finally vectors of features are classified to the distinguished classes.

EOG signals are much simpler to record and process than e.g. EEG (electroencephalographic) signals, however not many scientists know about their existence.

There are not many papers about processing and usage of EOG signals. Usakli and Gurkan used EOG signals for control of simple virtual keyboard. The accuracy of classification was 95 % [1].

Some scientists advanced conventional EOG apparatuses to their wireless forms. For example Ubeda et al. has created a wireless system which has been successfully used for industrial robot control [2].

Many papers concentrate on register and digital processing of signal [3, 4]. Researchers usually used following features: polarization of signal amplitude, slope (based on derivative of the signal), meaning value and duration of signal peak [4, 5]. Classification is usually done with the use of statistical models [5, 6].

2. The general description of the system

System consists of several parts. The first part is connected with a signal recording from the skin. It can be done with the use of specially designed EOG apparatuses. After that, the signal is routed to the high-level controller where it’s digitally processed. The final stage of it is classification with the use of an ANN, which assign signals to specific classes. These classes are connected to particular movements e.g. movement of an eye to its left or right position, closing of the eye, etc. After that these movements are mapped to specific motion mechanism of a 2-DOF manipulator. These motion commands are sent via USB to a low-level controller (ATmega8 microprocessor), which controls the manipulator directly (fig. 1).

Fig. 1. General scheme of the system
Rys. 1. Ogólny schemat systemu sterowania

2.1. EOG apparatuses

EOG apparatuses have been designed to properly record EOG signals from the skin. These consist of 5 important components. The first thing is instrumentation amplifier. After signal has been registered by the instrumentation amplifier, it is routed to a high-pass filter in order to remove constant component of the signal. After that the signal is amplified several times and finally is routed to the low-pass filter, which also acts as an antyaliasing filter. Such output signal can be finally converted to digital form. The PCB of the device was designed in Altium Designer (fig. 2). It has been used second order Butterworth filter.

Fig. 2. Circuit diagram of EOG apparatus
Fig. 2. Schemat elektryczny opracowanego aparatu EOG

2.2. High-level controller

High-level controller consists of functions written in C# language, which are responsible for communication with acquisition card, signal processing and communication with low-level manipulator controller.

During experiments which have been intended to show the nature of the EOG signals, there has also been used another application for data acquisition. In this program, the graphical commands appear in the application window that the user has to follow. During it the EOG signals are registered.

2.3. Low-level controller and manipulator

Low-level controller is responsible for manipulator control. The controller is based on popular ATmega8 Atmel microcontroller. The object is constructed from two hobby servos, which axis is shifted by 90 degrees. The end-effector of manipulator is a laser point. This kind of servos is controlled in a very specific way. These accept only pulse width modulation (PWM) signal with 50 Hz frequency. The position of the servo can be regulated by changing the duty cycle from 1 ms to 2 ms, where 1.5 ms is neutral position of servo and remaining two are its extreme positions. Microcontroller communicates with high-level controller through UART protocol with the use of USB transmission. There was used the FTDI UART-USB converter.

3. Signal Processing and Classification

Signal processing is composed of the following stages: filtering, division into sections based on the movement pattern, and finally calculation of signal descriptors. After receiving the signal, it has been filtered with low-pass filter, which transfers function that can be described as:

This kind of filter removes almost all noise, remaining, as a result, a pure determined signal (fig. 3).

Fig. 3. Time domain and spectrogram of filtered signal. It can be seen on the spectrogram that, inter alia, the 50 Hz noise has been removed
Rys. 3. Sygnał w dziedzinie czasu i spektrogram odfiltrowanego sygnału EOG. Na rysunku możemy zauważyć, iż między innymi została usunięta składowa 50 Hz

Then these signals were processed again in order to divide them into sections, in which motion appears. It has been done based on a modified threshold algorithm. The beginning of the motion has been designated based on the derivative of fragment of the signal (DS). When DS is bigger, then the established threshold, it means that patient has moved his/her eyes. The end of the motion has been designated based on moments passing by zero. The characteristics of the EOG signals require passing only once through zeros and then relatively slowly reaching zero through exponential curve. These calculations have been repeated, while windows travel through recorded fragment of the signal (fig. 4).

Fig. 4. Illustration of the signal division into sections. Blue points correspond to the beginning and red points to the end of motion
Rys. 4. Ilustracja algorytmu podziału sygnału na fragmenty, w których występował ruch. Niebieskie punkty symbolizują początek ruchu, a czerwone jego koniec

It is important to finish the motion before recording is stopped, otherwise signal may be divided between different recordings, which causes system malfunction.

After signal division, descriptors of motion have been designated. There have been used the following features: an amplitude of the signal, middle value, mean frequency, duration of divided fragment of motion and slope of the signal

Vector of features was routed to the ANN, where it was classified to one of several classes (fig. 5).

Fig. 5. Architecture of the ANN used for the signal classification
Rys. 5. Architektura sztucznej sieci neuronowej użytej do klasyfikacji sygnałów

4. Experiments

Experiments were done on the one subject (fig. 6). It is important that during signal recording there should be placed one additional electrode on the patient, which would connect the subject to the ground of the devices. The following classes of motions were used: left eye position, right position, upper position, button position, single blink, double fast blink, close and open eyes. Study of the signal shows that the values of signal amplitude of the blinks are too low, compared to other motions. Consequently, because the division algorithm has been used based on time domain of the signal, blinks have been excluded from the control algorithm.

Fig. 6. The system designer during the experiments
Rys. 6. Projektant systemu podczas przeprowadzania eksperymentów

After recording the principal movements, signals were processed, which final stage designated the vector of features. This vector of features was used to teach on ANN. After teaching, the ANN was prepared for real time operation. The code was written in C# language.

5. Results and EOG signals characteristic

Electrooculographic signals, similarly to other signals, can be considered in spectral or time domain. Signals in time domain give many information and are relatively easy to be interpreted. Experiments provide insight into the nature of EOG signals. Research study shows that left and right, as well as upper and bottom eyes movement can be easily distinguished. Depending on the placement of electrodes, the amplitude of upper and left motion pattern would be positive, while the amplitude of right and bottom eye movement would be negative. Besides, amplitude of upper and bottom eye movement is lower then left-right eye movement (fig. 7).

Fig. 7. Simple characteristics of EOG signals
Rys. 7. Przykładowe charakterystyki sygnałów EOG

User was able to successfully control the manipulator with the created algorithm. Results of experiments also show that only one channel of electrode is sufficient to correctly control the manipulator.

6. Conclusions

The aim of the project was achieved. There has been created a fully automated system which has been successfully used for manipulator control. Recorded signal has been high-quality compared to the signal of other researchers. This means that the EOG apparatuses have been properly designed. Overall accuracy calculated on the basis of several dozens of measurements has been 91 %. This results have been calculated in offline experiments after dividing the set in 7:3 proportion (training:test). This accuracy could be probably greater with the use of additional features connected to spectral characteristics of the signals. The experiments showed that EOG signals could be used even in more complicated control systems than 2-DOF manipulators.

7. Future Work

Future work will be connected with verification the degree of discrimination of other features. In the future there should be, for example, tested the non-linear dynamics descriptors e.g. fractal dimensions, Lyapunov exponent and others. These descriptors are not much correlated with previously used ones.

It is also important to improve the division algorithm by the spectral information. After such refinement, it would be possible to use blinks in the system.

There will be also created more complicated control system, probably virtual keyboard or some sort of mobile application.

Bibliography

1. Usakli A.B., Gurkan S., Design of a Novel Efficient Human – Computer Interface: An Electrooculagram Based Virtual Keyboard, “IEEE Transactions on Instrumentation and Measurement”, 2010, vol. 59, 2099–2108.

2. Ubeda A., Ianez E., Azorin J.M., Wireless and Portable EOG-Based Interface for Assisting Disabled People, “IEEE/ASME Transactions on Mechatronics”, 2011, vol. 16, 870–873.

3. Kuo C.-H., Chan Y.-C., Chou H.-C., Siao J.-W., Eyeglasses based electrooculography human-wheelchair
interface, Proc. IEEE Int. Conf. Systems, Man and Cybernetics SMC, 2009, 4746–4751.

4. Kherlopian A.R., Gerrein J.P., Yue M., Kim K.E., Kim J.W., Sukumaran M., Sajda P., Electrooculogram based system for computer control using a multiple feature classification model, Proc. 28th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society EMBS, 2006, 1295–1298.

5. Bulling A., Ward J.A., Gellersen H., Troster G., Eye Movement Analysis for Activity Recognition Using Electrooculography, “IEEE Transactions on Pattern Analysis and Mechine Intelligence”, 2011, vol. 33, 741–753.

6. Barea R., Boquete L., Mazo M., Lopez E., System for assisted mobility using eye movements based on electrooculography, “IEEE Transactions on Neuronal Systems and Rehabilitation Engineering”, 2002, 10, 209–218.

System sterowania wykorzystujący sygnał elektrookulograficzny

Streszczenie: Celem projektu było opracowanie metod przetwarzania i analizy sygnałów elektrookulograficznych (EOG) na potrzeby sterowania manipulatorów. Elektrookulografia jest techniką polegającą na pomiarze potencjału szczątkowego gałki ocznej, który wynika z elektrycznej aktywności siatkówki. W pracy przedstawiony jest kompletny system elektrookulograficzny, który steruje laserowym wskaźnikiem o dwóch stopniach swobody. W celu dostosowania sygnału EOG do sterowania manipulatora musi zostać on zarejestrowany przez czuły galwanometr zwany elektrookulografem, a następnie przetworzony w wieloetapowym procesie przetwarzania cyfrowego. Końcowym etapem przetwarzania jest klasyfikacja z wykorzystaniem sztucznych sieci neuronowych. Aplikacja może zostać wykorzystana przez osoby niepełnosprawne mające kontrolę jedynie nad ruchem swoich oczu.

Słowa kluczowe: elektrookulografia, EOG, HMI, BMI, manipulator, sterowanie

Artur Gmerek, MSc

He received the MSc degree in the field of Automatics and Robotics, majoring in Control of Industrial Processes and Applied Computer Science from Lodz University of Technology, Poland, in 2008. His research interests include rehabilitation robots and biomedical systems. He currently studies for a PhD at the Lodz University of Technology in Institute of Automatic Control.

e-mail: artur.gmerek@p.lodz.pl

Damian Pakulski, BSc, Eng.

He is studying an Automatic Control and Robotics at the Lodz University of Technology. In 2012 he defended his BSc, Eng thesis “Feasibility study of use of electrooculography signals for control of manipulators”. In his range of his interests are signal processing and robotics.

e-mail: dondmno@gmail.com