Initial tests of a trinocular vision system for the underwater exploration

Wojciech Biegański, Andrzej Kasiński

Institute of Control and Information Engineering, Poznań University of Technology

Abstract: This paper describes the basic idea of operation and assumptions of the Trinocular Vision System (TVS) designed to support the underwater exploration with the use of the autonomous vehicle. The paper characterizes the optical properties of the inland water environment, and the process of the image formation in that environment. The paper presents the aim of the image fusion and also the design process of a multimodal vision system, i.e. the selection of its components confirmed by prior research in the context of underwater operation.

Keywords: image fusion, underwater vision system, multimodal vision system, AUV

1. Introduction

Using a machine vision together with the underwater vehicles is the essential aspect of the underwater exploration activity such as rescue or inspection missions. Underwater vision systems are widely used in the ROV (Remotely Operated Vehicles) as well as in the AUV (Autonomous Underwater Vehicle) class robots (e.g. for navigation purposes or manipulation support). Using standard cameras together with the natural light source for underwater imaging is insufficient, especially in inland waters, where visibility is measured in cm (0.2 m for Odra River in Poland) and the maximum of 7 m in lowland lakes.

In this article the trinocular vision system for underwater exploration is presented as an example of a machine vision system that is more suited for unfavorable phenomena occurring in underwater environment disturbing the process of proper interpretation of the recorded images.

The system consists of three cameras, each capturing the images in separated band of the light spectrum. The bands are: near ultraviolet spectrum (NUV), the visual spectrum (VIS) and the near infrared spectrum (NIR). The images captured with cameras are next aligned and merged into single image containing more information than the image recorded by the only one of these cameras. The basic assumption is that the images captured in different bands of light spectrum consist of different image features. The resulting image, after the image fusion, should be more feature-rich than any of the component image. The trinocular vision system is a part of the Isfar Project – a hybrid of AUV/ROV classes vehicle designed and built to explore inland waters [6].

2. The optical characteristics of the underwater environment

Natural water is optically non-coherent. The unfavorable phenomena influencing the transmission of light beams are difficult to describe, because of their selective and volatile nature. The intensity of those effects depends on the chemical substances that water contains. Those substances are the dissolved organic matter (DOM) and the suspended organic matter (SOM) and they cannot be fully identified. They are the remains of the metabolic products of plants and animals and also are due to products brought in by other sources of water [1].

When a photon hits a water molecule it makes that molecule oscillate forcing the photon energy to drop. That effect is called the light absorption. The light absorption coefficient a and the absorption spectrum a(λ) differs significantly depending on the kind of molecules found on the optical pathway. According to the Lambert’s law, the radiance of the light (L) decreases logarithmically as the distance r between the light source and the observer (detector) grows:

The absorption coefficient increases with the light of longer wavelength. The Lambert’s law cannot be used to describe the radiance of the light in the turbid water environment, i.e. in the natural water, because of the presence of the light scattering effect.

The second effect influencing the light transmission by the water is that the light rays reflect and refract while coming across the area of the non-water suspended substance and hence change their directions. This effect is known as the light scattering. The inland waters are an environment where the light scattering effect is intense, since the light in such an environment can be scattered in every direction. The scattering is described by the scattering coefficient b and the volume scattering function β(θ) (describing the intensity of the light being scattered into the direction parametrized by the angle θ).

The scattering effect has significant impact on the transmission of light through the water especially for the light of shorter wavelength.

The combination of scattering and absorption effects (that are inseparable in the inland water environment) corresponds to the total attenuation of the light described by the beam attenuation coefficient c which is a sum of a and b [3]. The c coefficient changes as a function of the distance between the observer and the light source similarly as the coefficient a:

The beam attenuation coefficient is one of the fundamental parameters of water quality describing its clarity.

The nature of those effects depends not only on the type of water (inland/sea water) but also on the camera location within the water body, i.e. its depth and the horizontal position. Moreover, the intensity of the scattering and absorption effects has a temporal character. It could change according to the day/night cycle, and also it is seasonal (determined especially by the activity of the underwater vegetation). The beam attenuation coefficient may also vary within several years (unfortunately, in most cases it grows, which means the quality of water deteriorates) [5].

The spatial and temporal character of the scattering and absorption effects makes those coefficients difficult to measure (the scattering coefficient is almost impossible to measure, on the other hand the absorption coefficient can only be measured in clean waters because of the presence of the scattering effect, which dominates in natural environment).

Fig. 1. The transmission of the distilled water and natural water  taken from the Vistula River as a function of the wave length
Rys. 1. Przezroczystość wody destylowanej oraz wody pobranej  z Wisły w funkcji długości fali światła

The absorption coefficient a, the scattering coefficient b, the scattering function β(θ) and the beam attenuation coefficient c, are the so-called inherent optical properties of water that fully specify its optical character [2]. Figure 1 shows the optical transmission of clean, distilled water and water taken from the Vistula River (Masovia region).

The optical transmission (or transmittance) is a ratio between the radiance of the light measured at the beginning of the optical path (the radiance of the source – L0) and the radiance of the light measured at the distance r from the source – Lr expressed as percentages:

The sunk object is visible under water when the radiance of the light reflected from that object reaching the observer (or detector) differs from the radiance of the background behind that object. The difference between those two radiations determines the contrast of the image and consequently the visibility of the object or the ability to detect that object with the use of cameras.

The operation of the trinocular vision system (TVS) leans on the fact, that a and b coefficients differ depending on the light wavelength.

The process of underwater-image formation

The change of the light radiance L (emitted by the natural source) in the water environment along the optical path of the length r is described by the equation [3]:

where L* is the function that describes the scattered rays into the direction being considered, coming from the other rays and Lη – the radiance of light rays coming from the other sources (or newly induced sources found in the optical path of the considered ray such as microorganisms emitting the light as a result of the bioluminescence and fluorescence effects). The above equation is known as the radiative transfer equation.

Figure 2 illustrates the optical path of the light letting the observer to ‘see’ the sunk object.

Fig. 2. The path of a single ray of the natural light reflected  to the camera, letting to capture the image of that object
Rys. 2.  Droga pojedynczego promienia światła naturalnego odbitego w kierunku kamery, co umożliwia zarejestro  wa nie obiektu pod wodą

The remaining effects influencing the underwater-image formation are the absorption of the proper optical elements of the TVS such as the viewfinder, filters and lenses and the reflection/refraction at the air/water interface (in case of taking advantage of the natural light availability), but also water/viewfinder interface deforming in that way the registered image containing the object of interest.

Since the light intensity emitted from the natural source is insufficient, especially on the depths below 5 m (the IR radiation is entirely absorbed in the upper water layer few centimeters thick) or at night, the artificial light source able to emit the light of interesting wavelengths is required.

3. Data fusion and image fusion terms

According to the L. Wald’s definition: “data fusion is a formal framework in which are expressed means and tools for the alliance of data significating from different sources. It aims at obtaining information of greater quality; the exact definition of ‘greater quality’ will depend upon the applications” [4].

Data being fused could originate from a single or multiple sources thus the fusion could be single or multi-sensor.

With reference to the TVS, where data comes from three sensors of the same type (CCD matrix sensors), the carrier of data is an image, thus here we deal with the multi-sensor data fusion. In this application, the term ‘greater quality’ means the image containing more meaningful information than any of the component images and moreover carrying more information than the image being a simple ‘sum’ or composition of component images.

The aim of the image fusion is to output the composite (fused) image which is an input for the navigation algorithms of the control system of the underwater robot. The resulting image consists of multi-attribute pixels, each having 5 components (R, G, B plus V for ultraViolet, and I for Infrared).

4. Channels of image registration

The system is multimodal. The sequence of images is captured in three separated bands. The separation is assured by the use of appropriate optical filters with narrow characteristics of the optical transmission. Three different cameras are used. The illumination system is also designed in order to emit only the light of the desired wavelengths.

The image captured in these bands slightly extends the registration spectrum of the standard optical camera because of adding the close wavelengths neighborhood. Together with the specialized algorithms of the image fusion it leads to the original results. There is a need to balance the optical power of each light source to ensure the equal light conditions in each band. The energy conversion efficiency of the NUV LED is about 25 % smaller than the white LED, hence the electrical power of the LEDs differs.

The near ultraviolet channel – NUV

The near ultraviolet band, wavelength between λ = 315 nm and λ = 380 nm (UV-A). The camera is a Sony XC-EU50CE CCD. The maximum sensitivity of the CCD claimed is at λ = 360 nm. The lenses are standard C-mount with focal length of 3.6 mm, made of BK-7 (optical) glass letting in the NUV radiation. The filter is a Hoya U-360 band pass filter. The optical transmission of the filter as a function of the wavelength is shown in figure 3. The light source is a 10-Watt Power-LED. The measured maximum optical power of the LED is at λ = 365 nm.

Fig. 3.  The optical transmission of the NUV filter as a function of   the wavelength
Rys. 3. Transmitancja optyczna filtru NUV

Many underwater plants absorb the UV light (needed for the photosynthesis), thus the image including such areas differs in NUV from that in VIS band

The visible light spectrum – VIS

The radiation of visible light extends at λ = (380–780) nm. The standard optical camera was used (single board, small-sized, 1/3’’ CCD matrix, 520 TVL) with the mini-lens (ϕ12 mm) and the focal length of 3.6 mm. The pair of cut-off filters is used: UV cut-off filter UVK-2510 and the IR cut-off filter ICF-2510. The optical transmission of both filters is presented in figure 4. The illumination is provided with the 3 W Power-LED, emitting the white light.

Fig. 4.  The optical transmission of the VIS filters as a function of  the wavelength
Rys. 4. Transmitancja optyczna toru VIS

The near infrared channel – NIR

The near infrared spectrum is between the wavelength of λ = 750 nm and λ = 1400 nm. The image capturing device is a CCD single-board camera with the registration capabilities extended to NIR radiation equipped with the ϕ12 mm lens having the focal length f of 3.6 mm. Schott RG-712 long pass filter was used. Its optical transmission is shown in figure 5. The 1-Watt Power-LED with the maximum of the optical power at λ = 850 nm was used.

The areas of interest having different temperature than the temperature of the background reflect the light at the NIR band differently thus the image of such regions differs in NUV from that in the visual spectrum.

Fig. 5.  The optical transmission of the NIR filter as a function of the wavelength
Rys. 5. Transmitancja optyczna filtru NIR

5. Mechanism design

The cameras were placed in the way to make their optical axes parallel, at the vertices of equilateral triangle. The distance between them (the optical base, B) is 36 mm. The placement of cameras is shown in figure 6.

Fig. 6. The placement of elements (LEDs and cameras)   of the TVS
Rys. 6.    Rozmieszczenie elementów TVS (LED oraz kamery)

The viewfinder housing the elements of the TVS is mounted at the end of the main hull of the Isfar vehicle.

Fig. 7. The optical transmission of all tested transparent  materials as a function of the wavelength
Rys. 7.   Transmitancje optyczne testowanych materiałów przezroczystych

The viewfinder is made of the BK-7 (borosilicate) tempered glass. The selection of the material for the viewfinder was preceded by the survey conducted on several types of transparent materials with the use of a spectrophotometer. The material for the viewfinder has to be durable enough since the vehicle is designed to work at the depth of 30 m being able to transmit the light in desired bands.

All tested materials were proper transmitters for VIS and NIR spectrum. The main differences in transmission occurred for the NUV spectrum. The survey on materials found that none of the plastics is suitable for the NUV light transmission (λ=360 nm). The BK-7 glass is thus a compromise between endurance, costs and availability. The optical transmission of all tested materials as a function of the wavelength is shown in figure 7.

The housing of the TVS was made of the polyethylene PE HD1000, and fits to the front of the main hull of the Isfar underwater vehicle.

6. Conclusions and further work

The designed and built vision system described in this paper satisfies the assumptions of the multimodal vision system purposed for the inland, underwater exploration. The particular elements were tested in terms of the optical transmission in the desired channels of the light spectrum. The placement of cameras and filters let capture the image of the same object (or region) of interest simultaneously. The captured images are aligned to each other by the offset B = 36 mm.

The placement of the illuminating elements let the scene to be illuminated proportionally.

The operational environment was tested with respect to the optical transmittance. Research proved that water is a moderately good medium for the operation with light at interested channels. The capturing of the test images with the use of TVS in laboratory conditions (aquarium) is planned in the nearest future. The planned objects to be tested are the artificial landmarks and the objects that could occur in the natural, inland waters, i.e. plants. The use of the polarisation filter is also planned.

The next step is the development of a specialized image fusion algorithm for the images recorded by particular channels of the TVS. The algorithm will assign appropriate weights to the areas resulting from the segmentation at particular registration channels before the synthesis of the fused image will take place.

References

1. Dera J., Marine Physics, Elsevier Science, 1992.

2. Davies-Colley R.J., Vant W.N., Smith D.G., Colour and Clarity of Natural Waters. Science and Management of Optical Water Quality, The Blackburn Press, Hamilton, New Zealand, 1993.

3. Preisendorfer R.W., Application of radiative transfer theory to light measurements in the sea, IUGG-IAPO Symposium on Radiant Energy in the Sea, 4-5 Aug. 1960, Helsinki, Finland. Monograph No. 10, 83–91.

4. Wald L., Some Terms of Reference in Data Fusion, “IEEE Transactions on Geoscience and Remote Sensing”, Vol. 37, No. 3, May 1999.

5. Jassby A.D., Goldman C.R., Reuter J.E., Richards R.C., Origins and scale-dependence of temporal variability in the transparency of Lake Tahoe, California–Nevada, “Limnology and Oceanography”, 44(2), 1999, 282–294.

6. Biegański W., Ceranka J., Kasiński A., Design, control and applications of the underwater robot Isfar, “Journal of Automation, Mobile Robotics and Intelligent Systems” 02/2011, 60–65.

7. Jaffe J.S., Computer modeling and the design of optimal underwater imaging systems, “IEEE Journal of Oceanic Engineering”, Vol. 15, Iss. 2, April 1990, 101–111.

Wstępne badania trójokularowej głowicy wizyjnej do prac podwodnych

Streszczenie: Artykuł przedstawia podstawowe założenia odnośnie działania trójokularowej głowicy wizyjnej (TVS) zaprojektowanej i wykonanej do rejestracji obrazów w wodach śródlądowych przy wykorzystaniu autonomicznego pojazdu podwodnego. Opisane zostało środowisko operacyjne takiej głowicy, jakim są wody śródlądowe oraz proces powstawania informacji wizyjnej, w tym właśnie środowisku. Następnie wyjaśnione zostało pojęcie i cel fuzji obrazowej oraz proces projektowania głowicy, tj. dobór elementów optycznych i mechanicznych poparty wcześniejszymi badaniami w zadanych pasmach promieniowania.

Słowa kluczowe: fuzja obrazowa, podwodny system wizyjny, wielodomowy system wizyjny, AUV

Wojciech Biegański, MSc Eng

Graduated from the Poznań University of Technology (2009). He is a Ph.D. student at the Institute of Control and Information Engineering of the Poznań University of Technology. His interests are the mobile robotics, especially the visual perception of robots.

e-mail: wojtekbe@gmail.com

prof. Andrzej Kasiński, PhD, DSc Eng

Graduated from the Poznań University of Technology in 1973 and the Adam Mickiewicz University in 1974. He received the PhD and DSc degrees from the Poznan University of Technology in 1979 and 1998, respectively. He was a visiting professor on the Delft University of Technology and the Universidad de Murcia, ENSII Cartagena. Professor Kasiński has been the head of the Institute of Control and Information Engineering of the Poznan University of Technology since 2002. He is an author of over 150 papers and co-author of 5 patents in the fields of control theory, Pulse-Coupled Neural Network (PCNN), computer vision and biocybernetics.

e-mail: andrzej.kasinski@put.poznan.pl