Correction of projector’s nonlinearities in the vision based system for positioning of the workpieces on the CNC machines

Marek Grudziński, Krzysztof Okarma

West Pomeranian University of Technology, Szczecin

Abstract: In several last years a huge expansion of the optical measuring systems in industry and manufacturing is observed. The most advanced systems are the 3D geometry scanning systems based on images captured by cameras. A special group of them use structural light patterns projected onto the scanning surface and observed by the cameras. An appropriate coding and analysis of images allows obtaining any information about shape and size of the scanned object in relatively short period of time. Nevertheless, the 3D scanning accuracy strongly depends on complicated calibration process of the whole system, including a precise calculation of intrinsic and extrinsic parameters and correction of lens distortions both for camera and projector. The structural patterns emitted from the projector include gradients in a specified scale. Thus, using proper algorithms each pixel can be indexed and the precise depth map can be obtained. Using sinusoidal fringes illuminating the scanning surface, performing the linearisation of projector’s response is necessary in order to minimize the distortions of the sinusoidal waves. The correction method turns out to be difficult since the process depends on surface fracture, colour, and reflectivity. In this paper three methods for correction of the projector’s nonlinearity will be presented. The first one assumes a model of the nonlinear characteristics created from the gradient projected on a flat surface. The second approach is based on a series of captured images of scanned object illuminated by linearly increasing light allowing creating an individual correction curve for each pixel. The last method assumes an ideal inverted gamma correction model modified by dynamically changing coefficients calculated for each pixel. The efficiency and limitations of each method depending on the configuration of the system are also discussed in the paper as well as some experimental results obtained in the experimental vision based system for positioning the workpieces on the CNC machines.

Keywords: vision systems, 3D scanning, calibration, CNC machines

1. Introduction

The idea of the vision based positioning of workpieces on the CNC machines is based on the 3D scanning conducted using the analysis of images captured by three cameras. Nevertheless, the 3D points cloud representing the scanned object can be obtained from the acquired images only assuming the additional extraction of the depth information. The full scan of the 3D object can be obtained if each fragment of the scanned object is visible by at least one camera. Using three or more cameras there is an additional requirement of the proper data fusion from the images and geometrical fitting of the particular fragments of the obtained model.

The accuracy of such scanning strongly depends on the calibration of the system and its individual components – cameras and structural light projectors [1]. During this process the extraction of the cameras’ intrinsic parameters is conducted at first allowing further correction of some distortions introduced by imperfections of the cameras’ optical system, mainly lenses. The next step of the calibration procedure is related to the extraction of the extrinsic parameters of each camera, which are typically expressed as the rotation matrix and translation vector in a common coordinate system. In our experiments such common coordinate system is directly related to the machine tool, since the exact position of the calibration pattern is known and it is the same for all cameras during the calibration of their extrinsic parameters.

The experiments discussed in this paper have been conducted using a simplified partial 3D scanning system consisting of a single camera and a single projector. The intrinsic and extrinsic parameters of both camera and projector have been determined using Bouguet’s Camera Calibration Toolbox [2] and considering the projector as an inverse camera. The calibration of the projector have been conducted using the point-to-point mapping of the image of the calibration pattern with projected fringe patterns using the approach described by Zhang [3]. All the extrinsic parameters have been calculated in the global coordinate system with origin at the corner of the passive calibration pattern (in some applications the active displays can also be used [4–6]). The knowledge of the geometry of the system allows using the inverse projection in order to obtain the model of the 3D space of the table and its neighbourhood. The illustration of the obtained calibration accuracy is presented in fig. 1 acquired as the result of projection of the black and white chessboard is projected onto the surface of the scanned red and black physical chessboard (a white sheet of paper partially covering it has been used for better visualisation of projected image). This result has been obtained using the point-to-point mapping with full camera and projector calibration but without any correction of projector’s nonlinearities. As can be seen some of the projected points have been obtained as a result of errors.

Fig. 1. A black and white pattern projected by the calibrated projector onto the physical chessboard partially covered by a sheet of paper
Rys. 1. Czarno-biały wzorzec wyświetlony przez skalibrowany projektor na fizyczny wzorzec w postaci szachownicy częściowo przesłonięty kartką papieru

The verification of the scanning accuracy depending on the projector’s light characteristics has been conducted using three colours of fringe patterns (white, red and blue) projected on the table with printed various shades of greyscale (in particular some bright and dark regions). The results obtained applying using various correction methods are discussed and compared in the further part of the paper in terms of their possible applications in industrial environment.

2. Influence of the projector’s nonlinearities on the obtained phase and depth maps

Regardless of the accurate calibration of cameras and structural light projectors, a crucial element of the 3D scanning system is proper edge detection of the binary or sinusoidal pattern, which should be conducted with sub-pixel accuracy in order to obtain accurate depth information [7]. The photogrammetric analysis allowing further analysis of the obtained 3D points cloud representing the scanned object, requires the regular pattern for proper work. Unfortunately, most of the available projectors are equipped by their vendors with built-in gamma correction which is correlated with the logarithmic human perception of brightness and colour changes.

The value of the gamma coefficient is usually between 1.8 and 2.3 but in many cases the exact value is not provided in the projector’s documentation. For this reason using such projectors for photogrammetric analysis there is a need of experimental determination of its value in order to apply the inverse correction for the projected fringe patterns. The properly determined gamma coefficient is one of the most critical elements in the process of 3D scanning, especially using colour structural patterns [8]. Since the image captured by the camera sensor has the luminance of each pixel proportional to the intensity of light, it can be utilized in the 3D scanning systems but the light patterns captured by the camera should also be linear in such sense. In the consequence, changes of brightness of the pattern should be linearly related (proportional) to the unwrapped phase of the projected pattern and object’s depth.

Assuming the ideal sinusoidal light pattern shifted by a specified phase step, a locally linearly increasing phase map can be generated, indexing each pixel of the image in the range <0; 2p>. Depending on the number of periods, repetitive fragments with phase from 0 to 2p are obtained. A combination of sinusoidal fringe patterns with binary Gray code allows unwrapping the phase map into the range <0; 128p>.

Since the phase assignment is conducted by analysing luminance changes for each pixel, any distortions of the pattern the increments of the phase may be strongly nonlinear, leading to significant errors of the phase as shown in fig. 2.

Fig. 2. Captured images with projected fringe pattern, obtained sinusoidal signals and wrapped phase plots with (left images) and without (right images) gamma correction
Rys. 2. Zarejestrowane obrazy z widocznym światłem strukturalnym, uzyskane sygnały sinusoidalne oraz zwinięte mapy fazy bez korekcji (z lewej) oraz po korekcji gamma (z prawej)

The experiments discussed in the paper have been conducted using three methods of projector’s nonlinearity correction verified on the machine vision system for the 3D scanning and positioning of the workpieces on the CNC machine tools built at West Pomeranian University of Technology, Szczecin [9]. The results obtained for a flat surface have been compared by analysis of the mean square error between the obtained points and fitted model of the surface. During the experiments some local errors have been computed separately for dark and bright areas of the scanned flat table. The main steps of the experiments are presented in fig. 3 presenting the non-uniformly coloured table used in our experiments without and with projected fringe patterns, wrapped phase map and the unwrapped phase map directly related to the depth. It is worth to notice that all experiments have been conducted using monochrome Basler Scout 1600-14gf cameras.

3. Methods of the projector’s nonlinearity correction

3.1. The correction curve based on the polynomial model

The first method applied for the correction of projector’s nonlinearities is based on the polynomial approximation of the characteristics. The first step of the method is the identification of the projector’s response to the linearly increasing brightness of the input signal by the analysis of the image sequence captured by the camera. The camera’s response is linear so all the nonlinearities can be interpreted as related to the projector’s characteristics.

Fig. 3. Illustration of the consecutive steps from the captured image with fringe patterns by the wrapped phase map to the unwrapped phase map
Rys. 3. Ilustracja kolejnych kroków skanowania od rejestracji obrazu z oświetleniem strukturalnym, przez zwiniętą mapę fazy do rozwiniętej mapy fazy

Using a solid colour light with linearly increasing brightness in the range <0; 255> or narrower the mean luminance for each captured video frame is calculated. Such obtained data series illustrates the exponential characteristics of the projector with some small distortions introduced by the light pulsations. Nevertheless, using more images the polynomial fitting can be applied leading to the model expressed ad the polynomial of at least 4th order. Such model scaled to the range <0; 255> contains the basic information related to the deviation of each luminance level from the corresponding set value. Finally, such obtained correction curve can be applied for each pixel leading to the images with nearly linear characteristics allowing the proper creation of the phase map.

The projector’s responses and obtained polynomial models for the white solid light and the colour one are presented in fig. 4. The results obtained for red and blue light are very similar so only the results obtained for blue light are presented. Observing the achieved results smaller value if the gamma coefficient for the colour light can be easily observed (as more flattened curves). The obtained projector’s response curves are shifted down by 20 levels for a better visualisation.

Fig. 4. Polynomial models and the fitting curves obtained for the white and colour light
Rys. 4. Modele wielomianowe oraz dopasowane krzywe uzyskane dla światła białego i kolorowego

3.2. The inverse correction with variable gamma coefficient

The second method proposed in the paper is based on the separate identification of the gamma model for each pixel of the image and can be applied assuming fixed geometrical configuration of the camera and projector. Such approach can be useful for the scenes containing the objects reflecting the light in different ways, so the correction procedure should be independent on the types of the scanned surfaces, their colour and structure.

The first step of the experiments conducted by the authors is the projection of the solid light with three specified intensities: minimum IMIN, maximum IMAX and their mean value IAVG and capturing the respective images by the camera. As the result three luminance maps for the specified light intensities are obtained. The mathematical model of the gamma correction used in this method can be expressed as:

, (1)

where I0 denotes the set value of the intensity and I is the corrected luminance. Using this model the gamma coefficient can be determined for each pixel with (m, n) coordinates as:

(2)

Acquired images Imin(m, n) and Imax(m, n) determine the range of the gamma coefficient for each pixel and the image Iavg(m, n) determines the convexity of the gamma curve dependent on the local value of g(m, n) for each pixel.

The next step of the algorithm is the appropriate correction of the pixels’ luminance using the inverse model of gamma correction according to:

(3)

where Iimg is the image being corrected and Icor denotes the image obtained after correction.

3.3. Correction of the projected images

The last of the proposed methods is based on the modification of the source images instead of the correction of the images acquired by the cameras. It is assumed that the knowledge of the gamma coefficient the sinusoidal pattern can be appropriately modified in order to obtain the desired linear changes as the result of the projection. This method, similarly as the first one, requires the initial calculation of the gamma coefficient’s value, which can be conducted using the algorithm discussed above assuming the mean value of the gamma coefficients () from all pixels except the values from the range <1.05, 1.5> representing very dark pixels or being the effect of noise. The m-th column of the corrected image generated for the projection can be expressed as:

(4)

where f denotes the phase shift and m = 0…128p.

The images acquired by the camera representing the objects with such corrected fringe patterns do not require any further corrections and may be used further for determining the phase map. The illustration of the corrected sinusoidal signal for a fragment of the column is presented in fig. 5.

Fig. 5. Sinusoidal signal obtained after correction compared to the original one
Rys. 5. Sygnał sinusoidalny uzyskany po korekcji w porównaniu do sygnału oryginalnego

4. The experimental results

The results of the conducted experiments have confirmed the significant influence of the projector’s nonlinearity and its correction method on the accuracy of the 3D scanning. Depending on the chosen light colour, even 5 times lower error for the flat surface has been obtained. Nevertheless, the results can also be dependent on some local errors related to the improperly located phase map transitions. Fig. 6 illustrates the scanned fragment of the flat surface before and after applying the proposed correction of projector’s nonlinearity. The waving phase deformations visible in the left image result from the distortions of the phase map, which are much smaller using the proposed correction.

Fig. 6. Illustration of obtained results without any corrections (left image) and after the nonlinearity correction (right image) – depth changes are zoomed 10 times for better visibility
Rys. 6. Ilustracja uzyskanych wyników bez korekcji (z lewej) oraz po korekcji nieliniowości (z prawej) – zmiana głębokości została 10-krotnie powiększona dla poprawy czytelności

The comparison of the discussed correction methods has been performed experimentally by scanning the flat surface with non-uniform brightness and colours and choosing two regions: bright and dark. The 3D scanning procedure for each region has been conducted using three colours of light: red, blue and white using the sinusoidal pattern. Each scanning process has been repeated 5 times and the results have been averaged. The gamma coefficients obtained for red and blue light are much smaller (g = 1.32) than for the white light (g = 1.85), so the projector’s response is much more linear for colour patterns as supposed considering the white light as the combination of three basic RGB lights.

The results achieved using the increasing red light have been unstable causing the significant distortions in the obtained projector’s characteristics. Regardless of the fitting of the correction model the mean square error values differ significantly, probably as the effect of too strong interferences with the ambient light. Much better results have been obtained for the blue light. An interesting observation is related to slightly more accurate scanning of the dark regions than the bright ones regardless of the used method of nonlinearity correction. The results achieved for the blue light turned out to be the most similar in each experiment.

The application of the white light has also led to comparable mean squared errors despite of the strong nonlinearity. The best results have been obtained for the direct correction of the projected images but two other methods have also caused the significant decrease of the error.

The illustration of the obtained results is presented in fig. 7, where the advantages of the third correction method (direct correction of projected images) for the blue light can be easy observed. The numbers of the correction method denote: 1 – polynomial fitting, 2 – inverse correction with variable gamma coefficient, 3 – correction of projected images, 4 – no correction.

Fig. 7. Comparison of the mean square errors obtained for different regions of the scanned surface with indicated minimum, average and maximum error obtained for each method in 5 independent experiments
Rys. 7. Porównanie błędów średniokwadratowych dla różnych fragmentów skanowanej powierzchni (zaznaczono wartości błędu: minimalną, średnią oraz maksymalną) uzyskanych poszczególnymi metodami dla 5 nieza-leżnych eksperymentów

The other experiments have been conducted for the whole flat table of size 160 mm × 160 mm with local fitting of the surface model to the points cloud using 64 regions of size 20 mm × 20 mm. Due to the local fitting the influence of some potential irregularities of the physical surface should be minimized. The obtained results as the average values of 64 local mean squared errors are presented in fig. 8. The table used in all experiments and the obtained error map are presented in fig. 9.

Fig. 8. Comparison of the mean square errors obtained for the whole surface (average values from 5 experiments)
Rys. 8. Porównanie błędów średniokwadratowych uzyskanych dla całej powierzchni (wartości średnie z 5 ekspe-rymentów)

Fig. 9. The table with non-uniform brightness utilized in our experiments and obtained error map
Rys. 9. Tablica z nierównomiernym rozkładem jasności wykorzystywana w eksperymentach oraz uzyskana mapa błędu

The application of the white light causes significant errors, which can be significantly reduced only using the third correction method. Nevertheless, the application of the blue light always leads to better results. An interesting observation is also the increase of error for the red light after the correction. The first two correction methods introduce errors related to the position of some points in the 3D space so the changes of local error level are higher. These errors can be caused by the sensitivity of the two discussed postcorrection methods to the noise which may be present in the captured images. As shown in the error map (fig. 9) the highest errors are obtained for the dark areas of the table and in the place of physical deformation of the scanned table.

5. Summary

Correction of the projector’s nonlinearity influencing the accuracy of the 3D scanning in the discussed vision system for positioning the workpieces on the CNC machines should be conducted using the direct correction of projected images gamma coefficients. The best results can be obtained using the blue patterns, similarly as in some commercial systems such as e.g. ATOS scanner developed by GOM company. In the laboratory conditions the blue light is relatively least sensitive to external light (e.g. infra-red) and bright enough as well. For the strong ambient light conditions the white patterns can also be used.

In the further work the verification of the complete scanning accuracy is planned, which requires the proper fusion of 3D data acquired from three cameras as well as the filtering of some undesired points, which may appear in the cloud representing the scanned object as result of some errors.

Acknowledgements

This work was done partially under the Polish Ministry of Science and Higher Education grant no. N502 147238 The use of video techniques for positioning workpieces on CNC machine tools; Contract No. 1472/B/T02/2010/38.

Bibliography

Komagata H., Ishii I., Takahashi A., Wakatsuki D., Imai H., A Geometric Method for Calibration of Internal Camera of Fish-Eye Lenses, “Systems and Computers in Japan”, Vol. 38, No. 12, 2007, 55–65.

Bouguet J.-Y., Camera Calibration Toolbox for MATLAB, 2005, [www.vision.caltech.edu/bouguetj/calib_doc].

Zhang S., High-resolution, real-time 3-D shape measurement, PhD thesis, Stony Brook University, 2005.

Funk N., Yang Y.-H., Using a Raster Display for Photometric Stereo, Proc. 4th Canadian Conf. Computer and Robot Vision, Montreal, 2007, 201–207.

Mazurek P., Mobile System for the Estimation of Internal Parameters of Distributed Cameras, “Pomiary Automatyka Kontrola”, Vol. 56, No. 11, 2010, 1356–1358.

Zhan Z., Camera Calibration Based on Liquid Crystal Display (LCD), The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XXXVII, Part B3b, 2008.

Okarma K., Grudziński M., Poprawa dokładności detekcji krawędzi dla systemów skanowania 3D opartych na projekcji wzorców prążkowych, „Pomiary Automatyka Kontrola”, Vol. 57, No. 7, 2011, 721–725 (in Polish).

Zhang Z., Guo T., Absolute Phase Measurement Based on Combining Binary Color-Code and Phase-Shift Light Projection, Proc. Int. Conf. Advanced Phase Measurement Methods in Optics and Imaging (AIP), Monte Verita, Ascona, Italy, Vol. 1236, 2010, 427–432.

Domek S., Dworak P., Grudziński M., Okarma K., Pajor M., Układ wizyjny do skanowania geometrii i pozycjonowania przedmiotu obrabianego na obrabiarce CNC, „Modelowanie Inżynierskie”, Vol. 10, No. 41, 2011, 39-48 (in Polish).

Korekcja nieliniowości projektorów w wizyjnym systemie pozycjonowania przedmiotów obrabianych na obrabiarkach sterowanych numerycznie

Streszczenie: W ciągu ostatnich kilku lat obserwuje się znaczną ekspansję optycznych systemów pomiarowych w zastosowaniach przemysłowych. Jednymi z najbardziej zaawansowanych są systemy skanowania trójwymiarowego oparte na analizie obrazów zarejestrowanych przez kamery. Szczególną grupę tych metod stanowią algorytmy wykorzystujące oświetlenie strukturalne skanowanej powierzchni znajdującej się w polu widzenia kamery. Stosując odpowiednie kodowanie i algorytmy analizy obrazów możliwe jest pozyskanie informacji dotyczących kształtu i wymiarów skanowanych obiektów w stosunkowo krótkim czasie. Jednakże dokładność skanowania jest silnie uzależniona od złożonego procesu kalibracji całego systemu, włącznie z dokładną kalibracją parametrów wewnętrznych i zewnętrznych oraz korekcją zniekształceń układów optycznych zarówno kamer, jak również projektorów. Wzorce strukturalne wyświetlane przez projektory mają charakter gradientowy dla ustalonej skali, co pozwala na odpowiednie indeksowanie każdego piksela i precyzyjne odtworzenie map głębokości. Zakładając wykorzystanie wzorców sinusoidalnych rzutowanych na skanowaną powierzchnię, konieczna jest linearyzacja odpowiedzi projektorów w celu minimalizacji zniekształceń kształtu sinusoidy. Dobór metody korekcji nie jest jednak oczywisty ze względu na zależność od gładkości, koloru czy też połyskliwości powierzchni. W artykule zaprezentowano trzy metody korekcji nieliniowości charakterystyk projektorów. Pierwsza z nich wykorzystuje model charakterystyki projektora utworzony na podstawie obserwacji światła gradientowego na płaskiej powierzchni. Druga metoda jest oparta na serii zdjęć pozyskanych dla liniowo narastającego natężenia światła, co pozwala na stworzenie niezależnej krzywej korekcji dla każdego piksela. Ostatnia z metod wykorzystuje model idealnej odwrotnej charakterystyki gamma modyfikowany przez zmienne współczynniki wyznaczane w sposób dynamiczny dla każdego piksela. W pracy porównano również efektywność i ograniczenia każdej z metod w zależności od konfiguracji systemu, jak również przedstawiono wyniki eksperymentalne uzyskane z wykorzystaniem prototypowego wizyjnego systemu pozycjonowania przedmiotów obrabianych na obrabiarkach CNC.

Słowa kluczowe: systemy wizyjne, skanowanie 3D, kalibracja, obrabiarki sterowane numerycznie

Marek Grudziński, MSc Eng.

Graduated in automation and robotics from West Pomeranian University of Technology, Szczecin, Faculty of Electrical Engineering. His scientific interests include mechatronics and automation, especially manufacturing systems, applications of vision based methods and photogrammetry. He is a co-author of several scientific publications and currently is a PhD student at the Faculty of Mechanical Engineering and Mechatronics.

e-mail: marek.grudzinski@zut.edu.pl

Krzysztof Okarma, PhD Eng.

Assistant Professor in the Department of Signal Processing and Multimedia Engineering, West Pomeranian University of Technology, Szczecin. Graduated from Faculty of Electrical Engineering (1999) and Computer Science (2001), received his Doctor of Technical Sciences degree (2003) from Faculty of Electrical Engineering, where he is the Deputy Dean for educational affairs. Author of over 130 publications related to signal and image processing and analysis.

e-mail: okarma@zut.edu.pl