A novel approach for automation of stereo camera calibration process

Piotr Garbacz, Wojciech Mizak

Institute for Sustainable Technologies – National Research Institute, Radom, Poland

Abstract: The problem of stereo camera calibration has been studied for over many years by numerous researchers. A crucial task in this process is to discover the transformation between 3D world coordinates and 2D pixel image coordinates of image. The growth of the number of different applications of stereovision systems has led to specialization of developed calibrations algorithms. Nowadays, various calibration objects and self-calibration techniques are used. This paper presents a unique automatic calibration system for a stereovision system for inspection of specimen surface under fatigue tests. In order to allow analysis of surface in both a micro and macro scale, the system has been equipped with cameras with motorized focus and zoom lenses. The proposed calibration system is based on mechatronic framework which allows the use of a set of 2D plane calibration targets with varying size of region of interest. Such a solution allows automation of calibration process and guarantees repeatability of results with an assumed error.

Keywords: stereovision, calibration, fatigue monitoring

1. Introduction

Calibration is a critical task for a stereovision analysis of surface. It is a necessary step in a correct 3D reconstruction of objects surface and the key factor for accuracy of such a process. Depending on the chosen camera model, there are different parameters to be determined. In the pinhole camera model, used in this research, there are two groups of parameters:

– Intrinsic parameters that describe internal geometric and optical camera characteristics,

– Extrinsic parameters that describe camera position and orientation in a word reference system.

The quality of the calibration process has direct impact on stereovision measurement uncertainty [1]. Building a stereovision based system it is essential for ensuring a suitable calibration technique for required application. Numerous methods of stereo camera calibration are currently available. Self-calibration algorithms [2] are often used in robotics because of the ease of implementation. Although it is possible to perform calibration without any patterns; the results strongly depend on the quality of datasets chosen. Therefore, when it comes to high-precision measurement, special calibration targets are applied.

This paper discusses a novel method of calibration for the stereovision system for fatigue process monitoring [3]. Figure 1 illustrates the structure of the system dedicated to monitor fatigue.

Fig. 1. General view of the fatigue monitoring system mounted onto the fatigue testing machine
Rys. 1. System do monitorowania procesów zmęczeniowych zainstalowany na maszynie wytrzymałościowej

The modular structure of the system enables fast reconfiguration and adaptation for fatigue tests analysis in various scales of observation. Changing any physical parameter of the system requires the calibration process to be performed every time. A suitable fast and uniform method is required to ensure good calibration results in both a micro and macro scale analysis.

2. Calibration methods

In recent years, many researchers have been working to develop an optimal calibration technique for their applications. Depending on the requirements, the existing solutions can be grouped into four categories [4]:

– autonomous: do not require any action from an operator, e.g. giving initial guesses for optimization algorithm,

– accurate: a calibration technique guarantees high measurement accuracy,

– effective: optimization is limited to a few calibration parameters, hence it allows implementation of a fast and low-cost algorithm,

– versatile: guarantees autonomous and uniform procedure for various applications, a wide range of accuracy and optical setups.

An important step in developing calibration system is to choose a suitable calibration target [5]. Historically, 3D calibration targets were the first used targets. Calibration was performed by observing a special object that consisted of two or three planes, orthogonal to each other. 3D geometry dimensions characterizing such a target were precisely known. Also, attempts were made to use special mechanical constructions which allow change of 3D coordinates of test points with high accuracy [6]. Methods that use 3D targets guarantee good calibration results but their disadvantage is that expensive equipment and elaborate installation are required. Techniques that use 2D plan based calibration are more popular. The installation is much easier than the traditional methods: The calibration process requires only observation of a planar pattern shown at a few different orientations. The knowledge of the plane motion is not required. There are also new techniques based on 1D objects composed of a set of collinear points. These experimental methods are used mostly in multi-camera systems. Other solutions include self-calibration techniques that do not require any calibration object. These methods are based on rigidity of the static scene observed by a moving camera.

A suitable camera model is crucial to the calibration method. It provides a mathematical description of the physical processes occurring between the scene and the imaging plane. When it comes to high-precision measurements, it is advisable to use perspective transformation with a lens distortion model. However, depending on the application requirements, a calibration algorithm can consider only few parameters from the full camera model, e.g. ignore radial or tangential distortion. There are three most widely used plane-based algorithms [7]:

– Direct Linear Transform (DLT): in this algorithm, lens distortion is ignored. In the first step, the linear transformation from the object coordinates (xi, yi, zi)
to image coordinates (ui, vi) is solved. The projection matrix P is a 3 × 4 matrix. The coefficients of the matrix do not have any physical meaning. Also, there are decomposition techniques for extracting some physical camera parameters from DLT matrix; only a subset of them can be estimated.

– Tsai’s Method: the lens distortion effect is restricted to radial distortion in this algorithm. Skewness is also
ignored. Tsai’s algorithm is a two-stage process. No
initial guesses are required. By simplifying a camera model, a significant part of the computation is linear. Originally, this method requires only a single view of the non-coplanar calibration pattern but it can be adopted to be used with multiple views of the
coplanar calibration pattern. The first stage of the process determines most of the extrinsic parameters. In the next step, one radial distortion factor is determined and parameters estimated in previous stage are adjusted in non-linear optimization.

– Zhang’s Method: it is the latest technique that makes use of advanced concepts in projective geometry. The calibration procedure uses correspondences to determine homography transformation between the calibration plane and the image. In the first stage, camera parameters are estimated by analytical solution. Afterwards, a nonlinear technique based on the maximum likelihood criterion is used to optimize initial estimation. In the next step, radial distortion coefficients are estimated by solving linear least-squares. In the last stage, all parameters are refined by non-linear optimization. In a normal scenario, this method requires viewing a calibration target from few different locations and orientations.

In this study, the Zhang’s method available in OpenCV library is implemented [8]. The perspective transformation is based on a pinhole camera model:

(1)

where:

(X, Y, Z) – 3D point in the world coordinates system,

(u, v) – coordinates of the projection point in pixels,

(cx, cy) – principal point,

(fx, fy) – horizontal and vertical focal lengths.

Lens distortion is modeled by six radial and two tangential coefficients.

In the last stage, the quality of the calibration process is analyzed. In case that a 3D position in real world coordinates of calibration target points can be determined with high accuracy, there are several different methods of evaluating the quality of the calibration process [3]. One of them is Normalized Stereo Calibration Error [9]. For known coordinates (xi, yi, zi) of a given point and corresponding coordinates estimated by the 3D reconstruction process, the NSCE value can be calculated using the following formula:

(2)

where:

fx – horizontal focal length,

fy – vertical focal length,

n – number of control points.

However, in most cases it is much easier to apply methods that do not require to determinate real 3D world coordinates of test points. In practice, the main tool for evaluating camera calibration that is commonly used is the reprojection error. Let Pi be the projection matrix of a camera for the i-th calibration view. For detected n-grid points xj in the image, corresponding to 3D world coordinates of planar points Xj. The reprojection error value can be calculated by the following formula [10]:

(3)

It is common to evaluate the quality of calibration by RMS (Root Mean Square) target reprojection error for all calibration views using the following formula:

(4)

The RMS value is a useful measure of how well calculated cameras parameters correspond to actual system setup. This value should be as close to zero as possible, but in most of application calibration, the value under 0.3 is acceptable [11].

3. Automatic Calibration System

The proposed calibration method is dedicated for the experimental stereovision system for fatigue process monitoring. The main advantage of such a solution is flexibility. The vision modules are fixed on the 4-degrees of freedom positioning sets, which allow the adjustment of the camera position in relation to the specimen (fig. 2).

Fig. 2. View of the model of the vision module: a) degrees of freedom, b) basic geometrical parameters
Rys. 2. Widok modelu modułu wizyjnego: a) stopnie swobody,  b) podstawowe parametry

Table 1 presents selected parameters of the stereovision system. In order to ensure the required measurement resolution, the CCD Basler Pilot cameras with 2448 × 2050 pixels sensors were used. Due to the necessity to adjust optical parameters of the lens to the observation scale, the motorized lens Zoom 6000 by Navitar was selected [12]. Depending on the vision system setup (fig. 3), it is possible to analyze specimen surface in both a macro and micro scale.

Tab. 1. Parameters of fatigue monitoring vision system
Tab. 1. Parametry systemu wizyjnego do monitorowania procesów zmęczeniowych

Parameter

Value

Working distance WD

80 mm ¸ 360 mm

Field of view FOV (dimension H)

1 mm ¸ 50 mm

Maximum optical resolution

3 µm

Pan angle α

16º ¸ 110º

A wide range of parameters of fatigue monitoring vision system is an advantage with comparison to other stereovision systems which are not equipped with a positioning mechanical module or motorized lens system [14–15].

On the other hand, a suitable fast and uniform calibration method is required, which will guarantee good calibration results in full range of available FOV.

Fig. 3. Lens system setup for nominal working distance 356 mm
Rys. 3. Konfiguracja modułów obiektywu dla nominalnej odległości widzenia 356 mm

The following assumptions were made to select a proper calibration target: A calibration method should fulfill multi-scale condition and guarantee fast and approachable adjustment system for different ROI sizes. Therefore, and due to limited depth of field of the applied motorized lens system, only planar 2D calibration targets were considered. Active calibration targets proposed in [11] were discarded because they require micro scale observation in dedicated stereovision system. The proposed calibration system uses a set of a wide range of size planar calibration targets. Calibration module (fig. 4) enables precise positioning of the calibration target in two orthogonal planes XY and XZ. It is installed directly in a fatigue test machine gripper by cantilever-1, in the place where normally a specimen is mounted. On the base of the module, Newport’s high precision rotation stage-2 is installed, which can be rotated from 0º to 360º with fine tuning 5º. Rotation of the calibration target in XZ plane is available in a range of ±5º by goniometer-4 placed on adapter-3. The calibration target-7 is mounted in positioner-6. Motion of rotation and goniometer stages is executed by ultra-high resolution piezo-actuators-5. The maximum size of the calibration target corresponds to desired 50 × 50 mm FOV of the vision system.

Fig. 4. Model of the calibration module: 1 – cantiliver, 2 – rotating stage, 3 – adapter, 4 – goniometer stage, 5 – piezo actuator, 6 – targets positioner, 7 – calibration target
Rys. 4. Model modułu kalibracji: 1 – wspornik, 2 – stolik obrotowy, 3 – adapter, 4 – goniometr, 5 – piezonapęd, 6 – pozycjoner wzorca, 7 – wzorzec

Piezo-actuators are controlled by ASCII commands via RS-485 port and synchronized with the camera acquisition process. Minimum incremental linear motion of the actuators is 30 nm, which is an important parameter, due to limited depth of field of applied lens system (fig. 5).
In order to avoid blurred image regions, which is fundamental for good calibration results [13], high precision positioning of calibration targets is required. Fig. 5 shows that rotation of the calibration target is limited by the DOF (Depth of Field) value that varies depending on the lens system setup and the magnification value (tab. 2).

Fig. 5. Depth of field of vision system
Fig. 5. Głębia ostrości systemu wizyjnego

Table 2 shows that acquisition of sharp images during manual positioning of the calibration target is practically impossible for micro scale observation and very difficult and time-consuming process for larger field of views.
Proposed system guarantees the optimal quality of input images for calibration by implemented sequence of rotations.

Tab. 2. Navitar Zoom 6000 Field of View Matrix [12]
Tab. 2. Macierz pola widzenia obiektywu Navitar Zoom 6000

Parameter

Mag.

Lens system setup

WD = 356 mm, 0.25 attachment, 0.67X Telescope

WD = 113 mm 0.75X attachment, 3.5X Telescope

DOF [mm]

Low

13.89

1.73

High

1.54

0.18

FOV [mm]

Low

93.62

5.98

High

14.66

0.93

Before starting the calibration procedure, it is required to adjust camera orientations, set the optimal focus position in the motorized lens system and set the optimal lighting conditions. In order to adjust camera orientations that were previously set manually by an operator, the visual feedback system is applied (fig. 6).

Fig. 6. Visual feedback system for cameras orientation adjustment
Fig. 6. Wizyjna pętla sprzężenia zwrotnego systemu korekcji orientacji kamer

The laser beam is used to project the cross shape target pattern. The visual tracking system for both cameras is applied to detect vertical and horizontal lines. An implemented algorithm is used to adjust camera orientations to point directly in the centre of target cross. The second important step to validate the calibration procedure is to set the optimal focus value. The implemented algorithm is based on 2D Discrete Fourier Transformation. Navitar’s motorized lens system uses DC Servos with Encoder (fig. 7) that are controlled via RS-232 port in a range up to 3000 steps. Magnetic Hall-Effect sensors are used to reference position location. Starting from the low position limit of DC servo, the maximum focus value is searched and saved as a system parameter. The last stage before starting the calibration procedure is to adjust the optimal lighting conditions for detection of the calibration target. LED ring illuminators with the illuminance of ca. 18 klux at the distance of ca. 300 mm from the sample were used. Exposure time of Basler cameras is controlled via GigE Vision standard. For the best performance of calibration algorithm, the histogram analysis of acquired images is done to guarantee the optimal contrast value.

Fig. 7. Camera positioning module: 1 – piezo actuator, 2 – zoom DC servo with encoder, 3 – focus DC Servo with encoder, 4 – CCD camera
Rys. 7. Moduł pozycjonowania kamer: 1 – piezonapęd, 2 – silnik DC z enkoderem do regulacji powiększenia, 3 – silnik DC z enkoderem do regulacji ostrości, 4 – kamera CCD

Images with lower contrast results in longer execution time of implemented algorithm for circular grid detection. The calibration procedure requires several images of calibration target in different orientations. Images acquired from both cameras are analyzed during the calibration process. The whole process is automated and does not require any action from an operator. Calibration target points are detected autonomously by the implemented algorithm using full resolution (5 Mpx) images. The calibration process is done using a circular grid containing 65 control points. Developed software takes the advantage from multithreading which allowed for the implementation of fast executing code. Execution time is less than 2 minutes with the RMS error value below 0.2.

4. Conclusions

The proposed method is dedicated for stereovision system for fatigue process monitoring. A condition of scalability is fulfilled by designed calibration module that allows use of a set of a wide range of size planar calibration targets.

Therefore, it is possible to analyze the 3D surface in both a micro and macro scale. Ultra-high resolution piezo-actuators are used for automation of the calibration procedure. The automated procedure avoids positioning calibrations target out of focus, which is important due to limited depth of field of the applied vision system. The proposed solution reduces calibration time and guarantees repeatability of the calibration results below, the reprojection error assumed. Implemented algorithm is fully autonomous. No specialised knowledge or actions is required from an operator; that makes the calibration process straightforward compared to other existing solutions. Further work will focus on optimization of the system, which includes research on defining the finest calibration target. Both the size of the pattern and shape of control points will be considered.

Acknowledgments

Scientific work created within the framework of the “Innovative Systems of Technical Support for Sustainable Development of Economy” Strategic Programme within the Innovative Economy Operational Programme.

References

1. Wang Y.-Q., Sutton M.A., Ke X.-D., Schreier H.W., Reu P.L., Miller T.J., On Error Assessment in Stereo-based Deformation Measurements. Part I: Theoretical Developments for Quantitative Estimates, Strain, 2009, Vol. 45, 160–178.

2. Dang T., Continuous Stereo Self-Calibration by Camera Parameter Tracking, “Transactions on image processing”, Vol. 18, No. 7, 2009, 1536–1550.

3. Giesko T., Dual-camera vision system for fatigue monitoring, Materials Science Forum, Vol. 726, 226–232.

4. Tsai R.Y., A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-Shelf TV Cameras and Lenses, IEEE “Journal on Robotics and Automation”, 1987, Vol. 4, 323–344.

5. Liangfu L., Zuren F, Yuanjing F., Accurate Calibration of Stereo Cameras for Machine Vision, “Journal of Computer Science & Technology”, 2004, 147–151.

6. Urqunhart C.W., Siebert J.P., McDonald J.P., Fryer R.J., Active Animate Stereo Vision, The British Machine Vision Conference, Surrey, 1993, 75–84.

7. Zollner H., Sablatnig R., Comparison of Methods for Geometric Camera Calibration using Planar Calibration Targets, 28th Workshop of the Austrian Association for Pattern Recognition, Hagenberg, 2004, 237–244.

8. [http://docs.opencv.org] – OpenCV v.2.4.3.

9. Weng J., Cohen P, Herniou M., Camera Calibration with Distortion Models and Accuracy Evaluation, “IEEE Transactions on Pattern Analysis and Machine Intelligence”, 1992, 967–970.

10. Bradley D., Heidrich W., Binocular Camera Calibration Using Rectiﬁcation Error, Canadian Conference on Computer and Robot Vision, 2010.

11. Schmalz Ch., Forster F., Angelopoulou E., Camera Calibration: Active versus Passive Targets, “Optical Engineering”, Vol. 50, No. 11, 2011.

12. [http://www.navitar.com]

13. Ouellet J.N., Hebert P., Developing Assistant Tools for Geometric Camera Calibration: Assessing the Quality of Input Images, ICPR 2004, [in:] Proceedings of the 17th International Conference, Vol. 4, 2004, 80–83.

14. Sharpe W.N. ,Springer Handbook of Experimental Solid Mechanics, Springer, 2008, 589-597.

15. Yan J.H., Sutton M.A., Deng X., Wei Z., Mixed-mode crack growth in ductile thin-sheet materials under combined in-plane and out-of-plane loading, “International Journal of Fracture”, Vol. 14, No. 4, 2009, 297–321.

Automatyzacja procesu kalibracji systemu stereowizyjnego

Streszczenie: Zagadnienie kalibracji kamer w systemach stereowizyjnych było w ostatnich latach podejmowane przez wielu badaczy. Głównym celem tego procesu jest wyznaczenie transformacji pomiędzy współrzędnymi 3D punktów w układzie globalnym i odpowiadającymi im współrzędnymi pikselowymi 2D na płaszczyźnie obrazowania kamer. Coraz większa liczba obszarów, w których zastosowanie znajdują systemy stereowizyjne, doprowadziła do specjalizacji wykorzystywanych w nich metod kalibracji kamer. Aktualnie stosowane są różnorodne wzorce kalibracyjne, a także metody samokalibracji kamer. W artykule przedstawiony został autorski zautomatyzowany system kalibracji kamer dedykowany dla systemu stereowizyjnego, umożliwiającego monitorowanie procesów destrukcji materiałów na maszynie wytrzymałościowej. Dla zapewnienia możliwości przeprowadzenia badań zarówno w skali mikro jak i makro, system ten wyposażony jest w zmotoryzowane obiektywy, umożliwiające regulację zbliżenia i ostrości. Prezentowany system kalibracji poprzez zastosowanie mechatronicznego układu pozycjonowania oraz odpowiedniego typoszeregu wzorców płaskich, zapewnia kalibrację kamer w pełnym zakresie obserwacji. Rozwiązanie to umożliwia automatyzację oraz gwarantuje powtarzalność procesu kalibracji.

Słowa kluczowe: stereowizja, kalibracja, badania wytrzymałościowe

Piotr Garbacz, MSc

Assistant at the Institute for Sustainable Technologies – National Research Institute in Radom. He received Master degree in robotics from Warsaw University of Technology in 2010. He works in the area of vision systems and robotics.

e-mail: piotr.garbacz@itee.radom.pl

Wojciech Mizak, MSc

Assistant at the Institute for Sustainable Technologies – National Research Institute in Radom. Responsible for prototypes of mechanical and mechatronics design in the field of automatic optical inspection, thermography and erosive wear testing.

e-mail: wojciech.mizak@itee.radom