Experimental demonstration of Measurement System Analysis

Artur Król*, Wojciech Czaja*, Gabriel Kost*, Piotr Czop**, Grzegorz Wszołek*, Dawid Jakubowski***

*Institute of Engineering Processes Automation and Integrated Manufacturing Systems, The Silesian University of Technology, **AGH University of Science and Technology, ***GS-Hydro Sp. z o.o., Ruda Śląska

Abstract: This work presents the application of Measurement System Analysis (MSA) and the advantages of the Six Sigma approach in the validation of a servo-hydraulic tester. The diagnosis of repeatability and reproducibility by using repeatability (R) and reproducibility (R) gage analysis (so called Gage R&R Type II tool) which shows the correct operation of the measuring system including uncertainty of staff and measuring instrumentation. The Gage R&R Type II tool is based on the analysis of variance, called ANOVA. The structural approach to the subject has also provided knowledge of the error of the measurement system.

Keywords: measurement system analysis, MSA, variance analysis ANOVA, validation, servo-hydraulic tester

1. Introduction

The measurement system is a key aspect for each research and development work. Usually, measurements from advanced computerized systems are treated as a perfect source of data with a zero measurement system. This thinking can lead to the acceptance of measurement data without any critical look at the measurement system, and making conclusions based on incorrect data. There are very well-known measurement system analysis methods, but classically many of them concentrate on the calibration process. The automotive industry has developed the best practice method to perform measurement system analysis and asses the measurement error.

The main concept of MSA divides measurement error components into the following groups [2]:

– Accuracy:

• Calibration/Bias

• Linearity

• Stability

– Precision:

• Repeatability

• Reproducibility

In fact, the Accuracy part represents the calibration process because calibration/bias is meant to calibrate a given measurement system in line with the master system in order to give correct read out, and therefore to have zero read out at a zero point, which is called bias. The next component, linearity, is the calibration made over the whole range of interest. The last aspect within accuracy is stability of the measurement system over a period of time or other parameter related to time or environment, such us temperature or pressure.

The precision component of measurement error takes into account not only the complete measurement system, but also the measurement method, operators and measured specimen. Repeatability shows the difference between the two extreme measured values repeated for the same part per the same operator. Reproducibility is the difference between the average values of measurement carried out for the same part for different operators. There is a statistical tool called Gage R&R, used by Six Sigma methodology, to assess the precision of the measurement system [2]. It needs to be stressed that Gage R&R will show how precise the measurement can be, but this tool will not ensure calibration and other accuracy related aspects.

2. Measurement System Accuracy

The measurement system being evaluated is a servo-hydraulic system designed to measure force – velocity characteristics of components targeted to provide damping functions. The servo-hydraulic machine consists of a hydraulic actuator, and the system is equipped with sensors measuring displacement and force [1]. The performance of the machine executing a given displacement signal over time is kept by means of a controller with a closed control loop. This machine set-up ensures stable behavior and theoretically correct measurements performed on this measurement system. However, as described above, complete measurement system analysis also requires an operator, measurement method and the investigated parts to be included into the measurement evaluation experience. It has been agreed to perform the following steps in order to understand measurement systems [2]:

Calibration of sensors used in the system is treated as being correctly performed by an external laboratory. The reason to skip this step is that it requires special equipment, and additionally this activity has recently been performed. The mentioned calibrations cover two of the three accuracy aspects: (i) accuracy/bias, (ii) linearity – as the calibration was performed over a complete range of interest.

The next step is to check the stability of the machine with these experiments:

The stability of machine performance, which is the achieved velocity based on the requested signal. The requested signal is sin wave, velocity is measured as the maximum for two stroke directions, named rebound and compression. To assess the stability of the machine, 30 measurements were performed on a consolidated sin wave signal with 5 velocities. As a result, we gained an evaluation of 5 achieved velocities for 2 directions of actuator movement. For the compression movement we have the following standard deviation values of the achieved velocities [m/s] (tab. 1).

Tab. 1. Standard deviation values velocity for compression
Tab. 1. Odchylenie standardowe wartości prędkości kompresji

Variable

Velocity1

Velocity2

Velocity3

Velocity4

Velocity5

Standard

Deviation

0.000341

0.000352

0.000728

0.000705

0.000849

Fig. 1. Stability for velocity at compression stage
Rys.1. Stabilność prędkości kompresji

The relative standard deviation, calculated as a standard deviation divided by the mean value of speed, is around 0.1 %, which provides a very good and acceptable result. The graph in fig. 1 presents 30 repetitions of the same signal for the highest velocity. The black points show the average values for each repetition based on individual values from 5 loops for each velocity.

The same approach for analysis was used during the rebound (extension) stage, where the standard deviations for all velocities are the following (tab. 2).

Tab. 2. Standard deviation values velocity for rebound
Tab. 2. Odchylenie standardowe wartości prędkości rozprężania

Variable

Velocity1

Velocity2

Velocity3

Velocity4

Velocity5

Standard Deviation

0.000493

0.000250

0.000679

0.000768

0.000773

Fig. 2. Stability for velocity at rebound stage
Rys. 2. Stabilność prędkości rozprężania

The graphical representation of the maximum velocity for the rebound stage is shown in fig. 2.

Stability for the master piece, which is the mono-tube based damper known as stable damping components. The test was performed with the mentioned sin wave signal with 5 increasing velocities. In this case the measured output is not the velocity but the force measured at maximum speed. The graph in fig. 3 shows 30 repetitions of measurement for maximum speed during the compression stage.

The graph clearly shows an issue with the stability of the damping forces of the measured unit. There are multiple potential results of damper instability, however this is not the aim of this paper. The instability is presented in tab. 3 as standard deviation per each velocity.

Tab. 3. Standard deviation values force for compression
Tab. 3. Odchylenie standardowe wartości sił kompresji

Variable

Compression1

Compression2

Compression3

Compresion4

Standard Deviation

9.31

9.19

13.75

18.0

Fig. 3. Force stability for compression
Rys. 3. Stabilność siły kompresji

The standard deviation is very often used to present the confidence interval around mean values. In this case, the Confidence Interval (CI 95 %) for compression force at max speed would be ±36 N. As the variation of the force is not the same over 30 repetitions it can be allowed to assess the common variation of measured force over 1 repetition by means of – the Pooled Standard Deviation (PSD) calculated for each velocity are presented in tab. 4.

Tab. 4. Pooled standard deviation values velocity for compression
Tab. 4. Łączne odchylenie standardowe wartości prędkości kompresji

Variable

Compression1

Compression2

Compression3

Compresion4

PSD

2.15

2.66

3.85

5.51

Fig. 4. Force stability for rebound at maximum speed
Rys. 4. Stabilność siły rozprężania przy maksymalnej prędkości

Based on the pooled standard deviation for maximum speed we can say that CI 95 % is equal to ±11 N, which means that 95 % of single measurement is within the range of ±11 N. This value is much better, however it is valid only for 1 single measurement case.

The situation for rebound force at maximum speed is presented in fig. 4.

The standard deviations and pooled standard deviations are the following in tab. 5.

Tab. 5. Standard and pooled standard deviation values force for rebound
Tab. 5. Odchylenie standardowe i łączne odchylenie standardowe wartości sił rozprężania

Variable

R1

R2

R3

R4

Standard Deviation

55.3

32.7

15.6

18.4

Pooled Standard Deviation

14.1

8.15

10.4

10.7

Fig. 5. Damping forces measured over a period of time
Rys. 5. Zmierzone siły tłumienia w okresie czasu

Here the situation for the rebound stage is very similar to compression. Even though we can observe better stabilization of damper performance over time, the pooled standard deviations are similar over the velocities.

The stability over time exercise was meant to show the influence of different factors related to the time domain. There are 30 tests performed over a certain period of time with usually 4 repetitions in a row. The graph in fig. 5 shows damping forces for the rebound stage.

The pattern of decreasing forces in groups of 4 (in one case, 8) repetitions is easily explained because measurements were performed in rows of 4 tests, increasing the temperature, which leads to a damping force decrease due to lower viscosity of the oil. This experiment clearly showed the issue with stability of the damper and the strong influence of temperature as a noise factor. This means that the measurement system analysis performed based on a damper must be carried out with the elimination of all noise factors influencing the final measurement.

3. Measurement System Precision

In order to understand the measurement error components called repeatability and reproducibility, we need to use the Gage R&R statistical tool, which is based on the analysis of variance – ANOVA [4]. The standard approach for this experiment is to measure a minimum of 10 parts measured by 2 operators, each operator has to measure at least 2 times. In the presented case, there are 2 operators testing the same specimen 30 times over the complete test sequence, consisting of 4 test velocities, creating full force over the velocity characteristic for rebound and compression movement [2]. Nested Gage R&R was used to analyze the obtained results, and the statistical and graphical results for the first velocity for the rebound stage are presented in fig. 6. If the measurement system was perfect, we should see 30 identical values obtained for each of the operators, additionally, with no difference between operators. This would lead to a standard deviation equal to zero, when in our case we see that standard deviation coming from the measurement system is above 40 [N]. The six standard deviations show that almost 100 % of the measurement system is equal to 52.56 % of the tolerance band (tab. 6). This result is unacceptable as a marginally acceptable measurement system should not give a value above 30 %, according to the AIAG MSA manual. The huge difference arises from the different ways used for measurement between Operator 1 and Operator 2 (lower – right fig. 6).

Fig. 6. Gage R&R results for rebound, first velocity, tolerance band is 300 N
Rys. 6. Wyniki Gage R&R rozprężania, pierwsza prędkość, tolerancja 300 N

Tab. 6. Contribution percent rate
Tab. 6. Udział procentowy

Source

Variation Compression

Contribution [%]

Total Gage R&R

Repeatability

Reproducibility

1629.46

52.56

167.26

5.40

1462.20

47.16

Part-To-Part

1470.81

47.44

Total Variation

3100.27

100.00

Standard proces tolerance of study variation is classificate at value 300 N (tab. 7).

Tab. 7. Percent rate for tolerance elements of Gage R&R
Tab. 7. Wskaźnik procentowy tolerancji elementów Gage R&R

Source

Standard Deviation

Study

Variation

Study

Variation [%]

Tolerance [%]

Total Gage R&R

Repeatability

Reproducibility

40.3666

242.200

72.50

80.73

12.9330

77.598

23.23

25.87

38.2387

229.432

68.68

76.48

Part-To-Part

38.3511

230.107

68.88

76.70

Total Variation

55.6801

334.080

100.00

111.36

As Operator 1 was the root cause of the weak measurement system analysis results, it was decided to perform proper re-training of this operator. Additionally, analysis was repeated for the data obtained only by Operator 2. The improvement is visible on the following table – where the percentage of tolerance decreased to a level of 23.88 % (tab. 8).

Tab. 8. Values of elements included in the Gage R&R for Operator 2
Tab. 8. Wartości elementów wchodzących w skład Gage R&R dla Operatora 2

Source

Standard Deviation

Study 

Variation

Study 

Variation [%]

Tolerance [%]

Total Gage R&R

Repeatability

Reproducibility

11.9408

71.645

68.92

23.88

11.9408

71.645

68.92

23.88

0.0000

0.000

0.00

0.00

Part-To-Part

12.5526

75.316

72.45

25.11

Total Variation

17.3249

103.949

100.00

34.65

Gage R&R results for rebound, first velocity, tolerance band is 300 N, based only on the second operator.

The difference between the operators is even bigger. Additionally, Tolerance achieved values of 243.86 % where the maximum allowed value is 30 % (tab. 9). This result confirms the problem with Operator 1.

Tab. 9. Values of elements included in the Gage R&R for Operator 1
Tab. 9. Wartości elementów wchodzących w skład Gage R&R dla Operatora 1

Source

Standard Deviation

Study

Variation

Study

Variation [%]

Tolerance [%]

Total Gage R&R

Repeatability

Reproducibility

81.288

487.728

75.13

243.86

6.129

36.774

5.66

18.39

81.057

486.340

74.91

243.17

Part-To-Part

71.412

428.474

66.00

214.24

Total Variation

108.201

649.206

100.00

324.60

4. Conclusions

The classic approach for measurement system acceptance would finish with the statistical calibration of sensors used by the testing rig. Additionally, the stability of achieved velocities during test sequences would further check the acceptance for the machine. It needs to be stressed that the measurement system, besides measuring equipment, also includes operators, the test method and measured parts and their characteristic behavior. The presented statistical approach utilizes the basic concepts of automotive MSA and investigates the instability of measured units over factors in the time domain, and measurement error caused by unequally trained operators.

Acknowledgments

The authors gratefully acknowledge the financial support of the research project N N502 087838 funded by the Polish Ministry of Science (MNiI).

Bibliography

1. Czop P., Wszołek G., Jakubowski D., Czaja W., The application of a first-principle damper model for tracking the variation of eigenvalues under non-stationary road excitation, [in:] Condition Monitoring of Machinery in Non-Stationary Operations, (eds.) Fakhfakh T., Bartelmus W., Chaari F., Zimnoz R., Mohamed H., Springer, Berlin Haidelberg 2012, 565–572.

2. Król A., Czop P., Jakubowski D., Czaja W., Machoczek T., Measurement System Analysis of a Servo-Hydraulic Test Rig, Modelowanie Inżynierskie tom 12, Gliwice 2012.

3. Czop P., Sławik D., A High-Frequency Model of a Shock Absorber and Servo-Hydraulic Tester, “Mechanical Systems and Signal Processing”, 2011, Vol. 25, No. 6, 1937–1955.

4. Kai Y., Basem E., Design for Six Sigma: A Roadmap for Product Development, McGraw-Hill Professional, 2003.

5. MATLAB, The Math Works Inc., Natick 1998.

Dekompozycja oraz analiza składników błędu systemu pomiarowego z zastosowaniem metodologii Six Sigma

Streszczenie: W artykule zaprezentowano zastosowanie analizy systemu pomiarowego (MSA) oraz zalety Six Sigma w podejściu walidacji testera serwohydraulicznego. Ustalenie stopnia powtarzalności i odtwarzalności przez użycie analizy powtarzalności (repeatability – R) i odtwarzalności (reproducibility – R) zwanej Gage R&R Type II, która pokazuje prawidłową pracę systemu pomiarowego, w tym niepewność pracowników oraz narzędzi pomiarowych. Narzędzie Gage R&R Type II bazuje na analizie wariancji zwanej ANOVA. Podstawowym podejściem do wiedzy na temat badanego obiektu jest poznanie błędu samego systemu pomiarowego.

Słowa kluczowe: analiza systemu pomiarowego, MSA, analiza ANOVA, walidacja, tester serwohydrauliczny

Artur Król, MSc

He received his MSc degree in Technical Physics in 2000 from the Silesian University of Technology. He is European DFSS Coordinator and Six Sigma Certified Black Belt at Tenneco Automotive Eastern Europe Ltd. He is an expert in implementation of Six Sigma methodology in engineering environment with 5 years of experience. He is responsible to coordinate and roll out strategy for better utilization of DFSS on European level.

e-mail: krol.artur@gmail.com

Wojciech Czaja, MSc Eng.

He received his Eng. degree in Management and Production Engineering, specialization of Design and Operation and Maintenance of Machines in 2011 and MSc degree in Management and Production Engineering, specialization of Machine technology in 2012, both from the Silesian University of Technology.

e-mail: czajawojtek@gmail.com

Gabriel Kost, PhD, DSc, Eng.

He was born in 1960. In 1984 graduated at the Silesian Technical University in Gliwice, in the Faculty of Mechanical Engineering, and he got a degree of mechanical engineer in speciality of machine technology and he began work at the Institute of the Machine Building in the Faculty of Mechanical Engineering of the Silesian Technical University. In 1991 he was given a doctor‘s degree of technical sciences, and in 2005 a doctor of science degree in the scope of the robotization of technological processes. He is interested in problems of the automation and the robotization of technological processes, off-line programming and motions planning of industrial robots.

e-mail:gabriel.kost@polsl.pl

Piotr Czop, PhD

He received his MSc in 1998 and PhD in 2001, both from the Silesian University of Technology. His research interests include modeling and identification of multi-domain systems consisting of hydraulic, electrical and mechanical components.

e-mail: pczop@agh.edu.pl

Grzegorz Wszołek, PhD

He received a MSc degree in Automation and Robotics in 1999 and PhD degree in Applied Mechanics in 2002, both from the Silesian University of Technology in the Institute of Engineering Processes Automation and Integrated Manufacturing Systems Gliwice, Poland. From 2002 up to now he has held associate professor position at the Silesian University of technology and is a manager of the Laboratory of Electropneumatics and PLC Controllers as well as the Laboratory of Automation and Robotisation of Manufacturing Systems. His current research interests include designing and automation of machines and technological processes, mechatronics of robots and machines as well as computer-integrated manufacturing systems.

e-mail: grzegorz.wszolek@polsl.pl

Dawid Jakubowski, MSc

In 2003 received his MSc degree at the Opole University of Technology, in the Faculty of Electrical Engineering Automatic Control and Informatics specialization of diagnostics and control electromechanical systems. From 2004 up to 2009 he was a Design and Development Engineer in Tenneco Automotive Eastern Europe Ltd., Department of Control & Measuring Systems. From 2009 up to now he is working in GS-Hydro Ltd. He is an expert in non-welded high pressure piping solutions, NVH analysis, hydraulic test systems.

e-mail: dawid_jakubowski@02.pl