Approaches in the Area of Measurement Uncertainties

Approaches in the Area of Measurement Uncertainties

Introduction • Measurements • Uncertainties • Operations on measurands……and corresponding uncertainties 62.15 Kg 62.40 Kg

Introduction (cont.) • Measurements : difference = 62.40 - 62.15 = + 0.25 Kg • ??? Can a jogging (or gym) session have such an effect (i.e. putting weight on ! )??? • Is that realistic (or even possible) ? • Effect of Uncertainties in everyday life ! • Digital readings are misleading • Sources of uncertainties to be identified • What is the real effect of making “the same (or a similar) measurement twice“…and then making calculations on the values found ? • … The world of Radio measurements ...

History • In the days before the R&TTE Directive and the CEPT policies … • Equipment had to be type-approved in the various European Countries • Manufacturers would travel across Europewith equipment from Lab to Lab … • …and have their equipment measured • Panic … (very) different results were often found… • Can pass or fail be a random variable !

An initial Methodology had been developed (before the 1990’s) • Detailed description of the methods of measurement… • Equipment measured in the same way in the various European Countries • Results to be presented in the same way • Including the characteristics of equipment to be used in the measurements … • Requirements concerning the type of equipment to be used (instrumentation) • Requirements concerning the minimal performance (of instrumentation)

Drawbacks of this old way of handling uncertainties • Lack of freedom for the industry • In terms of type of equipment to be used • Prevents innovation • Benefits from having better equipment in one subset of the test set-up … lost … • Performance in terms of uncertainty could not be part of a commercial relationship… • Restrictive in terms of competition • How to relate the required performance in terms of equipment used for the measurement into parameters to be used for : • Systems deployment (operators … ( need also to take into account dispersion in the characteristic of equipment … )) ? • Compatibility and Sharing studies (ITU, CEPT) ?

Targets for a New Approach (i.e. TR 100 028) • Allow for a continued enhancement of the performance in term of uncertainties • Allow for positive effects of improvements in terms of performance of instrumentation • New concept : “uncertainty budget”. • Decouple new editions of Standards from the evolution of Technology in the area of Instrumentation(avoids having to update standards just to keep up with progress in the area of instrumentation ! ) • Enhanced transparency • Realistic uncertainty figures support, in particular : • the optimization of system deployment • avoiding discrepancies between results obtained by various partners (e.g. under the R&TTE regime) • Supports enhancement of the uncertainty figures.

The New Approach (in accordance with TR 100 028 V 1.4.1) • Theoretical approach for the evaluation of uncertainties • Evaluation of the measurement uncertainty for each measurement • Agree on pass/fail criteria • Define maximum values for the uncertainty (e.g. in “Harmonized Standards under the R&TTE Directive”) • Prepare supporting documentation (e.g. TRs, forms…)

The New Approach (in accordance with TR 100 028 V 1.4.1)(STEP 1) • Theoretical approach for the evaluation of uncertainties • Evaluation of the measurement uncertainty for each measurement • Agree on pass/fail criteria • Define maximum values for the uncertainty (e.g. in “Harmonized Standards under the R&TTE Directive”) • Prepare supporting documentation (e.g. TRs, forms…)

Worst case ? Statistical ? • First choice : • Worst case • statistical (usage of random variables) • Criteria • Is a worst case approach possible at all ??? • Impossible if non-finite contributions • Gaussians can often be found • Is it representative and realistic ?

Mapping of the measurement set-up into : • random variables (sources of uncertainty = contributions) and • operations between these random variables • Evaluate (or define) the probabilistic properties of each of the above random variables (e.g. rectangular distribution) • Find out the mathematical properties of the measurement set-up, or intermediate steps (based upon the properties of the distributions mapped to the various uncertainty sources).

Examples of sources of uncertainties • Related to instrumentation : • Levels of signal generators • Readings (e.g. value of a frequency) • Related to the environment or to the EUT (Equipment Under Test): • test conditions (e.g. effect of temperature) • Relations between antenna polar diagrams and EUT • Related to parts of the measurement set-up : • cables and hardware (e.g. attenuators) • mismatch ...

Example of combinations The table found in TR 100 028, part 2 Annex Dsection D.3.12 provides for usual operations : • the resulting distributions • the values of the means and standard deviations ...

… and also support ,in particular, for conversions between linear terms and dBs ...

Possible simplifications • Differentiation • 1st order approximations • Usage of sub-blocs (sub-systems) • Useful in the case where specific parts of a measurement system are often used together (e.g. automated test systems) • Useful when different units are used in different parts of the measurement set-up (e.g. Volts and dBs).

The basic trick about standard deviations • The table in D.3.12 shows, in particular, that : • a number of usual operations translate into simple operations on standard deviations … • As a result, one possible approach is the following : • list all contributions and characterize the corresponding standard deviations • combine all standard deviations in accordance with the rules given in the table (which provides a combined uncertainty, given by its standard deviation) • Invoke the “Central Limit Theorem” … which provides for a Gaussian having the standard deviation found above IN ORDER TO AVOID HAVING TO • Perform the direct calculation .

Result : The distribution corresponding to the combined uncertainty (i.e. the probability of error “r”) Gaussian curves Standard deviation

Changing Confidence Levels The standard deviation corresponds to 68 % (Normal distribution), i.e. the probability of the error being within the two bounds (plus and minus sigma). Expansion factor (k) : the factor allowing to change from one confidence level to anotherk=2 is an usual value ; provides approximately 95 %in the case of Gaussian (Normal) distributions i.e. the probability of the error being within the two bounds (plus and minus two sigma).

List of the contributions (complete … not forgetting cables, mismatch, etc.) • Define units, shape and characteristics for each distribution • Find any natural assembly (i.e. definition of sub-sets) • Possible help : ETSI TRs (including spreadsheets) • … find the right uncertainty value(s) (for the right confidence level or factor k )!

Pass or fail ??? (1) • Found in the appropriate standard : • limit value for each parameter • maximum acceptable measurement uncertainty for each parameter • The measurement provides : • the measured value • the estimation of the uncertainty • for a certain confidence level (i.e. 95%) or • for a certain expansion factor k (i.e. k= 1,96 or 2,00) • The “shared risk approach” • measurement uncertainty stated together with measured value • directly compare value measured to limit (found in the standard) • measurement uncertainty has to be better than the “maximum acceptable uncertainty” (found in the standard). • (Usually) Harmonized Standards published by ETSI under the R&TTE Directive clearly state the pass/fail criteria (i.e. the shared risk approach).

Pass or fail ??? (2) • Other approaches : • Make the limit harder to pass by the amount of the uncertainty (H) • Make the limit easier to pass by the amount of the uncertainty (E) (E) (H) Uncertainty Limit Measured value Shared risk PASS PASS Target FAIL

Pass or fail ??? (3) Why prefer the “shared risk” : • Comparing directly the result of the measurement to the limit is a natural approach … was used before taking uncertainties into account • Used for a long time and widely accepted • The only approach where the interest of all parties is to improve the uncertainty of the measurements • The maximum acceptable uncertainty is given in the standards, therefore, the risk is known by all parties beforehand and can be taken into account (in the design, planning or sharing) • Has been used safely by the industry (under both • the type approval and • the R&TTE regime).

Pass or fail ??? (4) How can a manufacturer avoid the statistical risks : • Target design = 1 maximum acceptable measurement uncertainty (or more) away from the limit • Dispersion of characteristics of equipment (due to the production line) to be also taken into account • In all the approaches, there is always a possibility of failing by a fraction of a dB if the margin indicated above is not taken into account !

Pass or fail ??? (5) • (Other) Problems with approaches other than the “shared risk” • The measured value has to be or may be one (maximum acceptable) measurement uncertainty away from the stated limit. • Therefore, the actual value of that parameter may be TWO (maximum acceptable) measurement uncertainties away from the stated limit … (if the uncertainty is + 6dB , equipment may be found at + 12 dB from the limit). • The expansion factor is arbitrary (ETSI has been using 1,96 or 2) … If the value of the uncertainty is taken into account de facto, in the pass/fail criterion, that criterion contains also implicitly the effect of some arbitrary value. • Furthermore the additive combination of a finite number of finite distributions can only result in a finite distribution … and therefore not in any Gaussian shaped curve. As a result, it may occur that an expansion factor of 2 results in an estimation of the uncertainty beyond the worst case (e.g. by 15 %). Under such situation, it makes no sense to use that over - estimated value to, de facto, relax or tighten the limit. • Recent experience (TFES) has shown that the evaluation of the uncertainty may depend upon frequency and other parameters (e.g. size of the EUT) … • Possible evolution, in time, of the maximum acceptable uncertainty • Incorporating the uncertainty implicitly in the limit … humpy limits ! • (risk of problems for the manufacturers due to both effects).

More precisely ... • One rectangular distribution ... s 2 s x 2 A -A

More precisely ... • One rectangular distribution ... • The usage of an expansion factor of 1,96 or 2 makes the expanded uncertainty go beyond the worst case ! s 2 s x 2 A -A

More precisely ... • Two rectangular distributions ( A > B ) • A > 2 B A+B -A-B -A+B s 2 s x 2 A -A

More precisely ... • Three rectangular distributions ( A > B > C ) • A > 2 B • B > 2 C A+B -A-B -A+B s 2 s x 2 A -A -A-B-C = worst case It may occur that the expanded uncertainty exceeds the worst case

Conclusion ... • This example shows that approaches other than the shared risk approach (which is based upon a direct comparison between the result of a measurement and the corresponding limit) may generate wrong decisions. • (The shared risk approach is the pass/fail criterion recommended in TR 100 028.)

The New Approach (in accordance with TR 100 028 V 1.4.1)(STEP 4 and 5) • Theoretical approach for the evaluation of uncertainties • Evaluation of the measurement uncertainty for each measurement • Agree on pass/fail criteria • Include the maximum acceptable values for the uncertainty (and the limits e.g. in “Harmonized Standards under the R&TTE Directive”) • Prepare supporting documentation (e.g. TRs, forms…)

Examples and support documentation (e.g. TRs, forms…) • ETSI has published a number of harmonized standards using the statistical approach together with the shared risk approach. These Harmonized Standards include the maximum acceptable uncertainties and have been listed in the OJ of the EU, which has given them the appropriate legal status. • ETSI has developed a number of documents e.g. TR 100 028 (version 1.4.1 published end 2001) in support of the methodology. TR 100 028 includes spread sheets as well, (V 1.4.1 also includes a presentation (the .ppt file)). • ETSI has also developed forms (e.g. test report forms) • Bibliography … ...Bureau International des Poids et Mesures(“BIPM method”)

Conclusion • The usage of the probabilistic approach (as defined in TR 100 028), on a global basis is expected to : • Facilitate global roaming while simplifying the legal aspects (e.g. in “R&TTE geographical areas”) • Allow for an enhancement of the performance in terms of uncertainties, as the performance of instrumentation increases, without a need to change the standards • Increase transparency and clarity and ease both : • system deployment • Compatibility and sharing studies between services.

Conclusion (2) • Further steps : • Make it clear and public that the ETSI preferred approach is the shared risk • publish a general standard • circulate other documents (LS, etc …) • Spread out the usage of the statistical / shared risk approach on a global basis (RAST, GRSC).

Approaches in the Area of Measurement Uncertainties