Epidemiology/Introduction - Precision: Page 2 of 3

EPI8-2: Sources Of Variability- How Does One Test For Precision?

SOURCES OF VARIABILITY - HOW DOES ONE TEST FOR PRECISION?:

Perform two or more independent measurements of the same variable and compare findings both for

different observers in inter-observer variation (inter-rater variability) which gives an estimate of validity of measurement between observers or,
a single observer measuring two sets of the same variable in intra-observer variation which gives an estimate of precision for that single observer.

The two sets of data can be analysed by examining the extent of agreement between observations (or simple concordance) which is (a + d)/T to see if the proportion of agreement is high. Disagreement can be expressed as (b + c)/T.

They can also be analysed to see how close the agreement is by using the Kappa test statistic for categorical variables (which is a type of Chi-squared statistic adjusted for chance) based on data in Table 1.

**TABLE 1: INTER-RATER AGREEMENT**
	Observer 2 screen test +	Observer 2 screen test -
Observer 1 screen test +	a	c	M₁
Observer 1 screen test -	b	d	M₀
	N₁	N₀	T

For continuous variables it is possible to obtain a Pearson's correlation coefficient or to conduct a linear regression analysis.

EXAMPLE:

Calculation of the Kappa* statistic for inter-rater agreement is performed by an Excel spreadsheet. The different terms are explained and you can use this as a calculator for your Kappa test statistics.

Guidelines for the evaluation of Kappa:

k > 0.75 means excellent reproducibility or agreement

0.4 Ł k Ł 0.75 means good agreement.

0 Ł k < 0.4 means poor agreement.

* Creative Commons Attribution-Noncommercial-Share Alike 2.5 South Africa License applies.

General Introduction to Occupational Health: Occupational Hygiene, Epidemiology & Biostatistics by Prof Jonny Myers is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 South Africa License.