Epidemiology/Critical Appraisal

Block 1: Epidemiology

EPI12-3: CRITICAL APPRAISAL

Having read the article and evaluated it in terms of the checklist, we can identify some problems. These are displayed in red.

Title - appropriate
This is not particularly clear. It suggests a simple description of surveillance over 10 years but in fact is mainly about showing the validity of ZPP vs Blood lead monitoring.

Abstract/summary - adequate and not misleading

Literature review - adequacy, appropriate and relevant research problem.
The literature review and introduction seem to be mixed with the methods. Although the US and UK action levels for BPb are given, none is given for South Africa.

The reference to Dornan is for the UK and not SA as the paragraph would lead one to believe. It is difficult to work out what refers to UK and what to SA.

There is a long section on mining and extraction methods which is not relevant.

The monitoring programme is poorly described as is the sampling of blood leads. It is not clear how may samples were take over which period. The period in question also seems to begin in 1981 which is different to what is stated in the title. The surveillance programme changed substantially in 1983.

It is unclear why the action level is 60 when 40 is used by the US and the UK and also the WHO health based limit for lead is 40.

Motivation

Problem - what is it? The research problem is never stated.
Purpose - Why is it being done? This is never stated.
Are there realistic implementation objectives?

Aim - Concisely stated. Not stated.

Objectives - attainable and precisely defined, do they relate to aim? Not stated.

Methods: There does not appear to be a methods section per se.

Definition of terms and variables adequate Poor.;
Study design clearly stated, appropriate and efficient Unclear.;
Study population adequately defined (tree diagram) Unclear.
Sampling of study group Unclear.
- sample size adequate for power and significance;
- index subjects;
- referents;
- ratio of index/referents;
- good definition of index and reference subjects;
- sample representative (covers) the scope of study or target population;
- selection biases arising from source of index and referent subjects;
- diagnostic bias;
- self selection of the ill;
- healthy worker effect;
- sick community effect;
- confounding bias likely and measures taken for this;
- sample randomly selected or some other method;
- matching present and adequate;
- response rate acceptable and attempts to compare responders and nonresponders.
Measurements Unclear.
- blindedness;
- quality control;
- precision/reliability/repeatability, any checks;
- sample size adequate;
- observers adequately trained;
- intraobserver difference;
- validity any checks made;
- misclassification considered;
- standardisation of procedures;
- recall bias considered/likely;
- calibration of equipment and laboratory quality control;
- validation against other criteria or sources;
- sensitivity;
- specificity;
- predictive value - positive;
- negative;
- interobserver differences analysed, concordance, bias, kappa;
- data reading errors checked;
- Biological/subject variation controlled.
Piloting
- pretesting of measurement instruments for appropriateness, precision and validity. Validation is referred to under medical monitoring but no data is provided for this.
Implementation/data collection
ongoing quality control for precision and validity, missing or lost data, dropout and death Unclear.
Logistics
- epidemiologic efficiency;
- cost efficiency;
Statistical methods
- appropriate for data;Not the best method.
- appropriate for study design; Design is unclear.
Resources required
Nature of the sponsors - interested parties or disinterested parties?
Ethics None of this is made clear.
- informed consent for participation of subjects;
- feedback to individuals;
- referral for treatment or follow-up;
- feedback to interested parties;
- stakeholder preview for comment;
- release of scientific information;
- release of information to general public;
- other implementation objectives;
- agreements between researchers;
- stakeholder contracts;

We are also suddenly informed at the end of the medical monitoring section that from 1983 only high risk (and presumably high exposure) workers were followed.

Results

An irrelevant anecdote about an individual worker opens the results section. Only at this point is the WHO health based limit introduced although it is not called this but rather a "recommended level". The ZPP meter was only introduced in 1988 the last year of the period under study! Findings from other studies are quoted as "statements". These have low correlation coefficients (0.7) and only explain half of the variation of blood lead. Although the measure of validity sounds high the specificity is low at 79% leaving room for lots of false negatives which is a concern in a screening programme.

Extraneous results from a group of workers in the lead industry are introduced here with no prior information in the methods, or aims and objectives. The correlation coefficient is similar to 0.7.

All of a sudden 195 paired samples (blood Pb and ZPP) are introduced without any prior explanation. How were these chosen? The correlation coefficient is much lower at 0.48, but reaches 0.7 for those whose blood lead levels are > 40. It is unclear whether these 195 have been selected as a low exposure group or not. It is known from later studies that the correlation levels are poorer when the exposures are lower. This reduces the preventive value of the ZPP in detecting blood levels that are rising dangerously from lower levels than 40.

Sometimes the cutoff for blood lead is 60 and at other times 50. It is not clear why these are used interchangably.

At no point does their medical examination seem to include neurobehavioural tests which typically pick up abnormalities at levels higher than 40.

sufficient data description; Not sufficient.
population and study group description initially and after exclusions; It is unclear which results are repeats for individuals at different times.
univariate and bivariate statistics;
computational mistakes;
arithmetical mistakes;
adequate treatment of variables;
causes;
confounders; These have not been taken into account, for example, smoking.
effect modifiers;
intermediate variables;
effects;
appropriate crude measures to estimate parameters;
point and interval estimates;
appropriate crude measures to estimate associations point and interval estimates;
adjustments made for appropriate summary measures;
confounding controlled for;
interaction investigated;
appropriate statistical testing and presentation of test results in terms of:
- point estimates of parameters
- specified interval estimates of parameters
- test statistic values
- p-values for test statistics

Discussion

The very low positive predictive value (this is only called the "predictive value" withouth specifying which one it is, means that only 2% of those who test positive will be truly abnormally high in terms of blood lead. This means it is a poor screening test for this particular population which has a low prevalence of those with blood lead levels higher than 60 - which can be seen from the range which goes from 9 to 66. This once again confirms that a group has been taken which has typically low exposure. Despite the seemingly high values for sensitivity and specificity (not so high) the test is not too useful as a screen as one would find only 2 of every hundred who screen positive to really have high blood leads!

The conclusion states that there is a low risk of excessively absorbing lead even for the WHO criteria. This is simply not true as we are not told what the figure is except in table I where it is quite high at 20% or 1 in 5 workers.

Awareness of limitations in methods and results especially biases; There seems to be no awareness of any limitations.
- likely causes of negative studies;
- small sample size;
- nondifferential misclassification;
- inappropriate comparison groups;
- likely causes of false positive studies;
- differential misclassification;
- other biases above;
Reasonableness of interpretation of own and other findings in literature.
Application of guidelines for assessing causality:
- strength of association;
- consistency;
- specificity;
- time ordering;
- biologic gradient or dose-response;
- biologic plausibility;
- experimental evidence;
- analogy;
- coherence of above evidence;
Generalizability of findings or External validity.
Speculative comment identified as such.
Implementation - appropriate or inappropriate recommendations
- clarity of implications;
- efforts made or suggested;
Basically this study is so poorly described in terms of aims, objectives and methods that any results - if one can be clear about what is being described - are pretty much useless and unamenable to interpretation.
Publication bias
- content area for negative or positive findings;
- nature of publication outlet - overseas, local;

General Introduction to Occupational Health: Occupational Hygiene, Epidemiology & Biostatistics by Prof Jonny Myers is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 South Africa License.