|
Block 1: Epidemiology |
EPI12-1: CRITICAL APPRAISAL |
SESSION OBJECTIVE |
At the end of this session you should be able to:
- use a checklist to help you to critically evaluate scientific publications in the literature,
- critically evaluate scientific publications in the literature.
|
You will first read an article on health surveillance of employees on a South African lead mine* and learn to look at it critically.
The article is:
Health surveillance of employees an a lead mine, 1979-1989. H G van Heerden, JT Mets. South African Medical Journal 1991;79:387-390.
HOW TO EVALUATE THE ARTICLE:
Go to the following points in the article or publication and evaluate each one for adequacy:
Title - appropriate
Abstract/summary - adequate and not misleading
Literature review - adequacy, appropriate and relevant research problem
Motivation
- Problem - what is it?
- Purpose - Why is it being done?
- Are there realistic implementation objectives?
Aim - Concisely stated
Objectives - attainable and precisely defined, do they relate to aim?
Methods:
- Definition of terms and variables adequate;
- Study design clearly stated, appropriate and efficient;
- Study population adequately defined (tree diagram);
- Sampling of study group
- sample size adequate for power and significance;
- index subjects;
- referents;
- ratio of index/referents;
- good definition of index and reference subjects;
- sample representative (covers) the scope of study or target population;
- selection biases arising from source of index and referent subjects;
- diagnostic bias;
- self selection of the ill;
- healthy worker effect;
- sick community effect;
- confounding bias likely and measures taken for this;
- sample randomly selected or some other method;
- matching present and adequate;
- response rate acceptable and attempts to compare responders and nonresponders.
- Measurements
- blindedness;
- quality control;
- precision/reliability/repeatability, any checks;
- sample size adequate;
- observers adequately trained;
- intraobserver difference;
- validity any checks made;
- misclassification considered;
- standardisation of procedures;
- recall bias considered/likely;
- calibration of equipment and laboratory quality control;
- validation against other criteria or sources;
- sensitivity;
- specificity;
- predictive value - positive;
- negative;
- interobserver differences analysed, concordance, bias, kappa;
- data reading errors checked;
- Biological/subject variation controlled.
- Piloting
- pretesting of measurement instruments for appropriateness, precision and validity.
- Implementation/data collection
- ongoing quality control for precision and validity, missing or lost data, dropout and death
- Logistics
- epidemiologic efficiency;
- cost efficiency;
- Statistical methods
- appropriate for data;
- appropriate for study design;
- Resources required
- Nature of the sponsors - interested parties or disinterested parties?
- Ethics
- informed consent for participation of subjects;
- feedback to individuals;
- referral for treatment or follow-up;
- feedback to interested parties;
- stakeholder preview for comment;
- release of scientific information;
- release of information to general public;
- other implementation objectives;
- agreements between researchers;
- stakeholder contracts;
Results
- sufficient data description;
- population & study group description initially and after exclusions;
- univariate and bivariate statistics;
- computational mistakes;
- arithmetical mistakes;
- adequate treatment of variables;
- causes;
- confounders;
- effect modifiers;
- intermediate variables;
- effects;
- appropriate crude measures to estimate parameters;
- point and interval estimates;
- appropriate crude measures to estimate associations point and interval estimates;
- adjustments made for appropriate summary measures;
- confounding controlled for;
- interaction investigated;
- appropriate statistical testing and presentation of test results in terms of:
- point estimates of parameters
- specified interval estimates of parameters
- test statistic values
- p-values for test statistics
Discussion
- Awareness of limitations in methods and results especially biases
- likely causes of negative studies;
- small sample size;
- nondifferential misclassification;
- inappropriate comparison groups;
- likely causes of false positive studies;
- differential misclassification;
- other biases above;
- Reasonableness of interpretation of own and other findings in literature.
- Application of guidelines for assessing causality:
- strength of association;
- consistency;
- specificity;
- time ordering;
- biologic gradient or dose-response;
- biologic plausibility;
- experimental evidence;
- analogy;
- coherence of above evidence;
- Generalizability of findings or External validity.
- Speculative comment identified as such.
- Implementation - appropriate or inappropriate recommendations
- clarity of implications;
- efforts made or suggested;
- Publication bias
- content area for negative or positive findings;
- nature of publication outlet - overseas, local;
* This work is licensed under a Creative Commons Attribution - Noncommercial Works License.
General Introduction to
Occupational Health: Occupational Hygiene,
Epidemiology & Biostatistics by Prof Jonny
Myers is licensed under a Creative
Commons Attribution-Noncommercial-Share Alike
2.5 South Africa License.