EPI9-6: THE QUALITY OF MEASUREMENTS: Precision Or Random Variation And Bias Or Systematic Variation

CHECKLIST FOR CRITICAL APPRAISAL OF EPIDEMIOLOGIC STUDIES:

Title - appropriate.

Abstract/summary - adequate and not misleading.

Literature review - adequacy, appropriate and relevant research problem.

Motivation

• Problem - what is it?
• Purpose - Why is it being done?
• Are there realistic implementation objectives?

Aim -Concisely stated.

Objectives - attainable and precisely defined, do they relate to aim?

Methods

• Definition of terms and variables adequate.
• Study Design clearly stated, appropriate and efficient.
• Study population adequately defined (tree diagram).
• Sampling of study group
  • sample size adequate for power and significance;
  • index subjects;
  • referents;
  • ratio of index/referents;
  • good definition of index and reference subjects;
  • sample representative (covers) the scope of study or target population;
  • selection biases arising from source of index and referent subjects;
  • diagnostic bias;
  • self selection of the ill;
  • healthy worker effect;
  • sick community effect;
  • confounding bias likely and measures taken for this;
  • sample randomly selected or some other method;
  • matching present and adequate;
  • response rate acceptable and attempts to compare responders and nonresponders;
• Measurements
  • blindedness;
  • quality control;
  • precision/reliability/repeatability, any checks;
  • sample size adequate;
  • observers adequately trained;
  • intraobserver difference;
  • validity any checks made;
  • misclassification considered;
  • standardisation of procedures;
  • recall bias considered/likely;
  • calibration of equipment & laboratory QC;
  • validation against other criteria or sources;
  • sensitivity;
  • specificity;
  • predictive value - positive;
  • negative;
  • interobserver differences analysed, concordance, bias, kappa;
  • data reading errors checked;
  • Biological/subject variation controlled.
• Piloting
  • pretesting of measurement instruments for appropriateness, precision and validity;
  • implementation/data collection;
  • ongoing quality control for precision and validity, missing or lost data, dropout and death.
• Logistics
  • epidemiologic efficiency;
  • cost efficiency.
• Statistical methods
  • appropriate for data;
  • appropriate for study design.
• Resources required
• Nature of the sponsors - interested parties or disinterested parties.
• Ethics
  • informed consent for participation of subjects;
  • feedback to individuals;
  • referral for treatment or follow-up;
  • feedback to interested parties;
  • stakeholder preview for comment;
  • release of scientific information;
  • release of information to general public;
  • other implementation objectives;
  • agreements between researchers;
  • stakeholder contracts.

Results:

• sufficient data description;
• population & study group description initially and after exclusions;
• univariate and bivariate statistics;
• computational mistakes;
• arithmetical mistakes;
• adequate treatment of variables;
• causes;
• confounders;
• effect modifiers;
• intermediate variables;
• effects;
• appropriate crude measures to estimate parameters;
• point and interval estimates;
• appropriate crude measures to estimate associations point and interval estimates;
• adjustments made for appropriate summary measures;
• confounding controlled for;
• interaction investigated;
• appropriate statistical testing and presentation of test results in terms of:
  • point estimates of parameters;
  • specified interval estimates of parameters;
  • test statistic values;
  • p-values for test statistics.

Discussion

• Awareness of limitations in methods and results especially biases:
  • likely causes of negative studies;
  • small sample size;
  • nondifferential misclassification;
  • inappropriate comparison groups;
  • likely causes of false positive studies;
  • differential misclassification;
  • other biases above.
• Reasonableness of interpretation of own and other findings in literature.
• Application of guidelines for assessing causality:
  • strength of association;
  • consistency;
  • specificity;
  • time ordering;
  • biologic gradient or dose-response;
  • biologic plausibility;
  • experimental evidence;
  • analogy;
  • coherence of above evidence.
• Generalizability of findings or External validity.
• Speculative comment identified as such.
• Implementation - appropriate or inappropriate recommendations
  • clarity of implications;
  • efforts made or suggested.>/li>
• Publication bias
  • content area for negative or positive findings;
  • nature of publication outlet - overseas, local.

CRITICAL APPRAISAL / CAUSAL ANALYSIS - SHORT CHECKLIST:

Application of rules of evidence to a study to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards, and so forth. The questions below are sequential.

1. What were the findings?
2. Are the findings internally valid?
   1. Are there non-causal explanations?
   2. Are there causal explanations?
3. Are the findings externally valid?
4. Was the study ethical?

1. What were the findings?

• What was the Population studied?
• What was the research Question?
• What were the Results?
• What was the Study type?

2. Are the findings internally valid?

1. Are there non-causal explanations?
   • Exposure/Effect: Was this determined equally for all groups?
   • Disease/Determinants: Was this determined equally for all groups?
   • Confounding.
   • Bias Selection and Information bias.
   • Analysis: Correct analyses? Type 1, Type 2 errors? Chance?

2. Are there causal explanations?

   C.A.U.S.E.R.:

   • Credible biologically; Coherence.
   • Association strength.
   • Ubiquitous/Universal.
   • Sequence; Specificity.
   • Exposure-effect gradient.
   • Reversibility.

3. Are the findings externally valid?

Do they apply to:

• Eligible population.
• Extraction (source) population.
• External (target) populations.

4. Was the study ethical?

Internal and external ethics:

• Non-maleficence;
• Autonomy;
• Beneficence;
• Justice;
• Utility.

General Introduction to Occupational Health: Occupational Hygiene, Epidemiology & Biostatistics by Prof Jonny Myers is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 South Africa License