Critical appraisal of published research

Wading through the scientific literature can be a time-consuming job, and as we rush to meet copy deadlines, most of us have become adept at skimming abstracts and conclusions.

A proper critical appraisal of scientific research is important, however, to assess the validity of the results. Understanding the strengths and weaknesses of different types of scientific study will help give some perspective on research findings, as some types of study will have more weight than others.

It is also important to fully appraise the scientific hypothesis (or theory) being tested in the research. The generation of a hypothesis is central to developing a scientific idea and may occur in many different ways. However, the critical process by which hypotheses are tested is common and leads to many being dismissed and forgotten. It is, therefore, important to recognise which hypotheses are gaining credibility and are worthy of reporting. Only the good hypotheses survive and these are the ones that need to be widely communicated.

Key issues to think about when appraising a scientific study

  • Is the study published in a good quality peer-reviewed scientific journal?
  • How big is the sample, and is the sample large enough to produce statistically valid results for the type of study involved?
  • What type of study is it – for example, a case study, which looks at just one interesting case, or a randomised-controlled trial which looks at many cases?
  • Are there any confounding factors? Often confounding factors are taken into consideration, but sometimes these can not be adequately accounted for and may have a significant effect on the results. For example, vegetarians may have a lower incidence of cancer than meat eaters, but avoiding meat may not be the factor responsible; vegetarians may smoke less, exercise more, eat more of the healthy fruits, vegetables and wholegrain cereals, and have a healthier body weight, and these confounding factors could affect the results.
  • Is the sampling random? Truly random sampling is often difficult to carry out and samples may be obtained in a convenient way. This needs to be taken into account when interpreting the results.
  • Could the results have occurred by chance alone? The answer to this may require expert input from a statistician.

Types of research study

There are two key types of research study – observational and experimental.

Observational studies

Observational studies involve observing a particular process, for example much epidemiological research is observational in nature, and data are obtained in measured groups (e.g. ecologic studies) or in individuals (e.g. case-control, cross-sectional or cohort studies).

Ecologic study

Ecologic studies focus on the comparison of groups, rather than individuals. Such studies are easy to perform using routinely collected data and can provide new hypotheses about causes of a disease or condition. A key weakness of ecologic studies is the potential for confounding factors to influence the results. Also, it should be remembered that observed associations in a group may not be applicable at an individual level.

Case study

A case study is a description of one interesting and unusual case. Limited conclusions can be drawn from case studies.

Cross-sectional study

A cross-sectional study is a survey of a defined population at one particular time. This type of study can investigate large sample sizes and is useful for estimating links between key risk factors and specific diseases. Cross sectional studies only give a snapshot of information at one time so don’t give information about cause and effect relationships and are not appropriate for measuring changes over time.

Cohort study

In a cohort study, information is gathered in a large group of individuals (a cohort) with a specified characteristic and the population is followed over a period of time. Often, large numbers of people need to be followed for significant periods of time in order to obtain valid results. This difficulty can be further increased where there is a long induction period between exposure to a hazard and the manifestation of a disease, as might occur with cancer for example. Studies of a long duration can lead to a high proportion of ‘drop-outs’, which may have an effect on the results.

Case-control study

A case-control study is an observational study where the characteristics of a group of individuals with a particular disease (cases) are compared with a group of individuals free of the disease (controls). A case-control study looks back over time to see if characteristics of the cases vary from the controls. Such studies often rely on personal recall, and the validity of the data may rely on the accuracy of recall. For example, people may remember quite well changes in geographical location or occupation; however, long-term recall of dietary habits may be less reliable. Case-control studies can provide valuable information but results should be additionally confirmed by further, more robust scientific evidence.

Experimental studies

Experimental studies usually involve the collection of data and the manipulation of a variable that is assumed to affect the outcome of the process being studied. For example, a study may test the response of a new drug or dietary intervention in humans. If well-designed and conducted, experimental studies provide the most compelling evidence of cause and effect. Types of experimental studies include randomised-controlled clinical trials and basic research experiments (e.g. in-vitro and laboratory animal studies).

Randomised-controlled trials (RCTs)

Randomised-controlled trials are considered the ‘gold standard’ in experimental studies and are the most rigorous way of determining whether a cause-effect relation exists between a treatment and the outcome. A key feature of a RCT is the random allocation of subjects to an intervention group. Double blinding (where the subjects and the researchers are unaware of who is receiving which treatment) ensures that any preconceived views can not bias the outcome. In long-term trials there may be problems with compliance, which could affect the results.

Basic research experiment

Basic research studies may include in-vitro (test tube) studies or experimental animal investigations. Findings from these types of study can be useful, but can not be extrapolated to human beings.

Fundamental Research

It is important to distinguish between different types of research study: blue sky, fundamental or investigator-led research may be different to some of the research described above. For example, the fundamental discovery of a potential new anti-cancer drug is not the same as a clinical trial investigating its efficacy. If early data on a fundamental discovery were reported it could be misleading, and it would be inappropriate to suggest a new ‘holy-grail’ cure for cancer when in reality, at best, all that has occurred is a small step in our understanding of cancer and its treatment.

Meta-analysis

A meta-analysis is a statistical technique that combines the results from separate but similar completed studies to yield overall conclusions about a particular hypothesis. Meta-analysis can be useful in terms of aggregating results, but results are only as good as the studies on which they are based.

Summary

A single study can rarely be used as justification for the scientific confirmation of a particular hypothesis. Findings of multiple studies that are both observational and experimental taken together usually contribute to our understanding of a particular issue. It is particularly important that fundamental research, which may be in its early stages, is not taken out of context or exaggerated.

Most scientific research is published in scientific journals intended for a scientific audience. It is strongly recommended that before reporting on any new scientific research the authors of the paper, or other appropriate scientists, are contacted for comment to ensure findings are accurately reported. The Science Media Centre has an extensive database of scientific experts in a range of subject areas who are available to talk to the media.

Contact us on tel: 04 499 5466 or email: smc@sciencemediacentre.co.nz to speak to an expert on any scientific issue.

Further information

Coggon D., Rose G., Barker D.J.P. Epidemiology for the uninitiated. BMJ Publishing Group, 1997.

Fowkes FG, Fulton PM. Critical appraisal of published research: introductory guidelines. BMJ.,1991; 302(6785): 1136-40.

Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0 [updated February 2008]. The Cochrane Collaboration, 2008.

Kirwan M, www.healthknowledge.org.uk

Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998;317:1185-1190.

Morgenstern, H. Ecologic Studies in Epidemiology: Concepts, Principles, and Methods. Annual Review of Public Health, 1995,16: 61-81.

Sibbald, B, Roland, M. Understanding controlled trials: Why are randomised controlled trials important? BMJ,1998; 316: 201

Wakefield, J. Ecologic Studies Revisited. UW Biostatistics Working Paper Series. May 2, 2007, Working Paper 308.

This Science Byte was peer-reviewed by Professor Ian Shaw, Pro-Vice Chancellor, College of Science, University of Canterbury.