4 Critical Appraisal

Can the study design answer the research question?

There are many different designs that can potentially answer most research questions, but not all designs are created equal. A graphic like Figure 4.2 is commonly used in the evidence-based medicine literature to convey this point. Meta-analyses and systematic reviews are ‘studies of studies’, and they sit atop the evidence hierarchy. They enjoy this status because they synthesize the best available evidence. No one study is the final word on a research question, so it makes sense that a meta-analysis that pools results and accounts for variable study quality could potentially provide a better answer than any one study alone.

Figure 4.2: Levels of evidence

However, the Cochrane Handbook for Systematic Reviews (2011) cautions researchers to pay attention to design features (e.g., how participants were selected) rather than labels (e.g., cohort study) because labels are broad categories. Therefore, this hierarchy is not absolute; these rankings reflect ideals. For example, RCTs can be poorly designed or poorly implemented, and the evidence from such a flawed study is not necessarily better than the evidence from a nonrandomized study just because it carries the label “randomized.” I’ll describe the logic and assumptions of common designs in Chapters 9 through 11.

How were participants selected and recruited?

⊕I focus on ‘human subjects research’ in this book, but sampling is also an important topic in research not involving people.

Sidebar: How should you refer to the humans who participate in your studies? Is it ok to call these humans “people” or “participants”? Sure. They are people. I use “participants” most of the time. Others prefer “volunteers”. Few insist on “subjects”, with the notable exception of US federal regulations governing research.

We can’t collect data from every person in our population of interest (unless we define our population very narrowly, e.g., “authors of textbooks named Eric Green”), so we have to sample a subset of people from the population. For instance, let’s say that we want to know what eligible voters in the US think of a certain presidential candidate. There were roughly 160 million people registered to vote in the 2018 US presidential election. We can’t contact all 160 million, so imagine we survey 1000 registered voters, or 0.000006% of the population. The details around who these people are and how they got into our study matter for our inferences. Can these 1,000 people represent all US voters? To answer this question for the reader, we have to describe sample selection and recruitment in the Method section.

• What made someone eligible or ineligible to participate? Who was excluded, intentionally or not?
  
• How were participants selected and invited to participate?
• Was this selection process random, or did the researchers invite participants based on availability?
• How many were invited, and how many accepted the invitation?
• Who refused the invitation to participate, and how are they different from the people who accepted?

We’ll revisit sampling and sample size in Chapters 12 and 13.

What materials and/or measures were used?

Almost every study uses some type of materials or measures. Diagnostic studies, for instance, evaluate a diagnostic test or a piece of hardware that analyzes the test samples. Environmental studies often use sophisticated instruments to take atmospheric measurements. Studies like these always provide specific details in the Method section about the materials and equipment used.

Study variables also need to be precisely defined in the Method section. For instance, hyperparasitemia describes a condition of many malaria parasites in the blood. But what constitutes “many”? The World Health Organization (WHO) defines it as “a parasite density > 4% (~200,000/µL)” (WHO 2015b). A manuscript should be precise with respect to how measurement is operationalized.

This holds for studies measuring social or psychological constructs. For instance, in a study of anxiety, a definition of the concept of “anxiety” should be provided. Is an anxiety disorder diagnosed by a psychiatrist? If so, what is the basis for this diagnosis? Or is anxiety inferred from a participant’s self-reported symptoms on a checklist or screening instrument? If so, what are the questions and how is the instrument scored?

We’ll tackle issues of measurement, including study outcomes and indicators, in Chapter 7.

How was the study conducted and how were the data collected?

The Method section should also describe what happened after participants were recruited and enrolled.

• What happened first, second, third?
• Who collected the data, and how were they trained?
• For intervention studies, the data collection procedures should describe how participants were randomized to study arms and what happened (or did not happen) in each arm.
• Were the participants, data collectors, and/or patients blind to the treatment assignment?

We’ll discuss data collection methods in Chapters 14 and 15.

How were the data analyzed?

In this section authors typically describe the approach and logic of the core analysis. In an economics paper, this might be called the empirical strategy.

• If analyzing qualitative data, what is the analysis method? Common methods include content analysis, narrative analysis, discourse analysis, and grounded theory.
• If analyzing quantitative data, is the statistical/econometric model described clearly?
• What are the assumptions of the analysis?

Was the study approved by an ethics board?

The US Federal Policy for the Protection of Human Subjects (i.e., the “Common Rule”) defines research as “a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge…” If the research involves human subjects, it must be reviewed and approved by an institutional review board (IRB) before any subjects can be enrolled. Most studies fall under IRB oversight, but some, such as retrospective studies or quality control interventions, may qualify as exempt.

Increasingly, researchers are taking the additional step of registering their study protocol prior to the study launch in a study clearinghouse like https://clinicaltrials.gov/. This registration is a requirement for drug investigations regulated by the FDA, and it’s expected by many journals. Preregistration does not ensure trustworthy results, but the practice fosters a welcome increase in research transparency. If the analysis described in an article deviates from the planned analysis, the authors are expected to provide a compelling justification. We will return to pre-registration and open science more generally in Chapter 18.

Can each finding be linked to data and procedures presented in the Method section?

Every finding in the Results section should be linked to a methodology and source of data documented in the Method section. Articles in medical journals are some of the shortest, so supplemental materials posted online may be needed to obtain a clearer sense of what the authors did and found.

Is the analysis correct?

This is a hard question to answer during critical appraisal. Most of the time (at least in public health and medicine) you do not have access to the data and analysis code, so you cannot verify that the analysis is correct. You have to base your assessment on the authors’ written description of the data and analysis.

Even if you did have access to materials to reproduce the analysis, some analyses are so complex that only people with extensive training feel qualified to question the accuracy of the results. When reviewing a study with complex analyses, it may be necessary to consult with colleagues.

Is each claim linked to a finding presented in the Results?

Each claim should be supported by results that are reported in the paper. If there is no link between a claim in the Discussion section and a finding in the Results section, the authors may be “going beyond the data.” For example, if a manuscript presents data on the efficacy of a new treatment for malaria but does not include any data on cost, then it would be inappropriate to claim that the treatment is cost-effective. Although it’s legitimate to speculate a bit in the Discussion section based on documented findings, authors should be careful to label all speculation as such—and these hypothetical forays should never be included the article’s Abstract.

Is each claim justified?

Consider each claim in relation to the results presented to evaluate whether the authors’ arrived at the correct interpretation of the data presented. Did the authors come to a reasonable conclusion, or did they make conclusions that are not supported by the analysis? For instance, if the analysis provided only weak or mixed evidence that a new program is efficacious, it would be inappropriate to recommend scaling-up the program.

Are the claims generalizable?

⊕One approach to promoting generalizability is to randomly sample participants from the population of interest. For example, Wanzira et al. (2016) analyzed data from the 2014 Uganda Malaria Indicator Survey, a large national survey, and found that women who knew that sulfadoxine/pyrimethamine is a medication used to prevent malaria during pregnancy had greater odds of taking at least two doses than women who did not have this knowledge. Because the UMIS is nationally representative, the results could apply to Ugandan women who did not participate in the study. Would the results be generalizable to women in Tanzania? An argument could be made that they would. Would the results be generalizable to women in France? No, probably not; among other things, malaria is not an issue there.

Just about every study is conducted on a narrowly constructed sample. Have you read a psychology paper recently? Chances are the sample was WEIRD: White, Educated, Industrialized, Rich, and Democratic. Most likely the participants came from the “undergraduate pool”—psychology majors willing to take part in the study for extra credit.

In global health, it’s common to read qualitative studies involving 20 patients who receive services from one rural primary healthcare facility—a handful of people from one small town in one small corner of a country of millions.

When we read these studies, we are not interested in the hyper-local story, per se (Shadish, Cook, and Campbell 2003). Instead, we want to know if the results can generalize to the broader target population. When a study is so highly localized that the results are unlikely to generalize to new people and places, we say that the study has low external validity.

I’ll tell you more about generalizability and external validity in Chapter 12.

Are the claims put in context?

A good Discussion section puts the study findings in context by telling the reader how the study adds to the existing literature.

• Do the results replicate or support other work?
• Or do the findings run contrary to other published studies? What are some ideas about why this might be?
• How does the study advance our knowledge or fill gaps in the literature?

What are the limitations?

No study is perfect, so it’s customary to include a paragraph or two outlining the shortcomings. Such limitations span all aspects of the study design and methods, from sample size to generalizability of results, to data validity and approaches to statistical analysis. Communicating shortcomings can provide a valuable resource for future researchers in terms of caveats and research directions.