Change in criteria for studies of diagnostic tests
ACP J Club. 1991 Nov-Dec;115:A13. doi:10.7326/ACPJC-1991-115-3-A13
Related Content in the Archives
• Letter: Criteria for the evaluation of diagnostic tests
The Purpose and Procedure section of ACP Journal Club contains the criteria that we use to select articles for abstraction. Beginning with this issue, we have changed the criteria for studies of diagnostic tests. Previously, we required that the “gold standard” for diagnosis be an objective, easily verified phenomenon (such as death) or, if the gold standard was subjectively interpreted, that there be data reported in the study indicating the reproducibility of the observation. Thus, investigators comparing a new test for carotid stenosis with the best available standard of angiography would need to demonstrate the extent to which independent (“blinded”) angiographic interpretations of the same angiograms agreed with one another.
In theory, this criterion is certainly desirable: It helps readers to determine how often disagreements between the test and the diagnostic standard are the fault of the standard rather than the test. In practice, however, few investigators assess the amount of variability in the diagnostic standard, or, if they do, the assessment lacks rigor. For example, “some” of the interpretations were checked, or two observers worked together rather than compared independent interpretations.
Applying this strict criterion has resulted in the failure of most diagnostic test studies to qualify for inclusion in ACP Journal Club and, consequently, in a very slim “Diagnosis” section. just 5% of the articles abstracted in ACP Journal Club during the first 6 months of operation achieved this category, and these select articles are only one quarter of the diagnosis articles that are considered for abstracting.
Aside from the effect of limiting coverage of diagnostic test studies, our policy discriminated against studies that were based on usual clinical practice. In usual practice, clinical decisions are often based on single interpretations of tests such as angiograms, with or without clinicians consulting with the test interpreter. Studies that compare a test with an enhanced interpretation of the diagnostic standard provide a scientifically rigorous assessment of the test's validity, but not of its clinical utility under less pristine circumstances.
For these reasons, and because ACP Journal Club's mission is to inform clinical readers about how innovative procedures perform in clinical practice, we have changed our criterion for subjectively interpreted diagnostic standards and will accept studies using a current standard of diagnosis, whether or not the investigators have reported the reproducibility of its interpretation. The first article benefitting from the change in criteria appears in this issue (1). The other criteria for studies of diagnostic tests remain the same.
R. Brian Haynes, MD, PhD
Hamilton, Ontario, Canada