13 steps, 100 people, and 1 000 000 thanks
ACP J Club. 1997 Jul-Aug;127:A14. doi:10.7326/ACPJC-1997-127-1-A14
How many steps are involved in creating each page of ACP Journal Club and Evidence-Based Medicine? And how many people does it take to carry them out? If you absorbed the title of this editorial, you already will have the answers; we were astonished when we toted them up! We decided to tell you about the steps and the people, and we especially wanted to thank all of our colleagues for the roles they play in “getting the evidence straight.”
It all begins with the meticulous scrutiny of every issue of over 60 clinical journals by our research associates, Steven Carino, Ann McKibbon, Cindy Walker-Dilks, and Nancy Wilczynski. They all have extensive research training and experience in information science and clinical epidemiology and biostatistics. They are experts in research design and analysis, not clinical content (indeed, we don't want them to be distracted or influenced by an article's content or clinical conclusions!). Rather, they begin by identifying every primary article and systematic review that concerns diagnosis, prognosis, prevention or treatment, causation, quality improvement, continuing education, or the econom ics of interventions or health care programs. Then they apply the ap propriate methodologic criteria previously developed for each type of article. These criteria are listed in the Purpose and Procedure section at the front of each issue (e.g., if the article is about prevention or treatment, were the participants randomly allocated to the different interventions; and did the investigators apply outcome measures of known or probable clinical importance to ≥ 80% of the participants who entered the investigation?).
Fewer than 1 in 20 primary and review articles pass the methodologic filter (1). The findings of those that do are highly likely to be true, but are they important for clinicians and patients? We reckon that this second question is best answered by front-line clinicians. Accordingly, in step 2, each article that “passes” these methodologic criteria is reviewed by one of the editors (Brian Haynes in Hamilton, Canada, for articles destined for ACP Journal Club; David Sackett in Oxford, United Kingdom, for those referred to Evidence-Based Medicine). The editor weeds out more of them (for several reasons, including the judgment that the confidence intervals on their conclusions are too great to be clinically useful) and then refers the remainder to one or more panels of practicing clinicians. We want to recognize and thank these clinicians for the thoughtfulness and speed with which they carry out their assessments and rush them back to us. For ACP Journal Club, the clinicians who carry out this 3rd step are clinician-epidemiologists in the relevant subspecialties of internal medicine and comprise the associate editors listed on both journals' mastheads (George Browman, John Cairns, Deborah Cook, John Cunnington, Hertzel Gerstein, Gordon Guyatt, Lawrence Hart, Anne Holbrook, Roman Jaeschke, Mitchell Levine, Eva Lonn, Lorraine Macdonald, David Morgan, James Nishikawa, Wieslaw Oczkowski, Akbar Panju, Alexandra Papaioannou, Bruno Salena, Fiona Smaill, and Timothy Whelan). For Evidence-Based Medicine, they are U.K.-based general practitioners (Martin Dawes, Andrew Chivers, and Andrew Polmear); obstetrician-gynecologists (Ian MacKenzie, Inez Cooke, and Jonathan Morris); physicians (William Rosenberg and Sharon Straus); surgeons (Jack Collin and Michael Greenall); psychiatrists (Clive Adams, Laurence Mynors Wallace, and David Gill); pediatricians (Peter Sullivan and Harvey Marcovitch); and purchasers and policymakers (Ruairidh Milne, Nicholas Hicks, and Muir Gray). We also occasionally seek opinions about articles concerning anesthesia (John Sears), pain control (Henry McQuay), and tropical medicine (Timothy Peto). In their “picker” mode, these colleagues rate each article on 2 scales, one for the importance of its conclusions and the other on the likelihood that their colleagues are aware of its results and clinical implications.
In step 4, these ratings are reviewed by the editor who requested them, and articles that have been judged most important and timely proceed to the key 5th step in which a research associate prepares structured abstracts describing the articles' objectives, design, setting, patients, results, and evidence-based conclusions. Given the restrictions of space and time, this step places high demands on conciseness, speed, and precision. Clinically useful expressions of study results, such as NNTs (the number of patients who need to be treated with the experimental therapy to prevent one additional bad outcome), often are omitted from the original publications and have to be calculated from the data in the report or provided by the authors, along with their confidence intervals (accompanied by a footnote explaining their origin).
In step 6, an associate editor who is also a clinical epidemiologist in the relevant clinical discipline collaborates with the research associate in editing each abstract. With the advent of Evidence-Based Medicine, the team of associate editors was expanded to include the additional clinical areas of general and family practice (Brian Hutchison and John Sellors); obstetrics and gynecology (John Collins, Salim Daya, and Murray Enkin); pediatrics (Angus MacMillan and Barbara Schmidt); psychiatry (David Streiner and Peter Szatmari); and surgery (Bernard Langer and Robin McLeod). Once these colleagues are satisfied with the abstract, they initiate the 7th step by inviting a second clinician to add a commentary that will place the article's conclusions into clinical context and suggest how its results might be implemented in front-line practice.
The commentaries for every issue generated in this 7th step are invited from a worldwide panel of over 1000 clinical experts who have agreed to complement the external evidence presented in abstracts with their personal clinical expertise, thereby providing readers with both of these essential elements of evidence-based medicine. This is a demanding task. Given the international readership of the journals, commentators have to avoid injecting parochial views based on local or even national practice. In addition, as experts in the field, often with strong views about current controversies and unresolved issues, they must avoid criticizing articles simply because the question posed was not the one they would have preferred to ask. Moreover, in responding to our request for a personal opinion, our commentators are asked to abjure the passive voice of the scientific journal (“Treatment X can be recommended for …”) and expose their clinical expertise in the active voice of the clinical consultation (“I am applying these results in my practice by …”). Finally, views about conflicting or ancillary evidence, however strongly held, have to be backed up by references to confirming evidence.
Step 8 sees a discussion and exchange of views and suggestions, not just among the commentator, associate editor, and research associate, but also with the lead author of each of the articles selected for the issue. Questions of fact and opinion are clarified and usually resolved (if not, space is provided for a dissenting view), and the abstract and commentary are fine-tuned to maximize their ultimate usefulness to the journals' readers. The methodologic and peer review of earlier steps continues here, and an abstract that passed all previous steps may still succumb at this stage.
In the 9th step (or earlier in the case of unresolved disputes), the requesting editor reviews the “package” of abstract plus commentary and may negotiate changes in both with the commentator and team. Meanwhile, the other elements of the relevant issue (an editorial for ACP Journal Club, a pair of EBM Notes for Evidence-Based Medicine, an updated glossary for both, and other recurring items for both journals) are completed and submitted. Steps 10 to 13 comprise copy editing at the American College of Physicians by Mary Boylan, Suzanne Brownholtz-Meyers, Shannon Donovan, Jennifer Travis, and Pat Wieland; another review by the editor; and 2 cycles of typesetting (by Wendy Smith) and proofreading the issue. Electronic versions of all the elements of both journals are then sent for printing and are amalgamated for inclusion in Best Evidence.
Moving 50 or more articles per issue through all these steps requires the consummate skills in organization, diplomacy, and time management possessed by our production supervisor, Dawn Jedraszewski, and our editorial assistants, Olive Goddard and Kathryn Smiley.
To each of the 100 people involved in creating each issue, we say thanks 1 000 000!
David L. Sackett
R. Brian Haynes