Volunteers Evaluate Quality of COVID-19 Research for Policy Makers and Providers

To help health care providers and policy makers better understand the immense amount of coronavirus research being released daily, health care volunteers are evaluating the quality of the articles and distilling the information into a manageable format.

Students at the University of Arizona College of Medicine – Phoenix, in collaboration with researchers, physicians, fellow aspiring MDs and others across the country, have created the COVID-19 Literature Surveillance Team (LST). The project began under the direction of Jasmine Rah, a third-year medical student at the University of Washington.

“The COVID-19 LST is a professional group of affiliated volunteers from across a variety of disciplines and under the guidance of physicians who have been brought together by one mission: to help increase situational awareness, while informing frontline health care providers and key decision makers,” said Erin Hartnett, a fourth-year medical student at the College of Medicine – Phoenix. “Many medical students have been sidelined by the pandemic and are looking for ways to contribute to the response.”

The team grew from two contributors, Hartnett and Rah, on April 5 to 35 contributors. Hartnett is among the 23 students at the College of Medicine – Phoenix who are involved with the COVID-19 LST. Other collaborators are associated with UA College of Medicine – Tucson, University of Washington, Western University of Health Sciences, United States Air Force and Bernhard Nocht Institute for Tropical Medicine in Hamburg, Germany.

“Medical students, physicians and scientists are taught to analyze data for quality and applicability, so it seemed prudent to use their skills during a time when many of them were unable to help the efforts in person,” Hartnett said. “We give our time to maximize the time of policy leaders, health care workers and scientists on the frontlines. By compacting and filtering the daily flow of information, our readers are able to quickly find information that is valuable to them and to instantly be able to know how reliable that information is.”

The COVID-19 LST has completed 22 reviews, 14 of which are available on their website. From April 22 to April 29, they reviewed 937 articles.

The COVID-19 LST reports are being shared across all sectors of the COVID-19 response — locally at the University of Washington and University of Arizona to other medical schools and facilities. It is being used by numerous other government agencies, such as in the U.S. Army by Col. Will Smith, MD, one of the co-founders of the project and advisor. It also has been shared among the Washington State American College of Physicians, and they have noticed a growing following in Europe and Canada, as well.

“From the feedback we’ve received, our level of coverage and level of evidence helps not only maintain situational awareness and learn up-to-date best practice guidelines, but also allows them to stay ahead of and anticipate patient questions,” Hartnett said.

The team uses the five-level Oxford Centre for Evidence-based Medicine Levels of Evidence descriptions to evaluate the quality of articles. However, they have taken it a step further by adding a sixth level for articles that do not cite any data. “We found this was necessary because many publications have suspended peer review, which means research is getting published and distributed without properly being vetted,” Hartnett said.

The range goes from Level 1, which would include highly reliable, meta-analysis reviews of published studies, to Level 6 (their addition), which would be an article without any cited data. Level 6 articles are often someone’s opinion without cited reasoning or a news article reporting on developments without a source.

Hartnett said there are a number of “predatory journals” where people pay a hefty fee and can publish anything they want regardless of evidence. “This is posing an increasing threat to the response and the scientific narrative surrounding the pandemic,” Hartnett said. “By critiquing levels of evidence, we hope to mitigate some of these effects.”

To make their work even easier to access, the group created a podcast: The COVID-19 LST Report. It presents highlights of each daily report in 10 minutes or less and can be streamed on every major podcast platform.

“The most surprising things have been the breadth of research being conducted now and the quality of the research published,” Hartnett said. “We have seen noticeable trends of improving research quality since the start of the pandemic. Even a couple weeks ago, we often saw articles that seemed to focus more on speculation or possible research directions as opposed to proven data with good quality evidence.”

• Learn more.