Splitting up inclusion status by "qualitative synthesis only" and "qualitative/quantative" synthesis#399

It is very common in a systemtatic review to present the results of various study designs, some of which may not be suitable for quantiative synthesis (i.e., meta-analysis) but are still important to include for qualitative synthesis. As such, I think users seeking to use NK for quantitative synthesis would benefit from having the option to include studies for “qualitative synthesis only” or “qualitative/quantative synthesis”. For example, in the Screening module a user could select from a dropdown in the “Include” tab. The dropdown might include the options “Qualitative synthesis only” and “Quantitative synthesis”

Copying John’s well-put rationale from #398 :

Issues with data collected for excluded studies usually arise when data is collected prior to rigorously assessing the quality of the overall study. As a workflow, its best to review studies in detail to ensure that they meet basic study criteria. Every once in a while, a mistake may be made and a study with collected data will be later excluded (the user should definitely change inclusion status to “Excluded” in this case). This is a different situation compared to when a study met baseline study criteria, moved on to RoB, and was excluded (from quantitative synthesis) due to very high risk bias. From a methodological perspective, it’s still best to document the full RoB results regardless of whether a study was excluded on the basis of risk of bias. While exclusion based on RoB is sometimes done in the literature (and some users will want this), it is not a recommended strategy, and the researcher should know if it makes sense to include a study based on their subject matter expertise. It’s also true that in a systematic review, the high risk of bias studies can be helpful to summarize some of the weaknesses/gaps in the literature, and to identify areas which should be further investigated (this is a good goal for any systematic review). However, I could see how this might be difficult to implement for qualitative and quantitative synthesis. This is getting off topic, but it would be ideal to have “include for qualitative synthesis” and “include for quantitative synthesis”

I think that PRISMA also makes allowances for this practice– the 2020 update noted “The PRISMA 2020 items are relevant for mixed-methods systematic reviews (which include quantitative and qualitative studies), but reporting guidelines addressing the presentation and synthesis of qualitative data should also be consulted.” Mixed-methods reviews (where some studies are only included in qualitative and some are pushed forward to quantitative) are not explicitly called out in their PRISMA diagram, but the checklist itself assumes there may be multiple syntheses for each review (see #5, 13a, 20b here http://www.prisma-statement.org/documents/PRISMA_2020_checklist.pdf).

We should consider whether to add a further clarification to the end of our PRISMA to say ## included in QLS/QNS… PRISMA 2009 (http://prisma-statement.org/documents/PRISMA%202009%20flow%20diagram.pdf) actually had separate boxes for “included in qualitative synthesis” and “included in quantitative”, do we think it would be against PRISMA principles to add that back in to reflect this?

Yep, I can definitely assure that all major authorities in SR/MAs (PRISMA, PROSPERO, Cochrane, etc.) not only allow this practice, but highly recommend that researchers make such considerations in their reviews.

I was also surprised that the 2020 PRISMA chart did not include a distinction between studies included for QLS and QNS. I personally think it would have been a good idea to make this distinction in the flow charts, but I’m also hesisitant to make changes to the structure that they have laid out…I think we should probably just stick with their current format and it is up to researchers to provide this information in their SR/MAs.

I also think that it could be beneficial for researchers to present study flow charts that are a little less “busy” in the main body of their articles. While the PRISMA 2020 diagram conveys a lot of good detailed information, it’s a bit cumbersome and can be difficult to read in an 8.5”x11” document. Maybe we could include an “NK-recommended simplified diagram”? This would be presented in the same section that the official diagram is shown, and a user would have the option of a simplified version that includes the main details. I think this would be very low priority, but think it would be a cool idea

Agreed with John here– there is no rush, and we should be exceedingly careful about recommending practices in simplifying PRISMA– but I personally find new PRISMA complex and think we could also help with interactivity (e.g. switch to/export simplified view)

Yeah, I think it would be important to sell this as a “bonus” diagram which is not meant to detract of the PRISMA statement, but only to add extra clarity to the main points (NK would still highly recommend that researchers adhere to the PRISMA statement if they plan on publishing). I could help with this design if we are interested in creating this and presenting on the site

I’d love to see even your most sketched version!

No problem - I’ll put this on the list for next week and send out to you, Karl, and Jeff.

An update on this– while not a complete solution because it’s not a default or trivial for the reviewer, we have added filters to synthesis. Meaning you may filter to only studies relevant to a given portion of synthesis (e.g. using a tag).