Dual data extraction/data QC?#72

Don’t know if you already have this feature in progress, but it would be awesome to be able to have two people complete data collection and a third person resolve differences/perform quality control, similar to dual screening

So agreed! We have this on the backburner due to higher priority projects, but dual extraction is probably more critical than dual screening, due to rates of inaccuracies.

One point of difficulty we’ve discussed in the past - how should we handle study design (# of arms, interventions, arm size, measurement timepoints). If extracted design isn’t in alignment across reviewers, it doesn’t even make sense to QC the two extracted datasets. As such, we’ve considered a few approaches:

• Adjudicator has to do two passes over studies: one to adjudicate design, one to adjudicate metrics
• Whichever reviewer gets to the study first establishes the design, and other reviewers must follow their design (if the design is wrong, it could still be edited, at the risk of modiyfing the work of the other reviewer)

There are variations on these themes - it boils down to pushing the work onto the reviewer vs. the adjudicator. Do you have thoughts?

I think in the spirit of two truly independent data extractions, I would lean towards having the adjudicator do two passes, and not allow either reviewer to see the other’s design. That does potentially introduce extra work for the adjudicator and redoing the data extraction if the designs are both incorrect, though. Definitely a lot of factors to consider!

I also feel this is the correct approach. When we have a spec, I’ll bounce it off you!

I also agree that dual extraction is much more important than dual screening. Usually just a few study qualities to validate for inclusion, but extraction from papers is messy…I also agree with Erin about two independent, blinded reviewers with adjudication done in two passes

We added dual extraction in release 1.41.0, in case you hadn’t noticed! We have a couple of QOL improvements we know we need to make (auto-adjudication at the top of the list, reviewer-level inspector filters next), but we’d love for you to try it out! To do so, just change the extraction mode under Admin Settings.

Awesome, thanks Karl!