Allowing precalculated effect sizes in the Extraction module#475
When collecting data for systematic reviews, we typically collect raw outcome data (event counts, means, etc.). However, raw outcome data may not be available in certain cases, and only precalculated effect size data may be present. If possible, it is preferable to use raw data, as it 1) makes it easier to replicate results, 2) makes it easier to understand how effect sizes were calculated and how they can be interpretted, and 3) ensures that effect size data are calculated in the exact same manner across individual studies. Yet, using raw data is often not possible in practice, because studies often report their results in a different way.
Many other point-click tools for performing systematic reviews/meta-analyses allow combining raw data and precalculated effect size data in a single meta-analysis, and could be a valuable feature for our users. Some of the more common effect size data that would be useful to collect include:
- Between-group mean differences
- Between-group standardized mean differences (usually Cohen’s d or Hedge’s g)
- risk ratios
- odds ratios
- hazard ratios
Corresponding standard errors and/or confidence intervals are typically reported alongside these and should be collected to enable synthesis.
If precalculated effect sizes are used, the user must take caution since different methods may be used to produce individual results. As such, it might be important to allow the user to add notes about how each effect size was calculated. While precalculated effect sizes from indvidual studies are often not interchangable and may not be a good idea or possible to directly combine, it is often still desirable to report these important study-level findings in a systematic review without direct quantiative synthesis (this may actually be the most common use case for collecting such data).
This is a use case we’ve considered and put off, as you can tell by absence, for the last 1.5 years :p The obstacles we face:
- the extraction interface would need to allow for entry of all nChoose2 treatment effects between n study arms, in addition to their inferentials
- precomputed treatment effects don’t fit well in most of our existing exports, which typically contain arm-level data, vs. arm-comparison-level data
- In Quantitative Synthesis, only our NMA tool would be able to leverage this data
But I totally agree, even if we pretend these are insurmountable problems, it would still be great to be able to just gather & report this data. I’ve recommended tagging for this in the past, although it can get very awkward depending on the domain (e.g. needing to handle nChoose2 comparisons, how to structure results as free text).
To help me build context & priority:
- How frequently, by % of studies and % of reviews you do, do you guess you encounter effect-only data in your primary outcome(s)?
- What other softwares allow gathering of this data? I’d be interested to see how they approach this problem.
Thanks John!
- the extraction interface would need to allow for entry of all nChoose2 treatment effects between n study arms, in addition to their inferentials
- precomputed treatment effects don’t fit well in most of our existing exports, which typically contain arm-level data, vs. arm-comparison-level data
I wonder if it would be easiest to think of the current data collection/export format and the “nChoose2” option as two separate things rather than allow for collection/export in the same format. Perhaps in Extraction there could be two different collection templates: 1) our current arm-level format, and 2) a pairwise treatment comparison format. For export, we could provide two .csv files, one formatted as arm-level data and another containing the user-specified pairwise comparisons (adding rows for each within-study comparison), Sort of clunky, but I think that may be the best option. It’s also possible to restructure the existing format to always appear as user-defined treatment comparisons when there is a relevant within-study comparison, but I think that would be less desirable.
- In Quantitative Synthesis, only our NMA tool would be able to leverage this data
That might be ok. I think it is an inevitable limitation that pre-calculated effect size data are not applicable for the current “Summary” and “Distribution” formats, but here some options to add to our visualizations:
Summary: Add another tabular format for effect sizes/confidence intervals for each study/treatment-comparison within study. This can be essentially the same format as the current “Summary” table for pooled means, proportions, etc.
Distribution: Not exactly the same idea as the other plots available in “Distribution”, but we could provide an alternative visualization of the distribution of treatment effects from study to study. A nice example, is found here (see attached). Where multiple treatments are present, you could stratify by each specific treatment comparison, including a separate header for each comparison made and sorting by size of effect
- How frequently, by % of studies and % of reviews you do, do you guess you encounter effect-only data in your primary outcome(s)?
Difficult to say % of studies that this would be required for in a single review (typically very small amount and maybe only one or two studies). In terms of % of reviews this would be useful for, I would say about 50% of reviews would benefit from collecting this data with the intention of quantitative synthesis. However, all systematic reviews could benefit from this since almost every study will provide adjusted effect sizes for a particular outcome/treatment of interest and this would be useful for at least summarizing findings at the individual study level.
- What other softwares allow gathering of this data? I’d be interested to see how they approach this problem.
RevMan5 (Cochrane’s review/meta-analysis software) and Comprehensive Meta-Analysis (CMA) have both implemented this, but there are likely others too.
Side note: There are also R packages like meta which make it easy to work with this kind of data.
