Sword Health has proposed extending its existing return on investment (ROI) performance guarantee methodology beyond the Thrive musculoskeletal program to its other existing programs: pelvic health and menopause support, cardiometabolic condition management, and mental health. Sword commissioned Milliman to independently review the proposed expanded ROI methodology and assess whether the underlying analytic framework can be reasonably applied to Sword’s pelvic health and menopause support, cardiometabolic condition management, and mental health programs, as well as the considerations that would apply to its use for future clinical programs.
Based on the materials Milliman reviewed at the date of this report, the underlying differences-in-differences (DiD) framework and overall ROI methodology are conceptually capable of supporting Sword's three expanded clinical program areas and are generally consistent with common industry approaches for evaluating the financial impact of healthcare interventions. However, the methodology has limitations and considerations that should be taken into account when interpreting results and applying the framework across programs. The methodology's utilization-based intervention cohort remains an important consideration when interpreting estimated program effects. As with other observational, claims-based ROI methodologies, the applicability of this framework to expanded programs is sensitive to several key methodological decisions, including:
- How the intervention cohort is defined, recognizing that treatment is based on program utilization rather than program eligibility or access;
- How concurrent participation in multiple Sword (or other similar solution) programs may affect attribution of program-specific results;
- How diagnosis-based cost attribution is applied across programs and service categories;
- How paid-to-allowed claims cost conversion is developed and applied; and
- The way savings drivers are categorized and attributed across programs.
For the three expanded clinical program areas considered in this review, Milliman believes the methodology is conceptually capable of application, subject to the program-specific considerations and limitations described in this report. As Sword considers applying the methodology to future clinical programs, the appropriateness of the methodology should be evaluated for each program based on factors including:
- Documenting how matching variables, condition definitions, and savings categories are adapted or held constant across programs;
- Reporting the extent of concurrent participation in multiple Sword programs and clearly disclosing that results for these members may reflect the combined effects of multiple interventions;
- Evaluating whether program-specific paid-to-allowed conversion factors are appropriate when actual allowed amounts are unavailable;
- Maintaining consistent reporting conventions as the framework is applied across a growing set of interventions; and
- Monitoring measurement-period extensions or cohort rollovers if statistical matching or participation thresholds are not achieved.
Milliman did not replicate claims analyses, validate statistical output, or assess clinical effectiveness for any individual program. Accordingly, this review does not provide an opinion regarding actual or expected savings for any specific client, population, or intervention, and findings should not be interpreted as guarantees of future performance. Rather, it evaluates the appropriateness, transparency, and consistency of the documented methodology as it is extended across Sword's other three clinical programs. Organizations relying on Sword's ROI methodology should independently validate results using their own data, assumptions, and analytical standards before using them for decision making.
This report was commissioned by Sword Health.