Sensitivity analysis for model risk management
Sensitivity testing with dependence has the potential for a wide range of applications in reporting, such as for Solvency II, IFRS 17, and balance sheet valuation.
Starting in 2008, the Centers for Medicare and Medicaid Services (CMS) began the effort to transition from using Risk Adjustment Processing System (RAPS) data files to using Encounter Data System (EDS) data files as the basis for Medicare Advantage member risk scores. The encounter data submitted by Medicare Advantage organizations (MAOs) was first used for risk adjustment in the 2015 payment year (PY), where encounters with dates of service from calendar year (CY) 2014 were used as a supplemental source of diagnoses to those submitted through RAPS. CMS has committed to using EDS data as the sole basis of payment after a multiyear transition of blending risk scores that are calculated from RAPS and EDS.
The diagnosis submission process for EDS is significantly different from RAPS and represents a large shift in the responsibilities that both CMS and MAOs have with respect to proper coding, filtering, and data submission. The shift to EDS data for calculating risk scores poses substantial financial risks for MAOs that are unprepared or ill-equipped to submit and monitor the EDS submissions. Therefore, MAOs must understand the impact of these system changes on risk scores and revenue well ahead of the deadlines for submitting these files so they can ensure the data are complete and accurate, which may require multiple re-submissions prior to the final CMS deadlines.
For many MAOs, risk-adjusted revenue makes up over 80% of total Medicare Advantage revenue. MAOs may be at significant financial risk if they have declines in risk scores and the revenue associated with their risk scores. During the transition to EDS, MAOs must be knowledgeable about EDS, understand the impact of the transition on revenue, and monitor and review EDS submissions closely to ensure their revenue will not be negatively affected.
Historically, MAOs have been responsible for conducting their own diagnosis code filtering when creating RAPS files based on guidance published by CMS. Those filtered diagnosis codes are compiled by MAOs into RAPS files and submitted to CMS. Although CMS reviews the RAPS submissions for duplicates and errors, it does not verify the validity of the MAO’s filtering logic or the resulting list of diagnosis codes at the time of submission. Instead, CMS relies on Risk Adjustment Data Validation (RADV) audits to ensure that the submitted diagnosis codes are supported by patient charts. A flowchart can be found on page 14 of CMS’s 2013 National Technical Assistance Risk Adjustment 101 Participant Guide.
The shift to EDS represents a significant change in this process. Under the EDS, all unfiltered encounter data are submitted directly to CMS. CMS then applies the filtering logic to extract the valid diagnosis codes from the data. These diagnoses are then used in the risk score calculation process. An illustration of the EDS data flow is presented on pages 1 to 3 of CMS’s 2012 Regional Encounter Data Technical Assistance Participant Guide. A summary of the EDS filtering logic, as published in CMS’s December 22, 2015, memo, is provided in Appendix A of this paper.
At first glance, it may seem that a burden has been lifted from MAOs because they no longer need to filter their claims data prior to submission. In actuality, EDS creates a different burden for MAOs because they will need to verify that the data submitted through EDS are complete and accurate and that all appropriate diagnosis codes are being accepted for risk adjustment by CMS.
MAOs may find that the CMS EDS filtering excludes diagnoses that the organization’s filtering would have included in a RAPS submission. Additionally, there may be claims that are inconsistent with Medicare fee-for-service (FFS) coding standards and that those claims are excluded by the EDS filters. Without identifying and reviewing diagnoses that are rejected by the EDS filtering logic, MAOs may find they have lower risk scores and lower risk-adjusted revenue compared to RAPS.
Therefore, it is essential for MAOs to understand the EDS filtering logic, monitor and review the EDS data submissions and response files from CMS, and compare the risk scores calculated using EDS diagnoses with those calculated using RAPS and other benchmarks. This work should be done before January 31, 2017, the deadline for submission of CY 2015 diagnoses, when the EDS risk scores will be blended at 10% into the PY 2016 risk score calculation. For PY 2017, this type of analysis will be even more important because25% of the PY 2017 risk scores will be based on EDS data.
To provide an effective review of an MAO’s diagnosis code submissions, the following analytics can be undertaken:
To perform the EDS submission review, a possible first step is to create a “plan report card,” which summarizes the risk scores under accepted RAPS and EDS submissions and the risk scores based on all diagnosis sources (claims and chart review data) after applying the MAO’s specific RAPS filtering logic and the EDS filtering logic released by CMS.
Figure 1 provides an example of a potential Plan Report Card for PY 2016 EDS submission review. In this example, there is a 4.1% gap between the EDS risk scores and the risk scores after the MAO applied the EDS filtering logic to the source claims data. Also, based on the CMS return data, the EDS risk scores are four points lower than the RAPS scores. This indicates that the EDS submissions may be incomplete and that there are diagnoses in the source claims data that the CMS filtering logic has rejected.
Figure 1: Illustrative plan report card for PY 2016 EDS submission review
|Data source||Risk score based on filtered source data||Risk score based on CMS return files||Percentage difference|
|Blended score (90%/10%)||0.854||0.846||-0.9%|
If submissions to CMS contain all necessary data elements to successfully pass the filtering logic, the risk scores based on RAPS and EDS return data should match the risk scores calculated from the source claims and chart review data. In addition, if the RAPS and EDS filtering logic are the same, the RAPS and EDS risk scores should also be the same. However, there can be gaps between what is submitted and accepted by CMS and the claims and chart review data because of:
Furthermore, comparison of the RAPS and EDS risk scores will indicate whether the MAO’s revenue is being adversely affected by the move from RAPS to EDS. A focused look at the MAO’s own coding practices as they compare to Medicare FFS coding standards and EDS filter criteria can identify the coding gaps that may drive lower risk scores under EDS.
Submission gaps are the differences between the filtered source data (claims and chart review data) and the RAPS and EDS return files from CMS. A possible first step in performing this analysis is to identify members with the most hierarchical condition category (HCC) differences between the RAPS/EDS return data and the source data to determine which diagnosis codes from the source data were not included in the return data.
Figure 2 provides an example of the comparison for a specific member. In this example, the member has four diagnoses in the filtered source data, but the EDS MAO-004 report only contains one diagnosis code. The MAO must identify whether the exclusions are due to CMS filtering (edits) or the MAO’s failure to submit the codes as part of the EDS submission files. This can be done by comparing the member’s detailed claims with both the original submission and return files.
An alternative approach is to create a unique identifier to link each diagnosis code from EDS return files back to the source claims and chart review data. The unique identifier can be the combination of member ID, claim date of service, and diagnosis code.
Figure 2: Comparing source and return data
|Diagnosis Sources||Diagnosis 1||Diagnosis 2||Diagnosis 3||Diagnosis 4|
The MAO may determine that the missing diagnoses were submitted successfully and have been excluded appropriately by the EDS filtering logic. However, it may be that certain nonstandard coding practices are causing claims to be excluded when adjustments to coding practices can avoid this exclusion. Specifically, the review should focus on the bill type (inpatient or outpatient facility) and the Current Procedural Terminology (CPT)/Healthcare Common Procedure Coding System (HCPCS)—outpatient facility and professional services—included in the data because they are the primary data elements used in the EDS filters. One key step is to identify outpatient and professional services that are filtered out by EDS and are not covered under Medicare standard fee schedules. The MAO can then determine whether these claims would have been accepted had they been coded using Medicare FFS coding standards (or, alternatively, whether they should not have been covered under the MAO’s benefit plan).
For example, if physical therapy claims are coded with HCPCS S9128 to S9131 (PT/OT/ST in the home) rather than the Medicare FFS standard (e.g., 97110 – therapeutic exercises, etc.), then the diagnosis associated with these claims will not be accepted under EDS. The MAO may need to review the codes submitted by these providers in the future to ensure the proper codes are submitted for payment and whether prior claims should be covered under the MAO’s benefit plan.
The transition toward EDS as the only source for risk adjustment is well underway, but will not be fully complete until 2020.1 Outlined below are several imminent changes occurring as part of this transition.
CMS will continue to phase in the reliance on encounter data in the risk score calculation process. During these transition years, risk scores will be calculated separately using RAPS and EDS submissions, and the resulting risk scores will be blended. Figure 3 shows the current schedule and proposed weights for RAPS and EDS through 2020. This schedule was specified by CMS in the 2017 Rate Announcement and may change. It is essential for MAOs to monitor CMS communications for future updates to this transition schedule.
Figure 3: Risk score weighting by payment pear
The blended 2016 risk score results will not be available until July 2017 when the final PY 2016 risk scores are calculated. Until that time, PY 2016 revenue will reflect risk scores based on 100% RAPS data. Starting in PY 2017, the RAPS/EDS blended scores will be available as part of the midyear adjustment (expected in August 2017) that is applied to the PY 2017 risk scores.
The submission deadlines for the next several risk score developments are shown in Figure 4.2 These deadlines apply to both RAPS and EDS submissions. Diagnosis codes submitted via RAPS or EDS after the deadline will not be included in the risk score calculation for the specified payment. MAOs have three submission deadlines for each payment year: initial, midyear, and final. The “final” is the last opportunity to submit diagnosis codes for the corresponding payment year. All deadlines are subject to change at the discretion of CMS.
Figure 4: Submission deadlines
|Risk score calc (Payment year)||Data sources||Dates of service||Data submission deadline|
|2017 Initial||RAPS Only||7/1/2015 - 6/30/2016||Friday, 9/9/2016|
|2016 Final||RAPS and EDS||1/1/2015 - 12/31/2015||Tuesday, 1/31/2017|
|2017 Mid-Year||RAPS and EDS||1/1/2016 – 12/31/2016||Friday, 3/3/2017|
|2018 Initial||RAPS and EDS||7/1/2016 – 6/30/2017||Friday, 9/8/2017|
The initial 2017 payment will be the last one based solely on RAPS data, so the impact of EDS on PY 2017 revenue will first be seen in the August 2017 MMR (midyear 2017 update) for MAOs that do not monitor the MAO-004 results. All subsequent payments will include accepted diagnoses from both RAPS and EDS.
CMS is currently making several significant changes to the MAO-004 reports, which will affect both the layout and content of these files. These changes are substantial enough that they are being implemented in future versions of MAO-004 files, and CMS has announced that in the fall of 2016, it will reissue all MAO-004 reports going back to January 2014.3
Some of the changes that CMS is implementing relate to the file layout. CMS has announced that fields will be modified in the MAO-004 reports to allow for more convenient and transparent reporting of diagnosis codes that are added or deleted subsequent to the original encounter submission. These file layout changes will hopefully provide greater clarity and transparency to the EDS processing. Appendix B provides a description of MAO files. For a full description of the changes to the MAO-004 report file layouts, refer to the July 8, 2016, CMS memo, Encounter Data Software Releases.4
The content of the MAO-004 files will also be revised to address errors that CMS found with the filtered and accepted diagnosis codes in the reports sent previously to MAOs. These errors are related to CMS’s implementation of the filtering process and affect the diagnosis codes that are ultimately accepted for risk adjustment. Additionally, issues have been identified with invalid unique member IDs being used in the MAO-004 reports. It is anticipated that these member ID issues will also be resolved and valid IDs will be included in the restated files. CMS has promised additional information regarding the schedule and process for distributing the revised MAO-004 files in the near future.
Although it is anticipated that the diagnosis codes found on the current MAO-004 reports will not completely reflect the diagnosis codes that will be reflected in the restated MAO-004 reports and used in final risk adjustment, it may still be worthwhile to analyze the current MAO-004 files to check for deficiencies or discrepancies. Depending on when CMS releases the restated files, there may not be much time to perform verification checks on them and resubmit diagnosis codes if needed before the final PY 2016 diagnosis submission deadline.
The final PY 2015 risk score reconciliation released with the July 2016 monthly membership report (MMR) files was accompanied by a June 29, 2016, memo from CMS announcing that a second final 2015 risk adjustment reconciliation would occur.5 It appears that this second reconciliation was necessitated by the same issues that are causing CMS to reissue all of the MAO-004 reports. CMS will conduct the second final 2015 risk adjustment reconciliation when these errors have been corrected and revised MAO-004 files are available. This is scheduled to occur in early 2017. It is anticipated that further details regarding the exact timing of this second reconciliation as well as the exact changes made since the prior final reconciliation will be released by CMS in the coming weeks or months.
EDS will have a meaningful impact on MAOs’ risk scores and revenue as early as PY 2016. For future years, the impact will be even larger. Now is the time for MAOs to take action to ensure that the transition to EDS does not lead to an unexpected negative impact on the risk-adjusted revenue received from CMS. It is critical that MAOs perform due diligence analyses using the MAO-004 files to identify any gaps in the EDS-based risk scores and take corrective action, if needed. Early discovery of any issues can help organizations proactively adjust internal processes to make the transition to EDS-only risk adjustment as seamless as possible.
For RAPS submissions, CMS provides MAOs general guidance on diagnosis filtering, and each MAO implements its own filters to conform to that guidance. These filters generally rely on provider type filtering for inpatient and outpatient facility claims, plus additional service type filtering for outpatient facility claims, and provider specialty filtering for professional claims.
Under EDS, MAOs are required to submit detailed claims data, including all medical diagnosis coding and financial information. Various data elements on the EDS submission files are used to identify whether the encounter is professional, inpatient, or outpatient, which is then reported on MAO-004 file as “claim type”. CMS then filters the submitted claims data and extracts the diagnosis codes eligible for risk adjustment based on filtering criteria that CMS has published. CMS primarily relies on a white list of type of bill (TOB) and CPT/HCPCS codes to determine which claims records (and corresponding diagnoses) are accepted. CMS has indicated that the risk adjustment diagnosis filtering rules at a high level have not changed, and that “for a diagnosis to be eligible for risk adjustment, it must be documented in a medical record from an acceptable provider type (hospital inpatient, hospital outpatient, or professional) and the result of a face-to-face visit.”6
Outlined below are the major components of the EDS filtering logic by service category. This information is based on the final EDS diagnosis filtering logic for PY 2015, which CMS released on December 22, 2015.7 Note that the EDS filtering logic is separate and distinct from the RAPS filtering guidance.
According to CMS, the EDS filtering logic is applied to the most recent version of an accepted encounter or a chart review record. Figure 5 summarizes the medical codes that the EDS system uses to filter out the encounter data by encounter category. It is followed by a brief outline of the EDS filtering logic for each service category in each encounter category.
Figure 5: Service Categories and Filtering Basis
|Service Category||Filtering Basis|
|Institutional Inpatient||Bill Type|
|Institutional Outpatient||Bill Type and CPT/HCPCS|
Figure 6: Institutional inpatient acceptable type of bill code8
|Medicare TOB code||Description (first 2 digits)|
|41X||Religious nonmedical (inpatient)|
Figure 7: Institutional outpatient acceptable type of bill code9
|Medicare TOB code||Description (first 2 digits)|
|12X||Hospital based or inpatient (Part B only) or home health visits under Part B|
|43X||Religious nonmedical (outpatient)|
|71X||Rural health clinic|
|76X||Community mental health center (CMHC)|
|77X||Clinic FQHC federal qualified health center|
|85X||Special facility critical access hospital (CAH)|
MAOs submit encounter data and chart review records to the EDS. CMS’s system performs processing checks to determine whether each encounter claim and diagnosis code will be included in or excluded from the Risk Adjustment System (RAS) for calculating risk scores via the CMS Medicare Advantage Prescription Drug System (MARx). There are seven reports that are provided to the MAO by CMS, numbered as MAO-001 through MAO-007, which contain critical information regarding the diagnosis codes ultimately used in risk adjustment for plan payment. Figure 8 shows a summary of the suite of response files that CMS plans to release.10
Figure 8: CMS response files
|Report number||Report name|
|MAO-001||Encounter Data Duplicates|
|MAO-002||Encounter Data Processing Status|
|MAO-003||Encounter Pricing Status|
|MAO-004||Encounter Data Risk Filter|
|MAO-005||Encounter Claims Summary|
|MAO-006||Encounter Edit Disposition Summary|
|MAO-007||Encounter Claims Detail Report|
MAO-001 and MAO-002 are initial error checking (or editing) reports that are generated within 48 hours of submitting EDS files. The MAO-001 report provides information for encounters that were rejected because of duplicated services or chart reviews. This report is only generated when duplicates are detected. The MAO-002 report shows the results of EDS editing and will indicate whether an encounter record is rejected, informational, or accepted, including an error (or edit) code indicating the reason that the record was rejected.
The MAO-004 report contains the diagnosis codes that are eligible for risk adjustment after the application of the EDS filtering logic. The MAO-004 file is the way that CMS will inform MAOs which diagnoses are accepted for risk adjustment. Thus, this is a key report to review. This file can be used to do an independent calculation of the MAO’s risk scores and be used to assess the MAO’s ability to successfully submit encounter data for risk adjustment. This report is available monthly and only includes accepted diagnosis codes for each member. It will not include rejected diagnosis codes, any HCC assigned to the code by CMS’s risk models, or the member’s risk score based on current submissions.
Currently, there is not a direct way to link diagnosis codes from MAO-004 files back to the initial EDS submissions and the source claims and chart review data. MAOs need to create a unique identifier for each diagnosis code (this could be the combination of member identifier, date of service, and diagnosis code) to track whether the diagnosis code was accepted or rejected. If a unique encounter-based diagnosis code is not found on the MAO-004 file, MAOs should look for the code in the data acknowledgement reports (TA1, 999, 277CA, MAO-001, and MAO-002) to see if the diagnosis code was excluded due to initial EDS error checking and editing. If the codes are not found to be rejected due to the initial editing, MAOs should check to see if the diagnosis code would have been excluded due to EDS risk adjustment filtering logic as outlined in Appendix A and CMS’s December 22, 2015, memo. Another potential reason that a diagnosis code would be excluded is that the MAO-004 report is provided on a monthly basis and the record could potentially be included in next month’s file.
CMS has not released the layout and the files to MAOs for the remaining four reports in the suite (MAO-003 and MAO-005 through MAO-007). A useful chart outlining the intended release schedule and flow of reports can be found on page 36 of slides from CMS’s Encounter Data Industry Update Presentation from October 27, 2011.
Medicare Advantage and the Encounter Data Processing System: Be prepared
During the transition to using Encounter Data System (EDS) data files, Medicare Advantage organizations must be knowledgeable about EDS, understand the impact of the transition on revenue, and monitor and review EDS submissions closely to ensure their revenue will not be negatively affected.