3 ways to navigate health equity data challenges

Before conducting a health equity data analysis, you’ll want to assess whether your data set gives a complete and accurate picture of the demographic groups you’re trying to compare. You can expect the category of age to be accurate. But other categories like race, ethnicity, sexual orientation, and socioeconomic status are less reliable because there’s less consistency in how and even if that demographic data is collected.

This challenge is compounded by the fact that there are few reliable data sets that can be used as benchmarks to validate the accuracy of the data you’re working with. For example, Medicare claims and enrollment data is typically relied on to provide accurate benchmarks for metrics like length of stay. But Medicare data isn’t as reliable for benchmarking demographic information.

Our experts’ experience

Before embarking on a race-stratified outcomes analysis, our experts Leena and Alex sought to first validate the reported race field in Medicare’s published enrollment tables. They compared race-stratified distributions of 65+ Medicare beneficiaries to similarly stratified estimates of the 65+ population in the U.S. census. They found the Hispanic population to be dramatically undercounted in Medicare’s enrollment data.

They found the main driver of this issue: While the census had separate questions for race and ethnicity, Medicare contains only six categories that combine race and ethnicity and does not allow multiple choices. This means that patients can be identified as, for example, either “white” or “Hispanic,” but not both. Therefore, many Hispanic Medicare patients are likely grouped under another category. This makes it difficult to accurately identify or measure a disparity for Medicare patients who are Hispanic.

How to navigate this challenge

It’s safe to assume there are some problems in your demographic data set. To move forward, investigate the sources of the data. Validate the data as much as possible to identify the specific ways in which your data is imperfect—even if it means going back to the source, form, or template with which the data was collected.

1. Understand the real-world context surrounding your demographic data and the ways that context introduces bias.

For example: How was the race field in your data set completed? How was the question phrased? Was it self-reported by the patient or filled in by someone else? At what point in the patient’s journey? Was it validated? Who is likely to be missing from or underrepresented in this data set?

Data is not objective. Data reflects the real-world environment and is influenced by how it’s collected. Accounting for that real-world context can help identify where the data may be biased. For example, if the race field was completed through a patient’s primary care physician, anyone who doesn’t have a PCP will be missed. Further, a well-resourced doctor’s office may have more capacity to sensitively gather patients’ demographic information than an understaffed clinic—and that will skew the information. And when race and ethnicity data isn’t self-reported, staff may draw conclusions based on someone’s skin tone rather than how a patient identifies. Ignoring this context and failing to acknowledge these biases can lead to skewed analyses and harmful decisions.

2. Go beyond your typical benchmark sources to validate your data set.

You cannot rely on typical foundational sources, like Medicare data, as a source of truth when it comes to certain demographic fields. You may need to use data sources you wouldn’t turn to normally. In the example above, when Alex and Leena identified issues with Medicare’s race distribution, they turned to the census as a comparison point.

You should also factor in your on-the-ground experience. For example, Natalie notes that using ZIP code as a proxy for a patient’s social determinants of health (such as income level or access to healthy food) is not always reliable because neighborhoods can vary significantly within a single ZIP code. Use your personal experience as a local resident to think critically about the details of your data set and reach a higher level of precision.

The goal is not to search for one benchmark that agrees with your data set. Rather, look at a range of data sets—including qualitative inputs—and how their demographic distributions agree or disagree to shed light on how representative your sample is.

3. Get specific about how your data set is bad or incomplete.

Once you’ve found the inevitable imperfections in your data, go beyond describing your data as “bad” or “imperfect”. This language makes the problem feel insurmountable. (After all, how can you make “good” comparisons based on “bad” data?) However, pinpointing the specific ways in which the data is imperfect—“Medicare claims data underrepresents Hispanic patients as race/ethnicity data is not accurately and discretely captured from source form through final data set”—pushes you toward action rather than frustration.

4. Improve the accuracy of your demographic data collection going forward.

We’re unlikely to see the government enforce standards for how to collect demographic information anytime soon. As a result, every provider organization, payer, and life sciences company is responsible for ensuring their data sets are as accurate and complete as possible.

Involve your data team, clinicians, and health equity experts in designing how your organization collects demographic data. Clinicians can make sure the process will integrate smoothly into workflow, health equity experts can illuminate important societal context, and data experts can minimize bias introduced during the collection process.

For more guidance on improving data collection and analysis protocols to minimize bias, use the Health equity measurement discussion guide.

Once you’ve acknowledged that your data set is likely biased in some way, it can be difficult to determine how to move forward with it—or whether to even use it at all. It’s easy to feel like imperfect data is unusable.

Our experts’ experience

Alex and Leena experienced this when they found that Medicare data significantly underrepresents Hispanic patients. They initially felt it would be impossible to make any meaningful comparisons based on race and ethnicity. To make progress despite these issues, they identified subsets of the data they felt were representative, validated those, and moved forward with a more limited analysis.

How to navigate this challenge

The need to identify and reduce inequities is too urgent to wait for perfect data. The key is to make principled trade-offs to make progress using imperfect data. Unfortunately, there’s no definitive guideline for making a go/no go decision when it comes to whether to use your data set. Every leader must carefully assess the benefits with the potential to do harm. Here are some guidelines to help you do that.

1. Focus on the goal your team or organization is trying to accomplish.

If, for example, you’re looking to quantify the largest disparity among all groups in a demographic category (for example, every racial and ethnic population), then having complete data is paramount. However, if your goal is to understand if there are disparities in just one service line, then you may be able to move forward with just a representative subset of the data.

2. Carve out subsets in the data where you are more confident.

In the example above, Leena and Alex weren’t comfortable using Medicare’s data for Hispanic beneficiaries. But they felt they could do a narrower analysis comparing Black and white patients, since their main objective was to determine whether Preventable Quality Indicators (PQIs) could be used to identify and quantify any health disparities to inform action.

Similarly, when looking at Covid-19 hospitalization rates, Natalie discovered some race data in commercial claims data sets was imputed—meaning each patient’s race was inferred based on a combination of inputs including location and last name. Natalie didn’t feel comfortable using the imputed race data after she noticed it was especially inaccurate in racially diverse neighborhoods and tended to underreport people of color. However, she decided she could move forward with Medicare Advantage claims, which gave a representative picture of hospitalizations for Black and white patients.

3. Use qualitative inputs to guide interpretations where the data ends.

When running a narrow analysis, bolster the quantitative analysis with qualitative inputs to draw reasonable conclusions outside of what the data alone can show. Leena and Alex were careful to acknowledge the limitations of the analysis they ultimately ran, including why they compared only Black and white patients. They also noted that the disparities between Black and white patients didn’t show the entire picture—and while this particular data set couldn’t prove it, we know qualitatively that Hispanic and other racial minority populations also experience disparities.

For more guidance on using qualitative feedback to surface disparities, review Take a data-driven approach to identifying and reducing disparities.

4. Avoid using demographic data to determine treatment decisions.

Natalie warns against using even narrow subsets of data in predictive modeling to determine which patients should receive which services. All health care data sets come from a society in which certain groups of people have systemically been left out; using this data to allocate care risks further magnifying disparities.

Consider the case of an algorithm created by Optum to guide treatment decisions. The algorithm was trained by looking at dollars spent. While the algorithm didn’t explicitly take race into account, the U.S. health system has historically spent less money on Black patients than on white patients. The algorithm magnified that bias in the data set it was trained on, leading it to recommend less care for Black patients than white patients. Instead of using demographic data to inform treatment, Natalie recommends using validated demographic data to evaluate patterns in outcomes and identify disparities.

(Note: Advisory Board is a subsidiary of Optum. All Advisory Board research, expert perspectives, and recommendations remain independent.)

Typically, someone performs an analysis because they have a strong hypothesis that an inequity does indeed exist. So, what happens when that hypothesis isn’t reflected in the analysis?

Our experts’ experience

Leena and Alex were faced with this situation when investigating the differences in avoidable hospitalizations for asthma between Black and white Medicare patients. They looked at AHRQ’s Prevention Quality Indicators (PQIs), a set of metrics that can identify hospitalizations that were avoidable specifically with better upstream preventive care. Using PQIs enabled Leena and Alex to specifically quantify how many Black patients experience preventable hospitalizations due to inequitable access and outcomes in upstream preventive care.

Qualitatively, the team knew that Black patients tend to have worse access to preventive care and more chronic conditions than their white counterparts. The team had validated that the race categories for those populations were accurate in the Medicare data. The team had also found a similar PQI rate in the overall population compared to AHRQ’s published benchmarks, and those held up for age and gender breakdowns as well. Yet, to their surprise, their analysis turned up essentially no difference in the PQI rate for avoidable hospitalizations due to asthma between Black and white Medicare patients.

How to navigate this challenge

Data and analytics leaders at Intermountain Healthcare shared their experience with running similar analyses: “If we did find a health disparity, it was likely to be real. If we did not find a disparity, we would have to acknowledge that a disparity may be hidden by the bias of our race and ethnicity data collection.” The three tactics detailed below can help when your data isn’t showing an obvious disparity that you believe it should be.

1. Question the metric—again.

While you can put in thoughtful effort to identify a relevant metric, it is still just a proxy for true inequities. If your analyses don’t present the results that you expect, place that within the story of your chosen metric(s) and inspect whether inequities could inflect outcomes even within that metric.

For example, Leena and Alex went back to the PQI metric methodology, which identifies “preventable” hospitalizations where the primary diagnosis code was for a low-acuity chronic condition. The team hypothesized that marginalized patients are, on average, higher acuity. Therefore, their hospitalizations may be related to several conditions, leading to PQIs “missing” encounters where untreated asthma was a factor, but the primary diagnosis was for a more acute chronic condition or comorbidity.

2. Cut the data several different ways to triple-check for disparities.

When evaluating inequities, it’s important not to stop at the first result, because disparities can show up in unexpected ways.

With their PQI analysis, while Leena and Alex found no disparity when looking at all adults age 65 and older, they did find that Black patients in the 65-74 age group had much higher rates of avoidable admissions for asthma than their white peers. This pattern continued until the white population caught up in the 85+ age group. This level of specificity opened a range of new hypotheses as to what could be causing the disparity in younger age groups but masking or ameliorating it for older patients. One hypothesis is that this may indicate a “survivor effect,” where patients with unmanaged chronic conditions die at younger ages, leading to relatively healthier patients being overrepresented in older age categories for certain demographic groups.

3. Use on-the-ground experience to gut check the results of your analysis.

It’s possible that an analysis finding zero evidence of a disparity is reflective of reality. Qualitative input from frontline clinicians and patients will typically be able to validate if that’s the case. If you do identify an area of equity, share learnings from what’s working well.