Hidden Health Data: 25,190 Medicaid Quality Findings Are Now Open to the Public
New research shows how artificial intelligence can uncover thousands of public health findings buried in government reports, opening new possibilities for health equity, research, and accountability.
What if some of the most valuable public health data in the United States were already available, but almost nobody could use them?
Every year, state governments publish thousands of pages describing the quality of health care provided to people enrolled in Medicaid. These reports contain information about maternal health, mental health services, preventive care, patient experiences, and whether insurance plans meet federal standards.
The problem? The information is scattered across state websites, buried inside lengthy PDF documents, and difficult to compare.
A new 2026 study published as an accepted pre-proof in Data in Brief offers a solution.
Researchers Sanjay Basu and Seth A. Berkowitz have created Hidden Health Data, an open dataset containing 25,190 findings from Medicaid managed care oversight reports.
Even better, the researchers have made both the data and the software used to extract them openly available.
Explore and download the Hidden Health Data dataset on Zenodo
The Public Health Data We Couldn’t Easily See
Medicaid and the Children’s Health Insurance Program (CHIP) provide health coverage to tens of millions of Americans, including children, pregnant women, people with disabilities, and families with low incomes.
Ensuring that these programs provide high-quality care is an important public health responsibility.
Under federal regulations, states operating Medicaid managed care programs must arrange independent External Quality Reviews (EQRs).
These reviews examine how well health plans perform in several areas, including:
- Quality of care: Are patients receiving recommended screenings, treatment, and follow-up services?
- Quality improvement: Are health plans working to improve care, and what results are they reporting?
- Regulatory compliance: Are health plans meeting federal and state requirements?
The findings are published in annual technical reports.
But until now, researchers seeking to compare findings across states have faced a major obstacle: there was no centralized, machine-readable collection of the information contained in these reports.
In 2025, the Medicaid and CHIP Payment and Access Commission (MACPAC) highlighted the need for greater transparency and usability in External Quality Review reporting.
This is an example of what researchers call dark data: information that exists but is difficult to discover, organize, analyze, or reuse.
The data aren’t necessarily secret. They’re simply difficult to access in a useful form.
What Did the Researchers Uncover?
Basu and Berkowitz collected 49 External Quality Review reports covering 44 states and jurisdictions, with report publication years ranging from 2023 through 2026.
They used artificial intelligence to extract information from the reports and organize it into three structured datasets.
| Dataset | Record | What it Contains |
|---|---|---|
| Medicaid quality measures | 11,733 | Health plan performance on clinical and patient-experience measures |
| Performance improvement projects | 2,653 | Health plan initiatives designed to improve care |
| Compliance findings | 10,804 | Findings about adherence to federal and state requirements |
| Total | 25,190 | Structured public health oversight records |
Most extracted quality-measure records refer to measurement years between 2022 and 2025.
This creates opportunities to examine information that previously required manually searching hundreds of pages of separate reports.
For example, researchers can investigate performance on:
- Postpartum care and maternal health
- Follow-up after hospitalization for mental illness
- Blood pressure control
- Cancer screening
- Patient experiences with health care services
Importantly, each record includes information about its original source document and page number.
That means researchers can trace a finding back to the original government report rather than relying entirely on the AI-generated extraction.
A Real Example: Measuring Postpartum Care
One of the most interesting examples in the study involves postpartum care.
The researchers identified 177 quality-measure records with postpartum rates, covering 61 health plans across 17 states.
They also identified 268 improvement-project records related to prenatal, postpartum, or maternal care.
Consider AmeriHealth Caritas District of Columbia.
According to the dataset, the health plan reported a 78.38% postpartum care rate for measurement year 2024.
The corresponding performance improvement project included a baseline rate of 76.76 and a most recent rate of 78.38.
A separate compliance record showed that the plan met the reviewed standard for coordination and continuity of care.
These records allow researchers to examine different dimensions of the same health plan:
- How well is the plan performing?
- What is it doing to improve?
- Is it meeting relevant regulatory standards?
Previously, answering these questions could require searching multiple sections of a lengthy technical report.
Now, the information can be connected through structured data.
The findings do not establish that a particular improvement project caused a change in care quality. They do, however, create opportunities to study relationships between improvement activities, measured performance, and compliance.
How Artificial Intelligence Made the Data Accessible
The study also shows how AI can support public health research infrastructure.
Rather than manually reviewing every report, the researchers developed an automated system that:
- Collects public External Quality Review reports from state websites.
- Extracts text from PDFs, including scanned documents when necessary.
- Uses a large language model to identify quality measures, improvement projects, and compliance findings.
- Organizes the extracted information into consistent data fields.
- Applies automated checks to identify potentially incorrect or inconsistent records.
- Preserves links to the original source documents.
The researchers used Anthropic’s Claude Haiku 4.5 model for the structured extraction.
But using AI to extract information introduces an important question:
How accurate are the results?
The researchers conducted a human validation study comparing extracted quality-measure records with their original sources.
They found:
- 92.3% accuracy: Extracted records matched source information in the human-validated sample, including the associated plan, measure, population, and year.
- 83.1% recall: The system captured approximately 83% of quality-measure values printed on the sampled pages.
- 9.3% flagged for review: Automated quality checks identified 1,086 quality-measure records requiring additional attention.
These findings are encouraging, but they also show why human oversight remains essential.
Some values were missed, especially when reports displayed multiple years side by side. Human validation covered quality-measure records, not the improvement-project or compliance datasets.
And importantly, automated consistency checks cannot guarantee that a value was assigned to the correct health plan or year.
The system makes large-scale analysis more feasible. It does not eliminate the need for careful verification.
Explore the open-source Hidden Health Data software on GitHub
Where to Find the Hidden Health Data Repositories
One of the most valuable aspects of this project is that the researchers aren’t simply publishing an article about the data.
They’re making the underlying resources available for others to use.
1. Medicaid External Quality Review Dataset
Download the Medicaid EQR dataset from Zenodo
This is the dataset described in the study.
The release includes downloadable CSV and Parquet tables covering quality measures, improvement projects, and compliance findings.
It also provides JSON Lines records, a data dictionary, validation information, and a file called documents.csv identifying the original state reports and their web addresses.
The data are released under a CC0 public-domain dedication, allowing reuse without licensing restrictions.
2. Hidden Health Data Open-Source Software
Access the Hidden Health Data GitHub repository
The researchers have released the software used to collect, extract, organize, and check the information.
This is especially valuable for public health data scientists and government analysts interested in reproducing the process or extending it to additional documents.
The software is released under the Apache 2.0 license.
3. Other Hidden Health Data Collections
The broader Hidden Health Data project also identifies additional collections of public health documents that can be transformed into structured datasets.
These are separate from the Medicaid EQR dataset evaluated in this particular paper:
- Hospital Community Health Needs Assessments: Findings about community health needs and hospital improvement strategies.
- Maternal Mortality Review Committee Reports: Findings and recommendations from reviews of maternal deaths.
- Medicaid Section 1115 Demonstration Evaluations: Evaluation findings from Medicaid demonstration programs.
- Nursing Home Deficiency Findings: Regulatory deficiencies identified through nursing home oversight.
Together, these repositories illustrate a broader opportunity: transforming public documents into data that researchers and practitioners can discover and analyze more easily.
Why This Matters for Public Health Research
The implications extend well beyond Medicaid reporting.
Better health equity research
Medicaid serves populations that often experience barriers to care.
More accessible, high-quality data may help researchers identify differences in care across health plans, states, and services.
However, the available measures differ between states, so meaningful comparisons require careful attention to definitions, populations, and reporting periods.
Stronger quality improvement and evaluation
Public health practitioners and evaluators could use these records to examine whether health plans are pursuing improvement projects in areas where measured performance suggests a need.
As additional years are added, the dataset could support longitudinal analyses of improvement activities and performance.
More transparent government oversight
Making publicly reported findings easier to search and compare creates opportunities for policymakers, oversight organizations, journalists, and communities to ask more informed questions about Medicaid managed care.
It may also help identify compliance issues that deserve closer investigation.
Connecting quality to spending and service use
Existing Medicaid research resources, including the CMS T-MSIS Analytic Files, contain information about enrollment, health care utilization, and payments.
Those restricted research files require an application and data use agreement.
The new open EQR dataset provides different information: quality measures, improvement activities, and compliance findings.
Linking these sources at appropriate state and health-plan levels could create new research opportunities.
Researchers can also explore complementary measures through the CMS Medicaid and CHIP Child and Adult Core Sets.
What Are the Limitations?
The dataset represents meaningful progress, but it is not a complete national scorecard of Medicaid quality.
The researchers identify several limitations.
Although reports from 44 states and jurisdictions were included, only 43 produced extracted records, and just 35 had quality-measure records.
Nearly 18% of quality-measure records did not contain a usable rate.
States also use different measures and reporting formats, and most states are represented by only one reporting cycle.
The AI extraction process missed some information, particularly historical values presented in multi-year tables.
Finally, the human accuracy assessment did not cover the extracted improvement-project and compliance findings.
Researchers should therefore use the dataset as a starting point, apply the available quality flags, and confirm important findings against original reports.
The Bigger Opportunity: Making Public Health Knowledge Usable
The most important contribution of Hidden Health Data may be the approach, not any single dataset.
Public health organizations produce enormous amounts of information through technical reports, evaluations, quality assessments, and regulatory documents.
Much of that information is technically public.
But public availability is not the same thing as practical accessibility.
When information cannot be searched, compared, or connected to other data, its value for public health decision-making is limited.
Hidden Health Data demonstrates how AI-assisted extraction, automated quality checks, human validation, and open repositories can make existing information more useful.
For public health researchers and practitioners, this raises an important question:
How much valuable evidence is already sitting inside documents that nobody has the time or resources to analyze?
And what could we learn if those documents became accessible, searchable data?
Read the Research and Explore the Data
Original study: Basu, S., & Berkowitz, S. A. (2026). Hidden Health Data: an open dataset of United States Medicaid managed-care External Quality Review findings. Data in Brief. https://doi.org/10.1016/j.dib.2026.113434
Primary resources:
- Download the Medicaid EQR dataset
- Explore the open-source software and related datasets
- Learn about Medicaid External Quality Reviews from CMS
This Week in Public Health explores research, evidence, and ideas that help public health professionals understand emerging challenges and improve population health. Subscribe for accessible summaries of research and its implications for practice and policy.
Build Your Public Health Issue
Tell us a little about your work, and we’ll build a personalized issue with five things worth knowing.


