What the American Evaluation Association Says About AI: Lessons for Public Health Leaders
Artificial intelligence has moved from an experimental tool to a defining force in program evaluation. Nowhere was this more visible than at the recent American Evaluation Association (AEA) conference in Kansas City, where AI-themed sessions were filled to capacity and conversations spilled into the hallways. Evaluators across fields expressed both excitement and caution as they grappled with a central question:
How can AI enhance evaluation without undermining ethics, equity, or community insight?
For public health, where evaluation guides policy, funding, and community well-being, this conversation is especially urgent.
AI Literacy Is Now Essential for Evaluation and Public Health
A clear shift emerged across the conference: evaluators are no longer asking whether AI should be used, but how to use it thoughtfully and responsibly. Presenters emphasized that AI already plays a role in everyday evaluation work, from rapid evidence scans (sound familiar?) to mixed-methods analysis, yet many evaluators have not been trained in AI literacy.
Several sessions urged educators to integrate AI instruction into public health and evaluation curricula. This includes not only understanding how AI tools function, but also learning how to critique their limitations, detect hallucinations, recognize embedded bias, and validate outputs. For the public health workforce, this message is stark: AI literacy is becoming a core competency for effective and ethical practice.
AI Is Already Transforming Evaluation Practice Across Methods
Presenters showcased an impressive array of real-world AI applications, demonstrating that the technology is far from hypothetical. In qualitative evaluation, tools such as ChatGPT, MAXQDA AI, Notebook LM, and Fireflies.ai are being utilized to summarize interviews, extract themes, and process large narrative datasets that previously required months of coding.
Yet comparative studies presented at AEA revealed that AI often misses nuance, cultural context, and positionality. These elements are critical to community-based and equity-focused public health evaluation. AI can efficiently identify patterns, but it cannot fully replace the interpretive insight that human evaluators bring to understanding lived experiences.
In quantitative evaluation, the field is experimenting with predictive analytics, early warning systems, and machine learning models that support decision-making in health and social care settings. One example highlighted at AEA was a machine-learning dashboard used to detect unreported adverse events among individuals with developmental disabilities, integrating case management and medical claims data to illuminate patterns invisible in traditional monitoring systems.
For public health agencies that routinely manage fragmented data, the implications are profound. AI could help fill gaps through simulation studies, identify risk earlier, and strengthen surveillance. However, presenters repeatedly stressed that predictive tools must be validated for accuracy, fairness, and unintended consequences before being adopted at scale.
AI Is Improving Survey Testing and Instrument Design
Another promising trend involved the use of AI-generated personas to test surveys and instruments before they are administered to real respondents. Presenters demonstrated how personas can simulate different demographic groups, anticipate misunderstandings, detect cultural mismatches, and identify sources of measurement bias early in the development process.
For public health professionals who design surveys for diverse communities or hard-to-reach populations, this technique provides a means to enhance data quality and minimize burden, particularly when resources for field testing are limited.
AI as a Thought Partner for Theory, Strategy, and Systems Work
A growing group of evaluators is experimenting with AI as a “thought partner” rather than only an analytic tool. Presentations demonstrated how AI can assist in refining program theories, testing logic models, generating alternative causal pathways, synthesizing stakeholder perspectives, and even simulating structured debates among AI agents to explore differing viewpoints.
While this approach is still exploratory, it highlights a shift toward using AI to support systems thinking and reflective practice, both of which are key components of public health evaluation. Rather than replacing human interpretation, AI becomes a collaborator that helps surface assumptions and broaden the evaluative lens.
Ethics, Equity, and Community Voice Remain Central Concerns
Across all sessions, one consistent theme emerged: evaluators are deeply concerned about the ethical and equity implications of AI adoption. Presenters highlighted risks including algorithmic bias, opaque model design, inadequate data privacy protections, environmental impacts from large-scale computation, and the possibility that overreliance on AI could eclipse the community voice that is essential to culturally responsive public health evaluation.
Several sessions introduced frameworks to help evaluators and public health organizations decide when AI tools are appropriate, how to conduct equity-oriented validation, and how to involve affected communities in the design and oversight of AI-enabled evaluations. Yet a significant gap persists: most public health agencies still lack clear AI governance structures or ethical guidelines, despite beginning to integrate AI tools into their daily operations.
Implementation Science is a Bridge Between AI and Real-World Use
Some of the most practical insights at AEA came from sessions that applied implementation science frameworks, especially the Consolidated Framework for Implementation Research (CFIR), to evaluate AI-enabled innovations. Evaluators used implementation frameworks to assess user readiness, determine workflow alignment, identify adoption barriers, and ensure equitable outcomes. This was especially relevant in case studies involving machine-learning dashboards, AI-assisted Outcome Harvesting, and predictive models within health and social care settings. For public health agencies introducing AI tools in real-world conditions, implementation science provides the rigorous lens needed to avoid harm and maximize benefits.
AI Is Reshaping How We Teach and Learn Evaluation
Another emerging theme centered on teaching and capacity building. Presenters demonstrated AI-powered chatbots for learning logic models, asynchronous assignments that compared human and AI-generated evaluation plans, and course activities that utilize AI to help students critique evidence. These innovations point toward a future in which AI serves as both a learning companion and a tool for practicing evaluative thinking. For schools of public health, which often lag behind in digital pedagogy, these approaches could dramatically expand access to high-quality, skills-based training.
The Biggest Gaps: Where Public Health Must Act Now
Despite rapid innovation, the conference revealed several pressing gaps that public health must address. Many organizations lack clear policies governing the use of AI, including procurement standards, data ethics requirements, and decision-making frameworks for evaluating the suitability of tools.
Equity remains a major challenge, as evaluators continue to raise concerns about biased models, inaccessible systems, and tools that may inadvertently reinforce existing disparities. Workforce development is another vulnerability. Although AI is increasingly embedded in evaluation workflows, most public health practitioners have limited training in validating AI outputs, assessing risk, or integrating AI findings with ground-level community knowledge.
Finally, despite the promise of AI tools, many agencies are adopting them without conducting meaningful pilot testing, implementing evaluation, or establishing mechanisms for ongoing monitoring.
AI Is Transforming Evaluation—But Public Health Must Shape Its Future
The AEA conference highlighted a simple truth: AI is not replacing evaluators, but it is reshaping what evaluation can be. For public health, the potential is extraordinary: Faster qualitative synthesis, stronger predictive insights, improved communication, enhanced accessibility, and the ability to elevate more community voices. However, these opportunities will only be realized if public health leaders establish the governance structures, ethical guardrails, workforce competencies, and community partnerships necessary to guide the use of AI.
The field of evaluation is moving quickly, and public health cannot afford to lag behind. With intentional, equity-focused adoption, AI can become a powerful ally in advancing health, justice, and community well-being.
Questions for Reflection
How will AI enhance—not replace—human judgment, community voice, or culturally grounded interpretation in our evaluation work?
What ethical and equity risks might arise from using AI in our evaluation, and what safeguards do we need to prevent harm—especially for marginalized communities?
How will we validate AI-generated insights to ensure they are accurate, trustworthy, and aligned with the real-world context of the programs we’re evaluating?
