A Mess of Mixed Methods

by Jon Scaccia August 27, 2024

Evaluating programs and policies is like trying to catch a greased pig. It’s messy, unpredictable, and downright challenging. But, as it turns out, embracing this messiness can lead to some of the most enlightening insights. A recent study explored just that by delving into the murky waters of mixed methods research (MMR) to understand how research evidence is used in U.S. federal policymaking. Here’s a breakdown of their findings, why they matter, and what they mean for public health practitioners and researchers.

The Jumbled World of Mixed Methods Research

Mixed methods research (MMR) is like a recipe that combines different ingredients to create a more flavorful dish. Instead of relying solely on quantitative data (numbers and stats) or qualitative data (words and meanings), MMR blends both to provide a richer, more nuanced understanding of a research problem. However, just like any complex recipe, MMR can be messy. The process might require adjustments and recalibrations, but the end result can offer profound insights.

In the study, researchers took a reflexive approach to a mixed-methods evaluation of an intervention aimed at increasing the use of research evidence in U.S. federal policymaking. They examined both the evaluation results and the effects of mixing methods on the evaluation process itself. Their goal was to shine a light on the often-overlooked messiness of MMR and its impact on research outcomes.

The Study’s Key Findings

The Research-to-Policy Collaboration (RPC) model was the focal point of this study. The RPC model is designed to increase the use of research evidence in policymaking by building relationships between researchers and legislative offices. Instead of broadly disseminating research findings, the RPC model targets specific research needs from legislative offices and provides tailored responses.

The researchers used a combination of surveys, interviews, and qualitative coding of legislative texts to understand how research evidence is used in policymaking. This mixed methods approach allowed them to gather a wealth of data and explore the nuances of the policymaking process.

One of the key methods they employed was multinomial logistic regression, a statistical technique used to analyze data with multiple categories. This method helped them interpret the qualitative information and provided insights into how research evidence is deployed in policy texts.

Throughout the study, the researchers encountered several challenges. Mixing methods introduced complexities and required constant recalibration. However, these challenges also led to valuable insights. For instance, the team found that the qualitative coding process, guided by a codebook, was initially clear but became muddled when certain codes were difficult to interpret consistently.

By using quantitative methods to triangulate their qualitative findings, the researchers identified areas of uncertainty and confusion. This process highlighted the importance of being flexible and adaptable when conducting mixed-methods research.

Why Should We Care?

1. Mixed Methods Yield Richer Insights: Combining quantitative and qualitative data can uncover hidden patterns and relationships that might be missed when using a single method, leading to more comprehensive and actionable findings.
2. Policymaking is Complex: Understanding how research evidence influences policymaking is crucial. Policies affect every aspect of our lives, from healthcare to education to the environment. By improving how research evidence is used in policymaking, we can create better, more effective policies.
3. Embracing Messiness: The study highlights the importance of acknowledging and embracing the messiness of research. This approach can lead to unexpected discoveries and a deeper understanding of the research process itself.

Join the Conversation

What do you think about the role of research evidence in policymaking? Have you encountered challenges in mixing methods in your own research or work? Share your experiences and thoughts in the comments below or on social media.

Conclusion

The study’s findings underscore the value of mixed methods research in evaluating complex processes like policymaking. By embracing the messiness and being open to recalibration, researchers can uncover deeper insights and produce more robust findings. For public health practitioners and researchers, these insights are invaluable in shaping policies that are informed by solid evidence and tailored to meet specific needs.

As we continue to navigate the intricate world of research and policy, it’s crucial to remain flexible, adaptable, and open to the unexpected. After all, it’s in the messiness that some of the most meaningful discoveries are made.

Be Part of the Change – Get Weekly Updates! 

Stay informed and connected. Subscribe for free and share this blog to make a difference in public health with others.