Spotlight On: Qualitative and Mixed Methods Research
Abt’s approach to qualitative and mixed methods research aligns with our commitment to providing clients with rigorous research and evaluation services that foster equity and positive social impact. We use qualitative data and analysis to generate actionable insights about program implementation, outcomes, and strategy. With equity as a central pillar across our work, we collaborate with our clients to ensure that our methods and results reflect a diverse range of voices and lived experiences.
Qualitative work at Abt may stand alone or complement quantitative data analysis. It may focus on answering implementation questions, generating insights into how or why a particular program works (or does not), gaining the insights and perspectives of program participants, and/or guiding program design and development. Our customized approach ensures that we produce evidence-based, high-quality results that consistently meet the expectations and needs of our clients and the communities in which we work.
Methods
Abt’s staff have experience designing qualitative and mixed methods studies, along with collecting and analyzing qualitative data across a range of program areas and in domestic and international contexts. Our teams work in policy areas such as housing and homelessness, financial well-being, public health, health service delivery, governance and systems strengthening, human services, workforce, and education. We carefully select qualitative and mixed methods designs that fit the size and scope of each project . Whenever possible, we look for ways to incorporate those with lived experience into the design, data collection and analysis phases of our work, and find ways to share findings with those most impacted. For projects that are more exploratory in nature, we may begin by collecting information through a variety of channels, including program documents, interviews, and focus groups. Where there are clearly defined research questions, we may take a more structured approach, with a clear sampling frame to collect systematic data, and comprehensive and detailed approaches to analysis. Our staff bring to bear solutions to client questions that include traditional interview and focus group research, ethnography, implementation science, qualitative impact protocol, participant perspectives, literature reviews, qualitative systematic evidence review, community engaged research, and qualitative evaluation technical assistance. We work to ensure equitable approaches to our research and evaluation.
Regardless of the data collection and analysis methods, we consistently focus on producing reliable, valid, and replicable insights that support our clients’ work and Abt’s mission of improving the quality of life and economic well-being of people worldwide.
Relevant Experience
Evaluating CMS’s InCK Model for Kids with Complex Health Needs
Client: Centers for Medicare and Medicaid Services (CMS)
Can new service delivery and payment models provide better care for children covered by Medicaid and CHIP? CMS contracted Abt to evaluate their Integrated Care for Kids (InCK) payment and delivery model, which was designed to better integrate service delivery across behavioral health, physical health, and other health-related services for children with complex needs enrolled in Medicaid.
Using a mixed methods evaluation design, Abt employs both qualitative and quantitative techniques tailored to different locales to capture information about the model’s implementation from local awardees and the experiences of medical and social service providers, frontline staff, and children and caregivers impacted by the model. Qualitative methods include interviews and site visits, beneficiary Journey Mapping and PhotoVoice and other visual techniques. The team uses these findings to inform the approach to estimating impact and interpret key findings. Quasi-experimental designs assess the impacts of the InCK model on quality of care, utilization of healthcare and social services, and out-of-home placements among children within participating communities and explore how model variations affect impacts.
Health Profession Opportunity Grant (HPOG) 2.0 Program Participant Experiences
Client: Office of Planning, Research, and Evaluation (OPRE), Administration for Children and Families (ACF), US Department of Health and Human Services (HHS)
The Health Profession Opportunity Grants (HPOG) Program funded local career pathways programs to provide occupational training to Temporary Assistance for Needy Families (TANF) recipients and other adults with low incomes. The goal was to prepare them for occupations in the healthcare field that paid well and were expected to either experience labor shortages or be in high demand.
As part of the evaluation of HPOG 2.0, Abt conducted in-depth, in-person interviews with 153 Program participants across 14 local programs. These interviews strengthened the data collected through the evaluation’s impact and descriptive implementation studies, providing important insights into the participants’ lived experiences of the HPOG 2.0 programs. The interviews shed light on potential “explanatory mechanisms” for some of the impact study findings. For example, the quantitative data indicated that some participants struggle to advance from entry-level to more advanced training in the three years after the training is completed; the interviews provided important insights into the types of barriers that participants encounter and how these barriers affect participants’ training and career trajectories. Along with supplementing and enriching the impact study findings, Abt used the results from the interview analysis to craft a set of policy briefs that provide usable insights for program improvements.
Supporting the Pathways to Work Evidence Clearinghouse
Client: Office of Planning, Research, and Evaluation (OPRE)
Improving outcomes for job seekers with low incomes remains a critical task for U.S. social policy. The Pathways to Work Evidence Clearinghouse, spearheaded by OPRE, systematically reviews existing evidence on employment and training programs to provide policymakers, program providers, and researchers with a curated repository of knowledge on interventions that help to support self-sufficiency and improve employment outcomes. Abt proudly operates the second generation of the Pathways Clearinghouse.
Our work includes a major website upgrade, development of new product lines (including evidence-to-practice guides), and the curation and updating of existing product lines, which include evidence gap maps, evidence synthesis reports, and a tool to help users pinpoint the evidence that aligns to their needs. We are also engaging field-based practitioners who advise the team and ensure that the perspectives of users are embedded in our approach and represented on the leadership team.
Moving to Work: Incentivizing Landlords in the Housing Choice Voucher Program
Client: U.S. Department of Housing and Urban Development’s (HUD)
As the number of landlords participating in HUD’s Housing Choice Voucher (HCV) program declined, the Landlord Incentives Moving to Work (LI MTW) demonstration aims to test whether providing incentives to landlords impacts their participation. HUD awarded 28 agencies LI MTW authority, and HUD contracted Abt to design and conduct an evaluation on the LI MTW program.
Abt developed a mixed-methods evaluation design that includes a quasi-experimental design (QED) and rich qualitative data collection and analysis. The QED uses a large comparison group of 112 PHAs that did not receive LI MTW designation to permit the detection of impacts and analysis of subgroup impacts for public housing agencies that choose to use varied types of incentives. Abt is conducting two rounds of on-site interviews and one round of remote interviews with PHAs to document variations in the incentives and their implementation. Abt’s design also includes interviews with up to 400 landlords to learn about their awareness of the incentives, their attitudes towards the HCV program, and whether they were influenced to change their current behaviors. Finally, the study engages with residents to assist in the research design and analysis and to better understand their experiences renting using a voucher and to consider.
Abt Team Analyzes Racial Bias in Employment Processes
Client: Office of Planning, Research, and Evaluation (OPRE), Administration for Children and Families (ACF)
Racial bias can be present in a broad range of employment proceses, including how jobs are advertised, applications are screened, tasks and work hours are assigned, mentoring is offered, compensation is set, and retention and promotion decisions are made. To improve racial equity in employment, we must understand the ways biased employment processes undermine the employment and earnings outcomes of workers of color.
On behalf of the U.S. Department of Health and Human Services' Office of Planning, Research, and Evaluation (OPRE), Abt is analyzing how employment processes in low-wage labor markets contribute to racial disparities in employment.
Abt developed a literature review and collected insights from people of color who have lived experience in low-wage work and can speak to the problems and potential solutions of racial disparity. The literature review captures practices that aim to better detect and address racial bias, suggests interventions, and serves as a public-facing resource for widespread use. The Abt team used insights from the literature review and conversations with workers of color and other individuals experienced with employment processes to help identify promising strategies to combat bias at various stages.
In consultation with OPRE, the Abt team also visited four sites that are implementing/promising anti-bias strategies to understand how they work, the experiences of staff members and participants, whether they might be good prospects for more in-depth evaluation, and how they might be studied.
Abt is synthesizing the knowledge developed from those efforts to produce a report for OPRE that highlights priority questions related to understanding racial bias and evaluating strategies to combat bias that future OPRE-funded research might address. Abt is also proposing multi-method research approaches that integrate equitable evaluation methods to answer those priority questions.
Implementation Evaluation of the 2020 Teen Pregnancy Prevention Grants
Client: U.S. Department of Health and Human Services’ Office of Population Affairs (OPA)
While the overall teen birth rate in the U.S. has been declining, large disparities persist by race, ethnicity, and among the most vulnerable youth. The U.S. Department of Health and Human Services’ Office of Population Affairs (OPA) contracted with Abt Global and its partners, Decision Information Resources and Data Soapbox, to evaluate the 2020 Teen Pregnancy Prevention (TPP) program specifically the Tier 1-Optimally Changing the Map for Teen Pregnancy Prevention and Tier 2-Innovation and Impact Networks grant strategies.
The evaluation includes a cross-grantee study of how grantees are implementing each grant strategy, case studies that highlight and explain specific TPP grantees’ strategies, and recommendations for additional TPP research and evaluation options.
Through virtual and in-person site visits to 75 grantees, a pre-interview informational form, and a document review, Abt’s work aims to understand (1) how grantees implemented the TPP Tier 1 and Tier 2 grant strategies, (2) the factors that influenced implementation, (3) what challenges grantees encountered, and (4) what factors facilitated their success in developing and implementing their projects to prevent teen pregnancy and reduce rates of STIs within their selected service or priority areas. Notes from the interviews and document review were thematically coded and analyzed to extract key themes across the grantees’ experiences.
Abt is using the findings from the data analysis to develop 75 grantee profiles; two summary reports; three case study reports that highlight and explain key strategies used by the TPP projects, and the benefits and barriers experienced by each grantee as it relates to the case study topic; a brief on how grantees adapted their projects to challenges and opportunities created by the COVID-19 public health emergency; and two infographics summarizing grantees’ approaches for each funding tier. The Abt team is also working with six grantees to produce two short video panels highlighting each grantee’s work in a key area for their grant strategy.
Evaluating California’s Project Roomkey
Client: Conrad N. Hilton Foundation and the California Health Care Foundation
The Project Roomkey (PRK) Program, led by the California Department of Social Services (CDSS) and implemented by counties and Tribes across California, aimed to protect people experiencing homelessness during the COVID-19 pandemic by providing alternatives to staying on the streets or congregate shelters. Serving 62,000 people, many of whom were elderly or had chronic health conditions, PRK increased the likelihood of participants moving into permanent housing.
With funding from the California Health Care Foundation and the Conrad N. Hilton Foundation, Abt evaluated the program to understand its successes and challenges, and the experiences of PRK participants. The mixed-methods approach included a web survey of CDSS PRK grantees from 54 counties and Tribes, telephone interviews with key stakeholders in 15 communities, site visits to five communities, and analysis of Homeless Management Information System (HMIS) data from three communities. During site visits, Abt interviewed staff from homeless services providers, local government partners and healthcare entities. The study team also interviewed individuals with lived PRK experience and observed the hotels and motels used as PRK sites.
The evaluation revealed that PRK successfully saved lives and transformed emergency housing practices in California, thanks to rapid partnerships between local service providers and health systems. The program's design, implementation, and demobilization offer valuable lessons for providing emergency and interim housing for populations with complex needs. Read the final evaluation reports.
Replication of Recovery and Reunification Interventions for Families-Impact Study (R3-Impact)
Client: Office of Planning, Research, and Evaluation (OPRE)
In the last 20 years, parental substance use disorder (SUD) has become an increasingly prevalent reason for children to enter foster care. Peer recovery coaching is a promising strategy to support parents’ recovery and help keep families safe and together when possible. Abt is evaluating two programs that use peer recovery coaches who also have lived experience as a parent in the child welfare system.
The impact evaluation of the Parent Mentor Program is a multi-site, longitudinal randomized controlled trial. It will measure the effect of the program on parent well-being and family functioning using survey data, along with prevention of foster care entry and child safety using child welfare administrative data.
The impact evaluation of Sobriety Treatment and Recovery Teams (START) is a multi-site synthesis of four quasi-experimental evaluations. It will measure the effect of the program on prevention of foster care entry and child safety using child welfare administrative data.
Both impact evaluations have an accompanying implementation evaluation, which will use mixed methods, including interviews with program leadership and participants, and data on parent characteristics and services received, to assess program fidelity, implementation facilitators and barriers, replicability, and other factors that may impact program effectiveness and future scaling efforts.
Throughout the project, Abt will engage collaborators with research, practice, and lived expertise to ensure the project is informed by multiple perspectives.
Abt Leads USAID’s Integrated Health Program in the DRC
Client: United States Agency for International Development (USAID)
Despite progress in health indicators, the Democratic Republic of Congo (DRC) remains among the lowest-ranked countries in health outcomes such as maternal and neonatal mortality and vaccination coverage. Root causes include fiscal and capacity constraints, armed conflicts, gender and cultural norms, and structural barriers, including distances to health facilities and limited transportation options. As part of the USAID-funded Integrated Health Program (IHP), Abt Global is providing programmatic, technical, and operational support and capacity building to the government, health facilities, health workers, civil society organizations, and communities to reach vulnerable populations in nine target provinces. Our common goal is sustainable improvements in health systems and the capacity for, quality of, and demand for health services.
USAID IHP has undertaken several research initiatives utilizing qualitative and mixed methods approaches to enhance the effectiveness of its interventions. One such initiative was a family planning survey that assessed the impact of outreach programs on participants' behavior and perceptions. This study involved 47 in-person interviews with women of reproductive age from four provinces, revealing that 96% of the respondents were reached by the campaign, and 89% were motivated to make family planning decisions and change behaviors. The analysis identified three emerging themes: increased family planning outreach and improved perceptions of family planning, perceived behavioral changes due to outreach, and the need for program improvements by including men and providing more information on family planning side effects.
Additionally, in response to the COVID-19 pandemic, USAID IHP conducted operations research through partner Viamo to evaluate the feasibility of continuing training sessions for health staff and community workers with enhanced safety protocols.
As part of its social behavior change objective, USAID IHP facilitates community dialogues to document and develop plans for addressing public health concerns. These community reports often inform future social behavior change communication (SBCC) mini-campaigns, targeting issues such as teen pregnancy and gender-based violence, to sensitize populations and provide resources for victims. They may also inform annual budget allocations at the health zone level, effectively giving citizens a voice in priority-setting. This comprehensive approach ensures that the program's interventions are both evidence-based and responsive to community needs.
Evaluating the American Apprenticeship Initiative
Client: U.S. Department of Labor (DOL)
DOL launched the American Apprenticeship Initiative (AAI) to expand registered apprenticeships into sectors with few apprenticeships, such as healthcare and information technology (IT), and to support underrepresented populations in apprenticeships, including women and workers of color. Abt and partners conducted an evaluation of the 46 AAI grants to build evidence on the effectiveness of registered apprenticeships for both apprentices and employers.
The initiative successfully expanded apprenticeships to underrepresented populations and new industries, including healthcare, IT, finance, transportation, and logistics. Abt used a mixed methods approach across the four sub-studies of the evaluation. The implementation study reported on quantitative data from a survey of all grantees that elicited primarily closed-ended responses to help ensure comparability across grantees. Qualitative data collected through site visit interviews with staff, partners, and employers affiliated with 10 grantees provided rich information about how grantees designed and implemented their apprenticeship initiatives, and lessons learned. The outcomes study relied on a closed-ended survey of over 2,500 apprenticeship participants and employment and earnings data from the National Directory of New Hires; while the employer return on investment (ROI) study gathered primarily quantitative data through a survey of almost 70 employers. The employer engagement demonstration used an impact study design to test an approach to engaging employers to develop apprenticeship programs. In addition to quantitative data collected by the two grantees in the demonstration, implementation study site visit interviews and observations of the approach informed the findings from this study. Qualitative data collected from the implementation study also contributed to interpreting findings from the outcomes, employer ROI, and employer engagement demonstration studies. Program administrative data reported by grantees to DOL was also used to describe the characteristics of grantees and participants, and participant experiences in the program.
