Spotlight On: Digital and Data Services for Research, Monitoring and Evaluation

At Abt Global, technology is an integral part of all aspects of our Research, Monitoring and Evaluation work. We use an array of digital tools to collect data including direct surveys; deploying data systems directly into programs we are evaluating; and processing administrative data, public use datasets, and social media and web scraping. Our technology solutions are the backbone of programs that we are evaluating, such as the data systems used by social service providers in a pilot program. We also implement and evaluate technology solutions themselves (our “digital interventions”). For example, we study the comparative effectiveness of online learning, mobile applications, AI-based detection, or televisits relative to traditional approaches. Finally, technology is at the foundation of Abt’s robust data analysis and visualization capabilities.

In addition to designing and deploying digital and data services in research, our team is well versed in a range of adjacent fields, including: market research, general survey research, health research, digital health, human services management, digital citizenship, grants management, and technical assistance.

Expertise

Abt’s experts understand how to use technology to support all aspects of policy research, monitoring, and program evaluation. Our approaches includes the following:

Facilitating and enforcing research designs through technology, including built-in rolling random assignment and service tracking modules that prevent cross-contamination.
Integrated solutions for all phases of research, from outreach, eligibility determination, consent, baseline and follow-up surveys, service tracking, intervention modes and “dosages,” withdrawals, monitoring, and data analysis and dissemination.
Digital support for ethics, equity, scientific integrity, security, and compliance, including informed consent; built-in roles that distinguish between service and research users; managing withdrawals and post-withdrawal data-processing; enforcing OMB requirements for data collection form changes; FISMA, HIPAA, and FERPA compliance; accessibility/Section 508 Compliance; preventing algorithmic and analytic bias; and digital equity.
Distinct technology approaches based on the research need, this includes knowing when to build or buy a new system or leverage existing ones and when to use a centralized system or a distributed approach.
Implementing interventions and demonstrations technologically. This includes cases where there is a digital component to the program being evaluated, such as online learning or mobile health. In other cases, digital work is needed to implement a program such as interfacing with a government system to adjust benefit amounts.
Data management and governance, including version control for data and tools, guaranteeing reproducibility; data dictionaries and modern data catalogs; coding of qualitative case notes; data quality review libraries; and integrating and de-conflicting survey data, administrative data, physical specimens, and case management data.

Promoting Opportuniy Demonstration (POD) and Benefit Offset National Demonstration (BOND)
Client: Social Security Administration

Under current law, Social Security Disability Insurance (SSDI) beneficiaries lose their entire monthly benefit if their earnings and work activity exceed Substantial Gainful Activity (SGA) after completing the Trial Work Period (TWP) and three-month grace period. The Benefit Offset National Demonstration (BOND) and the Promoting Opportunity Demonstration (POD) tested changes to the SSDI earnings rules that reduced earnings gradually as earnings increased. The demonstrations tested an SSDI benefit offset that reduced participants’ benefits by $1 for every $2 in earnings. For BOND, SSA applied the benefit offset to earnings that exceed SGA threshold. For POD, SSA used a lower earnings threshold, the monthly TWP amount. Abt Global served as the prime contractor for both the implementation and evaluation of BOND and the implementation contractor for POD.

For BOND, Abt developed the BOND Operations Data System (BODS). SSA transferred 1.2 million beneficiary records to BODS. Abt used BODS to randomize beneficiary records to treatment, control, and recruitment pool groups. Abt conducted outreach to the recruitment pool in groups, using uniform procedures across all groups and BOND sites. Applying uniform recruitment procedures ensured equal access to the demonstration for all potential volunteers. BODS also supported informed consent, eligibility tracking, enrollment, survey data collection, second stage random assignment, and withdrawals.

For both BOND and POD, the data systems Abt developed facilitated service delivery and data collection for demonstration participants after enrollment. BODS and the equivalent system for POD—the POD Data System (PODDS)—are comprehensive management information systems that support collection, storage, reporting, and transmission of participant-level and program information. The systems document work incentives counseling and centralized communications with participants in the treatment groups and monthly earnings and reporting impairment-related work expenses.

Central to both systems is a highly customized version of a commercial off-the-shelf human services and case management product called ClientTrack. This tool tracks all of the implementation team and work incentive counselors’ interactions with participants and it integrates earnings reporting processes.

PODDS also implements a portal for participants to enter earnings and Impairment-Related Work Expenses information. Both BODS and PODS interface with SSA’s internal systems that calculate benefit amounts.

BODS is hosted in a secure data center. PODDS is hosted in the cloud on Amazon Web Services (AWS). Both received a FISMA Moderate authorization to operate.

Research on the Epidemiology of SARS-CoV-2 in Essential Response (RECOVER)
Client: Centers for Disease Control and Prevention

In the RECOVER Study, Abt leads a network of clinical partners that offers CDC timely information on symptomatic and asymptomatic SARS-CoV-2 infections, vaccine effectiveness, and knowledge, attitudes, and practices among essential workers.

Abt uses its Data Collection and Analytic Computing Environment (DC-ACE) to drive study recruitment. We pushed data from potential recruits to study site staff for real-time follow up using application programming interfaces (APIs). For data collection, the study uses robust text message-based illness tracking through REDCap and Twilio. To process the data, Abt implemented its Data Implementation and Management Ecosystem (DIME) to facilitate the following:

Merging survey, respiratory specimen, serology, EMR, and registry data from five study sites
Creating and documenting hundreds of derived variables;
Conducting real-time checks for data
Review of free-text for categorization and to suppress personally identifiable information
Analytic file development, including rapid construction of five types of analytic files
Building Tableau dashboards to visualize completion rates of key study activities, participant demographics, recruitment progress, vaccination rates, and positive cases.

Abt delivers over 200,000 records and 250 million data points to CDC every week. Both DIME and DC-ACE are built on Amazon Web Services and are fully FISMA Moderate and HIPAA-compliant.

Evaluation and System Design for Career Pathways Programs: 2nd Generation of HPOG
Client: Administration for Children and Families (ACF)

The Health Profession Opportunity Grants (HPOG) Second Generation project documented the training and services received and the participant outcomes obtained under the HPOG program, which provided education and training to individuals with low-incomes for careers in the healthcare field. The project designed and built a customized management information system or MIS, the Participant Accomplishment and Grant Evaluation System (PAGES). Grantees used the system to enroll participants, manage program participation, track participant progress, and to generate the required semi-annual Performance Progress Reports for ACF. PAGES also supported and informed the National Evaluation through its random assignment and informed consent functions, by collecting baseline information and providing other program and participant data. Abt developed the extensive cloud-hosted system in Microsoft Dynamics CRM and maintained data for 32 HPOG 2.0 grantees, 27 non-tribal grantees (operating 38 distinct HPOG 2.0 programs) and 5 tribal grantees (operating 5 distinct programs). PAGES incorporated data on over 43,000 study participants across all of the tribal and non-tribal programs. At the end of the project Abt also generated a forthcoming lessons learned report describing the advantages and disadvantages of custom built and Commercial Off-the-Shelf (COTS) systems for similar projects.

National Corrections Reporting Program (NCRP)
Client: Bureau of Justice Statistics

NCRP provides offender-level data on admissions and releases from state prisons that are used to monitor the nation’s prison population and address specific policy questions such as recidivism, prisoner reentry, overcrowding, and changing demographic characteristics of the incarcerated population. Complementing NCRP is the National Prisoner Statistics (NPS) survey, which collects aggregate data on prison movements and is the basis for the Bureau of Justice Statistics’ widely-cited annual Prisoners publication on the Nation’s prison population.

Abt Global implemented a plan with two simultaneous components: collecting, processing and analyzing data submitted by state departments of corrections and developing an innovative technology solution designed to enable stakeholders and the greater public to analyze NCRP and NPS data via an interactive web tool—available at https://csat.bjs.ojp.gov/—that appropriately combines and weights the NCRP and NPS data. Abt has helped BJS achieve better and more extensive data collection participation from the states. It also has assisted BJS in developing sophisticated statistical products from those data, based on knowledge of the criminal justice field and experience manipulating complex datasets.

Prior to Abt becoming the NCRP data collection agent, NCRP data were divided into annual files that provided limited analysis options. Abt redesigned the NCRP as an assembly of offender prison term records linked over time within a state. The new longitidunal structure greatly improved quality control and created advanced new uses of the NCRP, such as calculating recidivism.

Program Evaluation for Prevention Contract (PEPC) – Strategic Prevention Framework for Prescription Drugs (SPF-Rx) Evaluation
Client: Substance Abuse and Mental Health Services Administration (SAMHSA)

The cross-site evaluation of SAMHSA’s Strategic Prevention Framework for Prescription Drugs (SPF-Rx). SPF-Rx is designed to address nonmedical use of prescription drugs as well as opioid overdoses by raising awareness about the dangers of sharing medications and by working with pharmaceutical and medical communities on the risks of overprescribing. The SPF-Rx program aims to promote collaboration between states/tribes and pharmaceutical and medical communities to understand the risks of overprescribing to youth ages 12-17 and adults 18 years of age and older. The program also aims to enhance capacity for, and access to, Prescription Drug Monitoring Program (PDMP) data for prevention purposes. This evaluation collects data from a cohort of 25 grantees across 21 states and four Tribal entities through an online data platform.

Key Digital Activities:

Developed a Federal Information Security Management Act (FISMA) moderate online data collection platform, the Data Management System (DMS), to collect annual data from SPF-Rx grantees
Developed a Resource Portal to house training resources and other grantee-specific materials
Conduct ongoing virtual training webinars for data collection activities
Established a project Help Desk to provide grantees with phone and email access to data collection support
Created data feedback forms for each grantee to improve data quality.