Education Research Case Studies
Evaluation of Chicago Public Schools Programs Title I Through IV Programs
Chicago Public Schools
Optimal was contracted to provide an independent evaluation for the Titles I through IV programs at their Private School Programs in Chicago. The evaluation team implemented a qualitative evaluation to analyze the impact of services to private school students, their teachers, and their families. The evaluation team conducted site visits, interviews, and focus groups and observed classrooms. The final product of the evaluation was a report that detailed the evaluation findings and provided recommendations for Chicago’s public schools to better track their programs and implement ongoing monitoring.
Implementation of the Credit Enhancement for Charter Schools Facilities
U.S. Department of Education
The U.S. Department of Education (ED) established, in 2001, the Credit Enhancement for Charter School Facilities Program (the Program). The Program makes available grants on a competitive basis to eligible entities—state or local government, private nonprofits, or consortia—which use Program funds for credit enhancements so that lenders will make loans. Optimal conducted an assessment of this project.
Please see the final report: https://www2.ed.gov/rschstat
Office of Vocational and Adult Education
U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy
Optimal and its subcontractors, Abt Associates Inc, the Conference Board of Canada, and DTI Associates, Inc. provided event and logistical planning support to the U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy (OVAE/DAEL) in efforts to promote the use of technology in the classroom, assist with distance learning programs, and enhancement adult education and reading/literacy programs. This project was authorized through the Adult Education and Family Literacy Act.
The ROI of the Foreign Student Exchange Program
U.S. State Department, Bureau of Educational and Cultural Affairs
Optimal developed a theoretical model that captures programs’ short-term and long-term economic impact on host communities. Short-term and long-term impact models were developed as part of the project. The short-term model, based on an Input-Output framework, describes the programs’ inputs and immediate and secondary impacts. Overall, the model is a theoretical framework with which one could assess the return to federal funds invested in the international exchange programs in the form of economic development in the host communities. The project results lay the groundwork for a user-friendly evaluation tool, which would allow program administrators to assess the economic impact of internal exchange programs based on a few key indicators.
Study of Teacher Preparation in Early Reading Instruction
U.S. Department of Education
The study examined the curriculum alignment of university elementary education programs with the findings of the National Reading Panel report (NRP) 2000. The study also assessed preservice teacher knowledge of the five key NRP components of early reading instruction for the U.S. Department of Education’s Institute of Education Sciences. As part of the study, Optimal was responsible for identifying a seven-person expert panel, recruiting over 100 participating schools, and collecting data through site visits, telephone interviews, and onsite student assessment.
Please read the final report:
Education Strategy Secondary Analysis for Results Tracking (SART)
U.S. Agency for International Development
Optimal has been contracted by USAID to design, implement, and operate a secure web portal to collect and track the progress of the $800M annual USAID Education Strategy. Optimal used ReveloTM to collect data and supporting documentation from 50+ countries and harmonize the data using an embedded ETL module and review and code documentation. ReveloTM uses embedded algorithms to analyze the data and auto-generate reports using SSRS and D3. Optimal obtained an authorization to operate (ATO) the FISMA low system from the USAID CIO. ReveloTM and Optimal also comply with NIST Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations (SP 800-137)
Ford Foundation for Low-Skilled Workers
Optimal Solutions Group (Optimal) was awarded a grant by the Ford Foundation’s Economic Development Unit to identify locally-based employment clusters that would be viable options for low-skill workers. Optimal based its research methodology on a recent report by Jobs for the Future (JFF), who attempted to define nationally-relevant job clusters for these low-skilled workers using a model based primarily on Bureau of Labor Statistics (BLS) data. While JFF’s study produced six potentially viable employment clusters, these clusters were not necessarily good choices throughout the country. Optimal researchers hypothesized that a model tailored to specific metropolitan area conditions (i.e., industrial-occupation mix, labor market conditions) could prove to be more useful to business leaders and policy makers as they seek to identify and build occupational clusters that will provide sustainable, living wage jobs to low-skilled workers.
Innovative Economics Support Services
Small Business Administration
In accordance with the Small Business Administration’s (SBA’s) Innovative Economies Support Services (IESS) contract, Optimal Solutions Group, LLC (Optimal) is responsible for fulfilling the following contract objectives:
- Analyzing seven Innovative Economies (IE) awardees and supporting SBA in identifying best practices for small-business integration and support.
- Analyzing a total of 20 Jobs and Innovation Accelerator Challenge (JIAC) awardees.
- Developing a program-evaluation framework and metrics for the JIAC.
- Monitoring and evaluating the IE and JIAC awardees.
- Providing technical assistance, business training, and business counseling to the IE and JIAC awardees.
- Communicating with the IE and JIAC cluster administrators.
Optimal is identifying and evaluating best practices to ensure that the economic clusters can create sustainable job growth, compete on national and global scales, and attract continuing business investment.
Child and Adult Care Food Program (CACFP) Assessment of Sponsor Tiering Determination
United States Department of Agriculture, Food and Nutrition Service
The CACFP provides reimbursements for nutritious meals and snacks served in family day care homes, child care centers, and other participating facilities and programs. Optimal conducted an assessment that provides for 2014 estimates of the number of FDCH misclassified by sponsoring agencies into the wrong tier, and the resulting erroneous payments for meals and snacks reimbursed at the incorrect rate. Optimal collected and analyzed data from State agencies and sponsors. State agencies provided the lists of sponsors and their FDCHs, which was used for the sampling, as well as provided schools’ free or reduced-price (F/RP) percentages for the purpose of determining FDCH’s eligibility. The Sponsors provided lists of their homes for the sampling, detailed meal counts for each sampled homes, and verification documents for the homes that could not be independently verified by Optimal.
Evaluation of Summer Food Service Program (SFSP) Participant Characteristics
United States Department of Agriculture, Food and Nutrition Service
This study is evaluating the current operations of the USDA’s SFSP. The program supports children’s nutrition through reimbursements to participating institutions for meals meeting USDA dietary guidelines to ensure that children who benefit from the NSLP and the SBP do not experience a nutrition gap during the summer. Optimal is conducting a process evaluation using a mixed methods approach consisting of web surveys and telephone calls to collect data on all 53 States participating in SFSP, as well as large, nationally representative samples of sponsors, and sites, augmented by qualitative data collection via one-on-one interviews with parents/caregivers of SFSP participants and non-participants. The data collection also involved States administrative data on characteristics and program outcomes for the sponsors and sites, and FNS administrative data on SFSP, NSLP, and SBP participation over time. The client indicates solid performance but the CPARS has not been completed.
Moving High-Performing Teachers to Low-Performing Schools
U.S. Department of Education (Subcontractor to Mathematica, Inc)
The U.S. Department of Education has contracted Mathematica Policy Research, Inc. (MPR) and subcontractors The New Teacher Project (TNTP) and Optimal Solutions Group, LLC (Optimal) to refine the design, implement, and rigorously evaluate an innovative program to induce high- performing teachers to transfer to low-performing schools. Referred to as the Master Teacher Residency Program (MTRP), the program will identify, screen, recruit, and place participants (“Residents” or “Master Teachers”) in low-performing schools, work closely with principals in those schools, and support the teachers in their new environments. The program will first be introduced in two districts in a two-year pilot: Charlotte-Mecklenburg Schools in North Carolina and Hillsborough County in Florida. If successful, it will then be implemented in 10 districts, and an evaluation of the program’s impact on student achievement will be conducted. Optimal is currently tasked with the development and implementation of a survey for employers who will verify the employment of Master Teachers as a requirement for the program.
National Center on Service Obligations (NCSO)
U.S Department of Education
For this contract, Optimal monitored grant recipients and reports findings relating to GPRA measures that speak to the success of this grant program. The NCSO team developed reports and protocols to address GPRA and PART requirements. The project team also developed the website for this project that provides an overview of the grant program and regulations and updates this website annually, similar to the requirements of task 8 for the IFLE contract. Optimal designed and implemented a real-time, secure, ED-accredited web-based data collection tool, automating the data collection, analysis and reporting of Personnel Development Program data through the Service Obligation Tracking System (SOTS). As a result, NCSO staff focuses on the analysis and review of data to supplement reporting efforts for to ED. To combat the volume of scholars and to better facilitate communication among grantees, scholars, and employers, Optimal designed and maintains the SOTS, including its multifaceted features of automated sending of e-mails, tiered access and unique user environments with varied permissions and end-user types, automated reports, and accommodation for GPRA program assessment measurement reporting.
The Evaluation of the Impact of Charter School Strategies
U.S. Department of Education (Subcontractor to Mathematica)
To evaluate the impacts of charter school models on student achievement and these other outcomes, school admissions lotteries served as the randomizer to create two identical groups of students from among those who apply to charter schools eliminating the bias in who selects to participate and making comparisons possible between the treatment group-students enrolled in charter schools-and the control group-students who applied but were not selected in the lottery. The study followed the treatment and control groups of students over time and measured and compared their academic outcomes. The study employs both quantitative data such as student test scores and qualitative methods such as student, parent, and principal surveys. As a subcontractor to Mathematica Policy Research, Optimal participated in the recruitment of charter middle schools in Florida, Michigan, Minnesota, California, and Arizona for the study sample as well as monitored the random assignment process at these schools through site visits. Optimal also led the data collection task working with state and local education agencies, participating charter schools, public schools and private schools to obtain participating student records data.
Integrated Postsecondary Education Data System (IPDES) Graduation Rates Validity Study
United States Department of Education
The study provides a detailed review of graduation rates from the Graduation Rate Survey (GRS) component of the U.S. Department of Education’s Integrated Postsecondary Education Data System. The review describes conceptual and coverage differences between the graduation rates calculated using the Student Unit Record Systems (SURs) data from three states to those reported in the IPEDS Graduation Rate Survey component. Estimates of bias for the GRS graduation rate statistic will be obtained for each state. To validate the quality of the GRS graduation rate data collected in IPEDS, the team will also use National Collegiate Athletic Association (NCAA) data as it contains comparable graduation rate information. Optimal, with subcontractor Mathematica Policy Research (MPR), has developed an in-depth analysis plan, identified states for participation, and begun analysis using SAS. To identify participating states, Optimal used the NCHEMS survey to rank states based on compatibility between the state’s student unit record systems and the IPEDS GRS. Additionally, the team conducted calls with all states with student unit records in order to gather a further understanding of their capabilities.
Education Statistics Support Institute Network (ESSIN): Assessment Division Support
U.S. Department of Education NCES
Optimal works closely with other ESSIN partner organizations to review NCES products, develop complex methodological frameworks to analyze educational trends, develop robust visualization to illustrate findings, and overall provide policy solutions to relevant stakeholders. Optimal plays a key role in the reporting and dissemination of the Nation’s Report Card and conducts policy relevant analysis on what students know and can do in the nation. Tasks in this project include collecting and standardizing data in different formats and conducting analysis based on the standardized data. For ESSIN, the Analytics team used a customization of the Real-time Framework using Excel VBA code that allowed them to take large volumes of NAEP Data Explorer (NDE) output/data (in Excel)) and transform it into tables and line graphs (as opposed to the ESSIN team having to build hundreds of tables and line graphs)
Learning System Data Management (LSDM)
DHHS Centers for Medicare & Medicaid Services CMMI
The Centers for Medicare & Medicaid Innovation contracted with Optimal to track and monitor learning and diffusion activities and events across 15+ innovation models designed to improve healthcare quality and reduce costs. Optimal designed DMARS, hosted on CMMI’s AWS FedRAMP environment, to facilitate data collection from the model contracts, automate quality assurance and reporting. Optimal is also facilitating CQI within the ecosystem (i.e., LDG, learning system contractors, and external stakeholders) while continuing to enhance DMARS. DMARS was granted an ATO in July 2017. DMARS allows learning system contractors to bulk import multi-dimensional learning event data and to receive automated feedback for any data elements that violate pre-defined parameters, download their own learning systems’ data after it has been normalized, and to view KPIs in an interactive dashboard.
Influence of Social Media on Youth Drug Use
Office of National Drug Control Policy (ONDCP)
Optimal and CESAR conducted a Youth Focus Group consisting of 19 young adults aged 18 to 25 to collect their perceptions on the use of social media by youth and young adults and its influence on health-related behaviors, such as substance use and abuse. Campus listservs, flyers, and student newspapers were used to recruit participants. The Young Adult Focus Group notice generated significant interest among young adults in the College Park, MD community. The participants were encouraged to provide their response to the research questions. The results identified as a product of the youth discussion comments reflect differences among the participant sample responses. The participants were diverse demographically in terms of gender and race and ethnicity. In general, the young adults agreed with the experts that youth are frequent and agile users of social media, and that drug and alcohol content is easily accessible.
Centers for Medicare & Medicaid Services
CMS requires Medicare Administrative Contractors (MACs) to build and maintain websites for Medicare beneficiaries and providers with high volumes of information about Medicare claims. CMS and the MACs now have access to monthly measures of their compliance with accessibility requirements from our iAccessible platform. Optimal scans the MAC websites for compliance monthly. Optimal also provides detailed data at the webpage level that allows MACs to identify, prioritize, and remediate compliance issues and move towards best practices. The collection of longitudinal data allows CMS to measure the MACs improvement over time. Role-based access on the eAccessibility Platforms allows authorized CMS users to view and interact with compliance results for all of its Contractors under assessment, which limiting access to individual Contractors to their own data and summary reports.
Mid-Term Performance Evaluation of Junior Achievement of Armenia (JAA) Entrepreneurship and Civic Activism for Young People
USAID/Armenia funded the 4-year Entrepreneurship and Civic Activism for Young People program, which was launched in April 2011 by Junior Achievement of Armenia (JAA). The project combines a longer-standing effort to improve youth education in economics with the added goals of increasing entrepreneurship and community-based civic activities that address community needs by equipping Armenian youth with the skills and knowledge necessary to compete and succeed in tomorrow’s world. JAA operates a number of related programs to educate students on international business practices, ethics, and corporate social responsibility issues. Under the USAID Learning, Evaluation, and Analysis Project contract, Optimal was tasked with planning, designing, and conducting a mid-term performance evaluation of the JAA program. In November and December 2013, Optimal conducted 31 site visits across 6 marzes (districts) and performed in-depth interviews and focus groups with key stakeholders and beneficiaries
Evaluation of the Graduate Nurse Education Demonstration (GNE), Phase I
Centers for Medicare & Medicaid Services
The Graduate Nurse Education (GNE) demonstration, authorized by the Affordable Care Act, aims to increase the supply of APRNs in the U.S. health care delivery system by providing Medicare payments to five selected hospitals for the reasonable cost of providing clinical training to APRN students. This demonstration also contributes to the creation of partnerships between hospitals, schools of nursing (SONs), and community-based care settings (CCSs).
Optimal and its subcontractor, AIR, designed and implemented a program evaluation to inform the demonstration’s Report to Congress (RTC). The evaluation design includes a mixed methods approach to executing a structure and process evaluation of how well the sites implemented the demonstration—including challenges, successes, and innovations—through qualitative interviews and focus groups.