2020 Federal Standard of Excellence

Innovation

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY20? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements)

Score
6
Administration for Children and Families (HHS)
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • In late 2019, ACF stood up a customer experience initiative to enhance ACF’s delivery and administration of human services. This initiative focuses on ways to improve the experiences of both grantees and ACF employees. One sub-initiative is to equip ACF leaders and staff with fundamental innovation strategies and ways of fostering a culture of innovation within their programs. In early 2020, ACF invited external experts to host two skill-based trainings for ACF staff: 1) Innovation as a Discipline: Empowering Employees to Change the Game, and 2) Human-Centered Design Training: Putting People at the Center of What we Do. This initiative is ongoing.
  • ACF leadership and staff have also collaborated with other federal agencies, local leaders, and entrepreneurs around the practice of innovation. In January of 2020, the ACF Office of Early Childhood Development (ECD) partnered with other ACF program offices and the Department of Education to put on the first-ever Showcase on Early Childhood Development and Learning within the Annual ED Games Expo. About 100 federal, national, state, and local leaders in early childhood, education, health, and human services joined ACF leadership and 11 thought leaders and entrepreneurs who presented a series of Big Idea talks about innovation and how to scale-up good ideas. This Showcase was one part of a multi-day event sponsored by ED’s Institute of Education Science which focused on games and technology targeted at children, parents, educators, funders, and other stakeholders with more than 1000 people attending.
  • HHS has embarked on a process called ReImagine HHS, which has engaged leadership and staff from around the department to identify strategic shifts to transform how HHS operates. One part of this larger initiative, called Aim for Independence (AFI), is using a human centered design approach to rethink how ACF does work and how that work translates into long-lasting, positive outcomes for parents and children. Engagement activities have included a leadership retreat and opportunities for staff input.
  • ACF leadership has proposed new Opportunity and Economic Mobility Demonstrations to allow states to redesign safety net service delivery by streamlining funding from multiple public assistance and workforce development programs and providing services tailored to their populations’ specific needs. The demonstrations would be subject to rigorous evaluation.
 7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • ACF’s mission to “foster health and well-being by providing federal leadership, partnership and resources for the compassionate and effective delivery of human services” is undergirded by six values: dedication, professionalism, integrity, stewardship, respect, and excellence. ACF’s emphasis on excellence, “exemplified by innovations and solutions that are anchored in available evidence, build knowledge and transcend boundaries,” drives the agency’s support for innovation across programs and practices.
  • For example, ACF’s customer experience initiative is supporting the development of innovative practices for more efficient and responsive agency operations, including the identification of new ways to streamline grantee compliance requirements, minimize administrative burden, and increase grantee capacity for service delivery.
  • ACF also administers select grant programs—through innovation projects, demonstration projects, and waivers to existing program requirements–that are designed to both implement and evaluate innovative interventions, as a part of an ACF-sponsored evaluation or an individual evaluation to accompany implementation of that innovation. For example:
  • ACF projects that support innovation include:
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • In addition to the list of ACF demonstration projects, innovation projects, and waiver programs with rigorous evidence activities built into their delivery (as described in sub-criteria 7.2), ACF also conducts rigorous research on other innovative human services.
  • The evaluations below are on-going rigorous evaluations conducted by ACF:
  • ACF administers a series of Head Start and Early Head Start University Partnership Grants in which university researchers partner with local Head Start or Early Head Start programs to conduct an implementation study and evaluate the effectiveness of innovative strategies for improving service quality and/or child/family outcomes. Past grants programs have examined promising parenting interventions, dual-generation approaches, integrated interventions in center-based Early Head Start, and approaches for working with dual language learner.
Score
4
Administration for Community Living (HHS)
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • In FY20 all ACL Centers were involved in funding innovative work. ACL released several funding opportunity announcements (FOA) focused on the identification and implementation of innovative approaches to improve programming. These included the Innovations in Nutrition Programs and Services – Demonstration , the Small Business Innovation Research (SBIR) Program, and Grants to Enhance State Adult Protective Servicesdemonstration addresses gaps and challenges in state APS systems through innovations and improvements in practice, services, data collection, and reporting. ACL also funded challenge competitions to spur development of the interoperable, statewide referral and analytics platforms needed to enable these partnerships ( IT Solutions) and to combat the social isolation and loneliness that older adults, people with disabilities and veterans often experience (Mental Health Challenge).
  • ACL is a funder of The John A. Hartford Foundation 2020 Business Innovation Award which recognizes aging and disability community-based organizations (CBOs) for their innovative approaches to reducing health care costs and improving the well-being of older adults and people with disabilities through strategic partnership with health care entities.
  • There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3.5 million to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.
  • State Councils on Developmental Disabilities (SCDD) are charged with identifying and addressing the most pressing needs of people with developmental disabilities in their state and territory. Councils work with different groups in many ways, including educating communities to welcome people with developmental disabilities; funding projects to show new ways that people with disabilities can work, play, and learn; and seeking information from the public as well as state and national sources.
  • The 2020 reauthorization of the Older Americans Act has provisions for technical assistance and innovation to improve transportation for older individuals.
  • In 2020, ACL awarded grants for demonstrations in Innovations in Nutrition Programs and Services to support the documentation of innovative projects that enhance the quality, effectiveness, and other proven outcomes of nutrition services programs within the aging services network. The goal of this funding opportunity is to support projects that can demonstrate potential for broad implementation throughout the aging services network, and with demonstrated value, i.e., improvements in participant well-being, cost savings, etc.
  • As previously mentioned, National Institute on Disability, Independent Living, and Rehabilitation Research(NIDILRR) uses a stages of research framework (SORF) process to gain insight into what is known and unknown about a problem; whether it is time to develop interventions to address a particular problem; whether it is time to test the efficacy of interventions (evaluate its innovation efforts); and whether it is time to scale-up interventions for broader use (improve the impact of its programs).
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • The 2020 reauthorization of the Older Americans Act requires a new Research, Demonstration, and Evaluation Center for the Aging Network and new demonstration programs to evaluate new strategies for the recruitment, retention, or advancement of direct care workers, and the soliciting, development, and implementation of strategies; and a demonstration to address negative health impacts associated with social isolation. Further, ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches towards coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e. Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.
  • As previously mentioned, NIDILRR’s research and development activities are guided by the Stages of Research Framework and the Stages of Development Framework. NIDILRR grantees conducting research projects must identify the stage of research their projects are in while grantees conducting development projects must identify the stage of development their projects are in. The stage a research project is in depends upon what is known and what is not known about a particular disability problem or topic. Research projects where relatively little is known, or the topic area is emerging, would be classified in the Exploration and discovery stage. Over time, as more becomes known, research projects progressively move into the Intervention Development phase. The next phase, known as Intervention Efficacy, means the stage of research during which a project evaluates and tests whether an intervention is feasible, practical, and has the potential to yield positive outcomes for individuals with disabilities. The final stage, known as Scale-Up Evaluation, means the stage of research during which a project analyzes whether an intervention is effective in producing improved outcomes for individuals with disabilities when implemented in a real-world setting.
  • Similarly, the stage of development a development project is in also depends upon what is known or not known about a need that informs the design and development of a product. The proof of concept stage means the stage of development where key technical challenges are resolved. Stage activities may include recruiting study participants, verifying product requirements; implementing and testing (typically in controlled contexts) key concepts, components, or systems, and resolving technical challenges. The proof of product stage means the stage of development where a fully-integrated and working prototype, meeting critical technical requirements is created. The proof of adoption stage means the stage of development where a product is substantially adopted by its target population and used for its intended purpose.
Score
7
U.S. Agency for International Development
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • In FY19, USAID appointed a new Chief Innovation Officer to advocate for innovation throughout development and national security strategies across USAID, the U.S. Government, and the international community. The Chief Innovation Officer promotes opportunities for entrepreneurs, implementing partners, universities, donors, and others to test and scale innovative solutions and approaches to development problems around the world. In FY2019, the U.S. Global Development Lab also engaged USAID leadership and Mission staff from around the world at the Mission Directors Conference, the Contracting Officer and Controller Conference, the Foreign Service National Conference, and the Private Sector Engagement Forum.
  • For innovations specific to a particular sector, Agency leadership has supported technical staff in surfacing groundbreaking ideas, such as how the Bureau for Global Health’s Center for Innovation and Impact (CII) used open innovation approaches to issue the Saving Lives at Birth Grand Challenge and identify promising, life-saving maternal and newborn health innovations.
 7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • In FY2020, USAID released its first Digital Strategy, moving to a “Digital by Default” position and USAID’s innovative approaches have helped get more than 40 million people in the developing world digital access. USAID’s New Partnerships Initiative (NPI) will allow USAID to work with a more diverse range of partners, strengthen existing partner relationships, and provide more entry points for organizations to work with the Agency. The principles behind NPI are outlined in the Agency’s first-ever Acquisition and Assistance (A&A) Strategy.
  • USAID and its partners have launched 41 innovative programming approaches including prizes, ventures, challenges, and Grand Challenges for Development since 2011. Across the Grand Challenges portfolio, partners have jointly committed over $535 million ($155 million from USAID) in grants and technical assistance for over 528 innovators in 107 countries. To date, more than $614 million in follow-on funding has been catalyzed from external sources, a key measure of success.
  • USAID’s investment in state-of-the-art geo and information intelligence centers mean that any program has the ability to leverage geospatial analysis and critical data sets to drive innovative solutions based on evidence and data. With over 20 programs experimenting with Artificial Intelligence and machine learning, and USAID’s strong work on Digital finance and connectivity, the Agency is using technology to drive our programs farther, faster. USAID has also completed more than 1,500 Global Development Alliances, leveraging private sector in-kind or financial investments.
  • In addition, the Center for Innovation and Impact (CII)—the Bureau for Global Health’s dedicated innovation office—takes a business-minded approach to fast-tracking the development, introduction, and scale-up of health innovations that address the world’s most important health challenges, and assessing and adopting cutting-edge approaches (such as using unmanned aerial vehicles and artificial intelligence).
  • Feed the Future Partnering for Innovation partners with agribusinesses to help them commercialize and scale new agricultural innovations to help improve the livelihoods of smallholder farmers, increasing their productivity and incomes. To date the program has worked with 59 partners in 20 different countries, investing more than $43 million in new technologies and services, and leveraging nearly $100 million in private sector investment. The program has helped commercialize over 118 innovations, which resulted in an estimated $99 million in sales. It has its own Innovation site that partners can easily see and connect with promising innovations and research.
  • Finally, USAID was honored when the co-founder and Scientific Director of USAID’s Development Innovations Venture (DIV) program, Dr. Michael Kremer received the 2019 Nobel prize for economics, along with Dr. Esther Duflo and Dr. Abhijit Banerjee. Some of his work that led to this honor was connected to USAID’s DIV program. DIV values rigorous testing methods such as impact evaluations or robust market tests to measure the impact of USAID innovations. Evidence of clear and measurable outcomes helps demonstrate what is working and what is not. Solutions that demonstrate rigorous evidence of impact can then be scaled to other contexts. Through the DIV program, Dr. Kremer helps USAID use evidence-driven approaches to take small risks, identify what works, and scale those approaches to provide greater impact, which helps partners on the Journey to Self-Reliance. Since 2010, the DIV program has supported over 200 awards to test and scale development focused innovations that have directly affected more than 30 million lives across 46 countries.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Within the U.S. Global Development Lab, the MERLIN program works to innovate on traditional approaches to monitoring, evaluation, research and learning. While innovative in themselves, these approaches can also be better suited to evaluating an innovation effort. Two examples include Developmental Evaluation, which aims to provide ongoing feedback to managers on implementation through an embedded evaluator, and Rapid Feedback, which allows implementers to test various methods to reach certain targeted results (more quickly than through traditional midterm or final evaluations).
  • Many of the agency’s programs such as Grand Challenges and Development Innovation Ventures (DIV) have been reviewed by formal audit and other performance and impact interventions. DIV is USAID’s tiered, evidence-driven open innovation program. It awards grants for innovative solutions to any development problem, on the basis of rigorous evidence of impact, cost-effectiveness, and a pathway to scale via the public and/or private sectors. The DIV model is designed to source breakthrough solutions, to minimize risk, and maximize impact by funding according to outcomes and milestones, to rigorously evaluate impact and cost-effectiveness, and to scale proven solutions.
  • DIV supports innovative solutions across all countries and development sectors in which USAID operates, including education, agriculture, water, energy, and economic development. Over ten years, since 2010, DIV has invested more than $129 million in over 200 innovations in 46 countries, improving the lives of over 55 million beneficiaries.  As of 2017, DIV funded entities have been able to catalyze the USAID DIV investment through private capital at a leveraged rate of $1.59 for every $1.
  • It has generated experimental or quasi-experimental evaluation studies of more than a third of those innovations as well as a forthcoming working paper that rigorously assesses the social rate of return of DIV’s early portfolio. Since DIV reopened in fall 2018, approximately 94 percent of its 2,147 applicants were new to USAID. DIV enhances the Agency’s engagement with non-traditional partners by partnering with innovators ranging from local entrepreneurs to researchers to high-growth start-ups to American small businesses trying to take their innovation global.  In total, 53 percent of organizations supported by DIV have been new to USAID. DIV is an important means for for-profit companies––comprising 43 percent of DIV’s portfolio––to engage with the Agency.  Various USAID Missions and Bureaus, including Egypt, Zambia, Bangladesh, Global Health, and the Women’s Global Development and Prosperity (W-GDP) Initiative, and many more have partnered with and funded innovations identified by DIV.
Score
5
AmeriCorps
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impacts of its programs?
  • Staff at all levels of the organization participate in work groups focused on implementing AmeriCorps’s Transformation and Sustainability Plan, aimed at supporting the agency’s efforts to improve operations, results, and meeting its mission. The CEO has also conducted Service Jams to elicit feedback from staff to support the plan. Service Jam topics have focused on what a best-in class learning organization looks like and how AmeriCorps could break down silos.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • AmeriCorps’s Evidence Exchange includes a suite of scaling products on the evidence exchange to help grantees replicate evidence-based interventions.
  • AmeriCorps continued to learn from its evidence-based planning grant program which “awards evidence-based intervention planning grants to organizations that develop new national service models seeking to integrate members into innovative evidence-based interventions.” AmeriCorps continued to learn from its research grantees, who receive grant funds to engage community residents and leaders in the development of new and innovative national service projects. In addition to national service project development, these grants foster civic engagement through community research teams and build community capacity for using research to identify and understand local issues as well as identify possible solutions. Examples of these research-to-action projects include:
    • A researcher at the University of Nevada worked with NCCC Pacific leaders to craft a series of local projects, like building sidewalks and community cleanups, emerging from her CBPR project with youth scientists working together to understand slow violence in their own communities as well as that of people experiencing homelessness in the area. The partnership between NCCC, the University of Nevada is quite strong and the local government is supportive of the work in the region. Early in 2020, housing had been donated by a local community organization and the NCCC team was assigned and scheduled to begin on April 21st but the work was placed on hold due to the COVID19 pandemic. Recently, PI and ADP have resumed discussions about rescheduling the team’s arrival.
    • Researchers at the Virginia Polytechnic Institute, State University Tech (Virginia Tech University), and Virginia Commonwealth University have brought together community partners and stakeholders in Martinsville, VA to address the local opioid crisis. They are using an evidence-based stakeholder engagement approach (SEED) that has led to successful outcomes in Martinsville. In Year 2, they collaborated with the Minnesota AmeriCorps state commission, ServeMinnesota, to replicate this project and approach with a focus on deploying AmeriCorps volunteers to meet unmet service needs around the opioid crisis in Minneapolis. Because of the success in rural Virginia and Martinsville, this approach will be further replicated in another town in rural Virginia and another town outside Minneapolis.
    • A researcher at Mississippi State University collaborated with NCCC Southern Campus to draft a concept paper for a NCCC team when COVID-19 struck – both agreed to table ideas for FAFSA support and ACT preparation until the crisis had subsided.
    • A researcher Drew University successfully collaborated with a former senior New Jersey state government official on a concept paper for VISTAs to work with their community partner, Family Promise, to build the organization’s capacity to work with local landlord’s and people experiencing homelessness – a role identified through their grant funded research. They were awarded two VISTAs.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • As part of the evaluation of the Social Innovation Program, which was designed to identify and rigorously test innovative approaches to social service problems, AmeriCorps continues to receive evaluation reports from grantees. As of May 2020, AmeriCorps has received 129 final SIF evaluation reports, of which 31 (24%) were experimental designs and 74 (57%) were quasi-experimental designs. Further, the evidence-based planning grant program and the research grant program both seek to generate innovative national service models. The planning grants require an evaluation plan. The research grants use evidence to inform action planning and solutions. The Office of Research and Evaluation is planning an evaluation of this grant program to identify outcomes, including the outcomes of national service projects developed through participatory research. The goal is to contract with a third party to evaluate the effects of the research grant program before the end of FY20.
Score
6
U.S. Department of Education
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • In FY19, the Office of Elementary and Secondary Education (OESE) made strategic investments in innovative educational programs and practices and administered discretionary grant programs. In FY19, the Innovation and Improvement account received $1.035 billion. ED reorganized in 2019, consolidating the Office of Innovation and Improvement into the Office of Elementary and Secondary Education. To lead and support innovation within the reorganized OESE, ED created the Evidence-Based Policy (EBP) team. EBP teams work within OESE and with colleagues across the agency to develop and expand efforts to inform policy and improve program practices.
  • In the reorganization that created the Office of the Chief Data Officer, ED leadership established a unit focused explicitly on Innovation and Engagement. These staff focus on innovation in data infrastructure and use, leading the development of the Open Data Platform using agile methodology. Innovations are discussed and disseminatedfor use at monthly meetings of the Data Strategy Team (DST), consisting of data professionals from across ED. Recent DST meetings have included presentations on the Federal Student Aid data warehouse, introductions to Tableau and PowerBI, and a demonstration of the Open Data Platform detailing the role of office data stewards.
 7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • To lead and support innovation within the reorganized OESE, ED created a new component: the Evidence-Based Practices (EBP) team. EBP is tasked with promoting evidence consistent with relevant provisions of the Elementary and Secondary Education Act (ESEA) as amended by Every Student Succeeds Act’s (ESSA). EBP includes two units—one to support the many discretionary grant programs in OESE and one to support the formula grant programs and any discretionary grant programs associated with the formula grant programs. EBP works to advance an evidence-based grantmaking agenda and seeks to operationalize the ESSA evidence framework for strengthening the effectiveness of ESEA investments within OESE programs by: 1) Leading OESE policy development and serving as consultants to grant programs on program design and implementation using evidence, data, trends, field experiences, and stakeholder input; 2) Assessing and improving evidence-based grant-making processes and decision-making to drive results and outcomes aligned with strategic goals; and, 3) Identifying and disseminating promising and evidence-based practices by convening practitioners and producing practitioner-friendly resources.
  • The Education Innovation and Research (EIR) program is ED’s primary innovation program for K–12 public education. EIR grants are focused on validating and scaling evidence-based practices and encouraging innovative approaches to persistent challenges. The EIR program incorporates a tiered-evidence framework that supports larger awards for projects with the strongest evidence base as well as promising earlier-stage projects that are willing to undergo rigorous evaluation. Lessons learned from the EIR program have been shared across the agency and have informed policy approaches in other programs.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • ED is currently implementing the Experimental Sites Initiative to assess the effects of statutory and regulatory flexibility for participating institutions disbursing Title IV student aid. ED collects performance data from all participating institutions, and IES is currently conducting rigorous evaluations of selected Experimental Sites, including two related to short-term Pell grants.
  • The Education Innovation and Research (EIR) program, ED’s primary innovation program for K–12 public education, incorporates a tiered-evidence framework that supports larger awards for projects with the strongest evidence base as well as promising earlier-stage projects that are willing to undergo rigorous evaluation.
Score
6
U.S. Dept. of Housing & Urban Development
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • The Department promotes evidence-based innovation by using program demonstrations to experimentally test potential policy enhancements, which have included eight low-cost, behaviorally informed experiments using interagency data matching and assistance from the GSA Office of Evaluation Sciences. Other innovative research ideas from external stakeholders are supported by the Research Partnerships program. Competitive awards for Healthy Homes Technical Studies generate innovation in the evaluation and control of housing-related health and safety hazards.
  • An interagency agreement with the Census Bureau has made datasets from HUD’s randomized control trials available for linkage with census data and administrative datasets. The RCT datasets are the first intervention data added to Federal Statistical Research Data Centers (RDCs) by any federal agency, and joint support is available to help researchers gain access and learn to use the restricted data successfully for innovative research, with seven projects currently underway.
  • HUD’s Rental Assistance Demonstration, which restructures the financing of the nation’s public housing to address capital needs backlogs, has the additional innovative feature of providing tenants with a Choice Mobility option. Choice Mobility supports self-sufficiency by offering priority receipt of a Housing Choice Voucher providing freedom to move to neighborhoods with greater economic opportunities or better schools and amenities.
  • HUD established the Office of Innovation in 2019 to advance innovation in several domains. The office managed the 2019 Innovative Housing Showcase and is developing a similar Showcase for 2021 and prize competitions to stimulate innovation in housing and HUD policy and programs. FY20 grants fund cooperative agreements for pre-competitive research in homebuilding innovations, with a similar program for Historically Black Colleges and Universities, for more affordable, energy efficient, resilient, and healthier housing
  • HUD’s regulation of manufactured housing production is guided by a federal advisory committee, the Manufactured Housing Consensus Committee, to provide increased ability for the industry to produce some of the nation’s most innovative, safe, and affordable housing.
  • HUD has a Robotics Process Automation initiative devoted to freeing the workforce from low-value, repetitive work through software robotics solutions. Specialized computer programs known as “bots” automate and standardize repeatable business processes without costly investments in conventional automation. Planned efforts involving payroll, accounts receivable and payable, invoice processing, inventory management, report creation, and data migration have potential to shift over 50,000 hours of employee time from low-value to high-value work.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
Score
5
U.S. Department of Labor
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • DOL’s Chief Data Officer and Chief Evaluation Office (CEO) Data Analytics team developed a secure data analysis platform accessible to all DOL staff, pre-loaded with common statistical packages and offering the capability to access and merge various administrative data for analysis. DOL supports staff in executing limitless web-based A/B testing and other behaviorally-informed trials, with the shared service of the advanced Granicus platform’s GovDelivery communications tool, including free technical support. This tool enhances the Department’s ability to communicate with the public, such as through targeted email campaigns, and to adjust these communications, informed by testing and data, to increase engagement on relevant topics. The CEO also has developed toolkits and detailed resources for staff to effectively design behaviorally informed tests.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • The CEO uses a variety of communication tools to share rigorous research results, lessons learned, promising practices, and other implications of its research. These include internal briefings from independent contractors and researchers, a brownbag series that features evidence-based promising practices and results shared by DOL staff, for DOL staff, and an external expert seminar series featuring new findings or innovations in relevant areas of work. CEO staff consistently use research findings in the development of new research, and DOL agencies use findings to design and guide new discretionary grant programs, to refine performance measures for grantees, and to make decisions on compliance and enforcement practices.
  • DOL is strongly committed to promoting innovation in our policies and practices. For example, the Employment & Training Administration’s (ETA) competitive funding routinely funds innovative programming, since grantees typically bundle various program services and components to best meet the needs of the people being served by them in their local contexts. A particularly good example of where this innovation is happening is in the Administration’s high priority area of apprenticeships. DOL is funding $150 million to support sector-based innovations in apprenticeship. DOL has invested more than $95 million through the ApprenticeshipUSA initiative – a national campaign bringing together a broad range of stakeholders, including employers, labor, and states as well as education and workforce partners, to expand and diversify registered apprenticeships in the United States. This includes more than $60 million for state-led strategies to grow and diversify apprenticeship, and state Accelerator Grants to help integrate apprenticeship into education and workforce systems; engage industry and other partners to expand apprenticeship to new sectors and new populations at scale; conduct outreach and work with employers to start new programs; promote greater inclusion and diversity in apprenticeship; and develop statewide and regional strategies aimed at building state capacity to support new apprenticeship programs. All of these grants include funding for data collection; additionally, ETA and CEO are conducting an evaluation of the American Apprenticeship Initiative.
  • In addition, CEO’s Behavioral Insights team works with a number of DOL agencies on a continuous basis to identify and assess the feasibility of conducting studies where insights from behavioral science can be used to improve the performance and outcomes of DOL programs. The Wage and Hour Division’s (WHD) Transformation Team is one such example where continuous improvement efforts are driving innovation. Their work has identified potential areas where behavioral interventions and trials may inform program improvement. CEO is also working across agencies – including WHD, ETA, Women’s Bureau, Veterans Employment & Training Service (VETS), Office of Federal Contract Compliance Programs (OFCCP), and International Labor Affairs Bureau (ILAB) – to identify and assess the feasibility of other areas where insights from behavioral science can be used to improve the performance and outcomes of DOL programs.
  • DOL has also built capacity for staff innovation through the Performance Management Center’s Continuous Process Improvement (CPI) Program, an agency-wide opportunity which trains and certifies agency staff on Lean Six Sigma (LSS) methodologies through real-time execution of DOL process improvement projects. The program includes classroom sessions that prepare participants for LSS Black Belt certification examinations, including the American Society for Quality (ASQ) as well as DOL’s own certification.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • DOL, through the annual Learning Agenda process, systematically identifies gaps in the use of evidence. Innovation is about filling known gaps via dissemination, further research, or generation of quick turnaround assessments, like those offered to the Department by CEO’s Behavioral Insights Program.
  • DOL typically couples innovation with rigorous evaluation to learn from experiments. For example, DOL is participating in the Performance Partnership Pilots (P3) for innovative service delivery for disconnected youth which includes not only waivers and blending and braiding of federal funds, but gives bonus points in application reviews for proposing “high tier” evaluations. DOL is the lead agency for the evaluation of P3. A final report is available on the CEO’s completed studies website.
  • DOL routinely uses Job Corps’ demonstration authority to test and evaluate innovative and promising models to improve outcomes for youth. Currently, the CEO is sponsoring a rigorous impact evaluation to examine the effectiveness of one of these pilots, the Job Corps Experimental Center Cascades, with results expected in FY21.
Score
7
Millennium Challenge Corporation
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • MCC supports the creation of multidisciplinary country teams to manage the development and implementation of each compact and threshold program. Teams meet frequently to gather evidence, discuss progress, make project design decisions, and solve problems. Prior to moving forward with a program investment, teams are encouraged to use the lessons from completed evaluations to inform their work going forward.
  • In FY20, MCC launched its second-ever internal Millennium Efficiency Challenge (MEC) designed to tap into the extensive knowledge of MCC’s staff to identify efficiencies and innovative solutions that can shorten the compact and threshold program development timeline while maintaining MCC’s rigorous quality standards and investment criteria.
  • In September 2014, MCC’s Monitoring and Evaluation division launched the agency’s first Open Data Challenge, which continued into FY20. The Open Data Challenge initiative is intended to facilitate broader use of MCC’s U.S.-taxpayer funded data, encourage innovative ideas, and maximize the use of data that MCC finances for its independent evaluations.
 7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • MCC’s approach to development assistance hinges on its innovative and extensive use of evidence to inform investment decisions, guide program implementation strategies, and assess and learn from its investment experiences. As such, MCC’s Office of Strategic Partnerships offers an Annual Program Statement (APS) opportunity that allows MCC divisions and country teams to tap the most innovative solutions to new development issues. In FY20, the Monitoring and Evaluation division, using MCC’s APS and traditional evaluation firms, has been piloting partnerships with academics and in-country think tanks to leverage innovative, lower cost data technologies across sectors and regions. These include:
    • using satellite imagery in Sri Lanka to measure visible changes in investment on land to get early indications if improved land rights are spurring investment;
    • leveraging big data and cell phone applications in Colombo, Sri Lanka to monitor changes in traffic congestion and the use of public transport; independently measuring power outages and voltage fluctuations using cell phones in Ghana, where utility outage data is unreliable, and where outage reduction is a critical outcome targeted by the Compact;
    • using pressure loggers on piped water at the network and household levels to get independent readings on access to water in Dar es Salaam, Tanzania; and using remote sensing to measure water supply in water kiosks in Freetown, Sierra Leone.
  • MCC regularly engages in implementing test projects as part of its overall compact programs. A few examples include: (1) in Morocco, an innovative pay-for-results mechanism to replicate or expand proven programs that provide integrated support; (2) a “call-for-ideas” in Benin for information regarding potential projects that would expand access to renewable off-grid electrical power; (3) a regulatory strengthening project in Sierra Leone that includes funding for a results-based financing system; and (4) an Innovation Grant Program in Zambia to encourage local innovation in pro-poor service delivery in the water sector.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • Although MCC rigorously evaluates all program efforts, MCC takes special care to ensure that innovative or untested programs are thoroughly evaluated. In addition to producing final program evaluations, MCC is continuously monitoring and evaluating all programs throughout the program lifecycle, including innovation efforts, to determine if mid-program course-correction actions are necessary. This interim data helps MCC continuously improve its innovation efforts so that they can be most effective and impactful. Although 37% of MCC’s evaluations use random-assignment methods, all of MCC’s evaluations – both impact and performance – use rigorous methods to achieve the three-part objectives of accountability, learning, and results in the most cost-effective way possible. Of particular interest in the innovation space in FY20, MCC conducted its first impact evaluation of an institutional reform program with the publication of the evaluation of the Indonesia Procurement Modernization Project. This project was specifically designed as a pilot with the evaluation results being used to determine further scale. MCC also published an evaluation of another pilot effort in Namibia that sought to improve community-based rangeland and livestock management. MCC took a comprehensive approach to measuring the various aspects of the program logic, including direct measurement of livestock (weighing and aging cows), direct measurement of rangeland health (measured grass height), and direct observation to verify self-reported behaviors. The resulting learning is extremely nuanced, which has proved especially useful to MCC and Namibian stakeholders since the intervention was advertised as a pilot.
Score
3
Substance Abuse and Mental Health Services Administration
7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?
  • SAMHSA participates in collaborations with other HHS agencies to promote innovative uses of data, technology and innovation across HHS to create a more effective government and improve the health of the nation, via the HHS IDEA Lab. SAMHSA has co-developed and submitted several innovative data utilization project proposals to the Ignite Accelerator of the HHS IDEA Lab, such as Rapid Opioid Alert and Response (ROAR), a project to monitor and prevent opioid overdoses by linking heroin users to resources and information.
7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?
  • Pursuant to the 21st Century Cures Act, SAMHSA established the National Mental Health and Substance Use Policy Laboratory (NMHSUPL) as an office, led by a Director. The NMHSUPL promotes evidence-based practices and service delivery models through evaluating models that would benefit from further development and through expanding, replicating, or scaling evidence-based programs across a wider area. Specifically, according to the SAMHSA website, NMHSUPL:
    • Identifies, coordinates, and facilitates the implementation of policy changes likely to have a significant effect on mental health, mental illness (especially severe mental illnesses such as schizophrenia and schizoaffective disorders), recovery supports, and the prevention and treatment of substance use disorder services;
    • Works with CBHSQ to collect information from grantees under programs operated by the Administration in order to evaluate and disseminate information on evidence-based practices, including culturally and linguistically appropriate services, as appropriate, and service delivery models; and
  • Carries out other activities as deemed necessary to continue to encourage innovation and disseminate evidence-based programs and practices.
  • The SAMHSA Program Portal, a collection of technical assistance and training resources provided by the agency, provides behavioral health professionals with education and collaboration opportunities, and ample tools and technical assistance resources that promote innovation in practice and program improvement. Located within the Knowledge Network are groups such as the Center for Financing Reform and Innovation, which works with states and territories, local policy makers, providers, consumers, and other stakeholders to promote innovative financing and delivery system reforms.
7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?
  • SAMHSA does not list any completed evaluation reports on its evaluation website. Of the nine evaluation reports found on the publications page, none appear to use experimental methods.
Back to the Standard

Visit Results4America.org

Copyright 2024 Results for America. All rights reserved. Privacy Policy

Site by Wide Eye Creative