Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY20?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The Director of the Office of Learning, Evaluation, and Research (LER) serves as the USAID evaluation officer. In compliance with the Foundations for Evidence-Based Policymaking Act, the Administrator of USAID designated the Agency’s Evaluation Officer (AEO) through an internal Executive Message that was shared with the Agency on June 4, 2019.
• USAID’s AEO works in conjunction with the Office of Learning, Evaluation, and Research (LER) in the Bureau for Policy, Planning, and Learning (PPL) to help the Agency build a body of evidence from which to learn and adapt programs. The LER Director is a senior staff member with the authority, staff, and budget to ensure agency evaluation requirements are met, including that all projects are evaluated at some level, and that decision-making is informed by evaluation and evidence. The LER Director oversaw approximately 25 staff and an estimated $6.6 million budget in FY2019.
• USAID has proposed creating a Bureau for Policy, Resources, and Performance (PRP), which will align policy, resources and evidence-based programming, and elevate the evaluation function by creating an Office for Learning and Evaluation that will manage the Agency’s Evaluation Policy. The office will also create and update the Agency Learning and Evaluation Plans, and commission or conduct cross-cutting evaluations. If approved by Congress, the estimated timeline for establishing the bureau is approximately a year and a half. In the meantime, working groups for each new office are developing work plans and focus areas for the new bureau to ensure PRP will be able to meet its mandate.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• The Agency’s Chief Data Officer (CDO) serves as the USAID Chief Data Officer. The Chief Data Officer reports to the Chief Information Officer in the Bureau for Management. In compliance with the Foundations for Evidence-Based Policymaking Act, the Administrator of USAID re-affirmed the designation of the Chief Data Officer through an internal Executive Message that was shared with the Agency on June 4, 2019. The CDO manages the USAID Data Services team which focuses exclusively on improving the usage of data and information to ensure the Agency’s development outcomes are supported and enhanced by evidence. The CDO’s team includes several direct hire data science and IT professionals along with a budget for contract professionals who provide a comprehensive portfolio of data services in support of the Agency’s mission. The CDO oversaw approximately 80staff and an estimated $11.7 million budget in 2020. The CDO is a senior career civil servant, and the USAID Data Services team is regularly called upon to generate products and services to support the Agency’s highest priorities. USAID also invests in roles including the Chief Innovation Officer, Chief Geographer, Chief Economist, Chief Scientist, and other key roles that drive the use of evidence across the agency.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, and other related officials in order to inform policy decisions and evaluate the agency’s major programs?

• The Agency uses several governance structures and processes currently and will be updating these in accordance with OMB guidance related to the Foundations for Evidence-Based Policymaking Act. Some notable current examples include:
  1. Data Board: In September 2019, USAID established a Data Administration and Technical Advisory (DATA) Board, as mandated by the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) and subsequent guidance from the Office of Management and Budget (OMB) in Memoranda M-19-18 and M-19-23.The DATA Board acts as USAID’s data governance body. It serves as a central venue for seeking input from Agency stakeholders regarding data-related priorities and best practices to support Agency objectives. The DATA Board informs data-related policy, procedures and standards for the Agency. The DATA Board supports the work of the Agency Evaluation Officer by directing data services to facilitate evaluations. In addition to the Agency Evaluation Officer, Chief Data Officer and Statistical Officer, its membership includes the Performance Improvement Officer, the Chief Financial Officer, the Chief Technology officer, the Senior Accountable Official for Privacy and the USAID Geographer as well as representation from across the Agency. The USAID Chief Data Officer, Agency Evaluation Officer, and Statistical Official confer monthly to coordinate policy and activities.
  2. Management Operations Council: USAID also uses a Management Operations Council (MOC) as the platform for Agency leadership to assess progress toward achieving the strategic objectives in USAID’s Strategic Plan and cross-agency priority goals and additional management issues. Established in 2014, the MOC provides Agency-wide leadership for initiatives and investments to reform USAID business systems and operations worldwide. The MOC also provides a platform for senior leaders to learn about and discuss improving organizational performance, efficiency, and effectiveness. The Assistant Administrator for the Bureau for Management and the Agency’s Chief Operating Officer co-chair the MOC. Membership includes, among others, all the Agency’s Chief Executive Officers (e.g., Senior Procurement Executive, Chief Human Capital Officer, Chief Financial Officer, Chief Information Officer, Performance Improvement Officer and Project Management Improvement Officer). Depending on the agenda, it also includes the Chief Data Officer, Agency Evaluation Officer, and the Agency Senior Statistical Official.
  3. Weekly/Monthly Meetings between the Chief Data Officer, Chief Evaluation Officer, and Statistical Official: USAID established a standing meeting between the three officials named in the Evidence Act to coordinate on mandatory actions and milestones, evaluate resource requirements, and reconcile any potential discrepancies. The meeting includes leadership from the Office of Learning, Evaluation and Research which manages Agency requirements on performance monitoring, evaluation and organizational learning. As this meeting pre-dated the first Chief Data Officer council and Chief Evaluation Officer council meetings, it was critical for information sharing and addressing priorities.
  4. Privacy Council Meetings: USAID holds monthly Privacy Council meetings to address necessary actions and raise any privacy and confidentiality concerns. Representation includes the Senior Agency Official for Privacy, the Agency Statistical Official, and the Chief Privacy Officer, among others.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY20?

 2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• The agency-wide USAID Evaluation Policy, published in January 2011 and updated in October 2016, incorporates changes that better integrate with USAID’s Program Cycle Policy and ensure compliance with the Foreign Aid Transparency and Accountability Act (FATAA). The 2016 changes to the evaluation policy updated evaluation requirements to simplify implementation and increase the breadth of evaluation coverage, dissemination, and utilization.

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• USAID has an agency-wide evaluation registry that collects information on all evaluations planned to commence within the next three years (as well as tracking ongoing and completed evaluations). Currently, this information is used internally and is not published. To meet the Evidence Act requirement, USAID will include an agency-wide evaluation plan in the Agency’s draft Annual Performance Plan/Annual Performance Report submitted to OMB in September 2020.
• In addition, USAID’s Office of Learning, Evaluation, and Research works with bureaus to develop internal annual Bureau Monitoring, Evaluation and Learning Plans that review evaluation quality and evidence building and use within each bureau and identify challenges and priorities for the year ahead.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• USAID has an agency-wide learning agenda called the Self-Reliance Learning Agenda (SRLA). The SRLA prioritizes evidence needs related to the Agency’s mission to foster country self-reliance which covers all development program/sector areas, humanitarian assistance and resilience, and agency operations. This vision and mission is articulated in USAID’s Policy Framework and reorients the Agency’s programs, operations, and workforce around the vision of self-reliance or ending the need for foreign assistance.
• USAID used a strongly consultative process for developing SRLA, as described in the SRLA Fact Sheet. First, the Agency compiled learning questions from a number of feedback processes to initially capture 260 questions which through consultations were reduced to the final to thirteen that represent the Agency’s priority learning needs related to Self-Reliance.
• USAID is currently implementing the learning agenda and partnering with internal and external stakeholders to generate and gather evidence and facilitate the utilization of learning. These stakeholders include USAID’s implementing partners, other U.S. agencies, private coalitions and think tanks, researchers and academics, bilateral/multilateral organizations, and local actors and governments in the countries in which it works. Examples of learning products generated to date include a Paper Series on Capacity and Capacity Strengthening; SRLA Review of Selected Evidence.

2.4 Did the agency publicly release all completed program evaluations?

• All final USAID evaluation reports are published on the Development Experience Clearinghouse (DEC), except for a small number of evaluations that receive a waiver to public disclosure (typically less than 5% of the total completed in a fiscal year). The process to seek a waiver to public disclosure is outlined in the document Limitations to Disclosure and Exemptions to Public Dissemination of USAID Evaluation Reports and includes exceptions for circumstances such as those when “public disclosure is likely to jeopardize the personal safety of U.S. personnel or recipients of U.S. resources.”
• To increase awareness of available evaluation reports, USAID has created infographics showing the number and type of evaluations completed in FY2015, FY2016, and FY2017. These include short narratives that describe findings from selected evaluations and how that information informed decision-making. USAID is creating a public dashboard to share evaluation data from FY2016 through the most recent year of reporting. The information for FY2019 is being finalized.

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))

• USAID recognizes that sound development programming relies on strong evidence that enables policymakers and program planners to make decisions, improve practice, and achieve development outcomes.
• USAID has commissioned a Capacity Assessment in response to the Evidence Act requirements. The assessment is using a four-phased approach: assessment design, implementation and analysis, reports, and communication/ dissemination. USAID is currently in Phase 1, which involves developing a Maturity Model to assess the Agency’s capacity to generate, manage, and use evidence.
• USAID staff also review evaluation quality on an ongoing basis and review the internal Bureau Monitoring, Evaluation and Learning Plans referenced in 2.2 above. There are several studies that have looked at parts of this question over the previous several years. These include GAO reports, such as Agencies Can Improve the Quality and Dissemination of Program Evaluations; From Evidence to Learning: Recommendations to Improve Foreign Assistance Evaluations; reviews by independent organizations like the Center for Global Development’s Evaluating Evaluations: Assessing the Quality of Aid Agency Evaluations in Global Health – Working Paper 461; and studies commissioned by USAID such as the Meta-Evaluation of Quality and Coverage of USAID Evaluations 2009 – 2012. These studies generally show that USAID’s evaluation quality is improving over time with room for continued improvement.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• USAID uses rigorous evaluation methods, including random control trials (i.e. assignment studies) and quasi-experimental methods for research and evaluation purposes. For example, in FY2019, USAID completed 12impact evaluations, four of which used random control trials.
• The Development Innovation Ventures (DIV) program makes significant investments using randomized controlled trials and quasi-experimental evaluations to provide evidence of impact for pilot approaches to be considered for scaled funding. USAID is also experimenting with cash benchmarking—using household grants to benchmark traditional programming. USAID has conducted five randomized control trials (RCT) of household grants or “cash lump sum” programs, and three RCTs of more traditional programs with household grant elements.

Did the agency invest at least 1% of program funds in evaluations in FY20? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, rigorous evaluations, including random assignments)

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY20 budget.

• USAID invested at least $201.8 million in FY19 and prior year money on a combination of evaluations completed in FY2019, evaluations that are ongoing during FY2019, evaluation technical assistance, and evaluation capacity-building, representing 01.07% of the agency’s $18.8 billion FY19 budget.

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• In FY19, USAID operating units invested approximately $67.4 million in FY19 and prior year money on 190 evaluations that were completed in that fiscal year. Another 180 evaluations were ongoing in FY2019 (many spanning more than one year in duration) with total ongoing evaluation budgets estimated at $127.8 million. LER’s budget for evaluation technical assistance and evaluation capacity-building in FY19 was $6.6 million (up from $4.6 million in FY18), coming to a total of $201.8 million. This represents 1.07% of the Agency’s $18.8 billion FY19budget.^[1] This total does not include evaluation capacity building done by other Agency offices or other research, studies, analysis or other data collection that is often used for evaluation, such as USAID’s investment in the Demographic Health Survey or some of the assessments done by third-parties across USAID’s innovation portfolio. It also does not include funding by agency sub-components for evaluation technical assistance.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• While specific data on this is limited, USAID estimates that investment in contracts or grants that provide support to build local organizational or governmental capacity in data collection, analysis, and use could be as high as $250 million.
• For example, USAID’s Data for Impact (D4I) activity helps low- and middle-income countries—primarily in sub-Saharan Africa—to increase their capacity to use available data and generate new data to build evidence for improving health programs, health policies, and for decision-making. D4I’s goal is to help low-resource countries gather and use information to strengthen their health policies and programs and improve the health of their citizens.
• In another example, the MEASURE Evaluation project, funded by USAID, has a mandate to strengthen health information systems (HIS) in low-resource settings. The Project enables countries to improve lives by strengthening their capacity to generate and use high-quality health information to make evidence-informed, strategic decisions at local, subregional, and national levels.

^[1] Source for FY2019 Agency budget: FY 2021 Congressional Budget Justification (p. 2). Bilateral Economic Assistance total ($24,500,700,000) minus State’s Global Health Programs ($5,720,000,000) is $18,780,700,000.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY20?
(Example: Performance stat systems, frequent outcomes-focused data-informed meetings)

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• USAID partners with the U.S. Department of State to jointly develop and implement clear strategic goals, strategic objectives, and performance goals, which are articulated in the FY 2018 – 2022 U.S. Department of State – USAID Joint Strategic Plan (JSP). The Agency measures progress towards its own strategic goals, strategic objectives, and performance goals using data from across the Agency, including from annual Performance Plan and Reports (PPRs) completed by operating units, and uses that information to report on performance externally through the Annual Performance Plan/Annual Performance Report (APP/APR) and the Agency Financial Report.
• To aggregate and track performance in key sectors, USAID works with the U.S. Department of State to develop and manage over 100 standard foreign assistance indicators that have common definitions and defined collection methods. Once finalized, USAID publishes illustrative indicator data on a publicly available website known as Dollars to Results. Finally, USAID reports on Agency Priority Goal (APG) and Cross Agency Priority (CAP) goal progress on www.performance.gov.

4.2 Does the agency use data/evidence to improve outcomes and return on investment?

• Most of USAID’s innovation or co-created programs and those done in partnerships reflect a data-driven “pay for results” model, where milestones are agreed by all parties, and payments are made when milestones are achieved. This means that, for some programs, if a milestone is unmet, funds may be re-applied to an innovation or intervention that is achieving results. This rapid and iterative performance model means that USAID more quickly understands what is notworking and can move resources away from it and toward what is working.
• Approaches such as prizes, Grand Challenges, and ventures can also be constructed to be “pay for results only” where interventions such as “Development Impact Bonds” are used to create approaches where USAID only pays for outcomes and not inputs or attempts only. The Agency believes this model will pave the way for much of USAID’s work to be aligned with a “pay for results” approach. USAID is also piloting the use of the impact per dollar of cash transfers as a minimum standard of cost-effectiveness for applicable program designs. Most innovations funded at USAID have a clear “cost per impact” ratio.
• Additionally, USAID Missions develop Country Development Cooperation Strategies (CDCSs) with clear goals and objectives and a Performance Management Plan (PMP) that identifies expected results, performance indicators to measure those results, plans for data collection and analysis, and regular review of performance measures to use data and evidence to adapt programs for improved outcomes. USAID also promotes data-informed operations performance management to ensure that the Agency achieves its development objectives and aligns resources with priorities. USAID uses its Management Operations Council to conduct an annual Strategic Review of progress toward achieving the strategic objectives in the Agency’s strategic plan.
• To improve linkages and break down silos, USAID continues to develop and pilot the Development Information Solution (DIS)—an enterprise-wide management information system that will enable USAID to collect, manage, and visualize performance data across units, along with budget and procurement information, to more efficiently manage and execute programming.  Several USAID field missions are testing the system prior to world-wide deployment: El Salvador, Peru, Rwanda, Ethiopia, South Africa, Vietnam, and Nepal.

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• USAID’s Program Cycle policy (ADS 201.3.2.18) requires that Missions conduct at least one portfolio review per year that focuses on progress toward strategy-level results. Missions must also conduct a CDCS mid-course stocktaking at least once during the course of implementing their Country Development Cooperation Strategy, which typically spans five years.
• USAID developed an approach to explicitly ensure adaptation through learning called Collaborating, Learning, and Adapting (CLA). It is incorporated into USAID’s Program Cycle guidance (ADS 201.3.5.19) where it states: “Strategic collaboration, continuous learning, and adaptive management link together all components of the Program Cycle.” Through CLA, USAID ensures its programming is coordinated with others, grounded in a strong evidence base, and iteratively adapted to remain relative throughout implementation.
• In addition to this focus through its programming, USAID has two senior bodies which oversee Enterprise Risk Management, and meet regularly to improve the accountability and effectiveness of USAID programs and operations through holistic risk management. USAID tracks progress toward strategic goals and annual performance goals during data-driven reviews at Management Operations Council meetings. Also, through input from the Management Operations Council, an annual Agency-wide customer service survey, and other analysis, USAID regularly identifies opportunities for operational improvements at all levels of the Agency as part of its operational learning agenda as well as the agency-wide learning agenda, the Self-Reliance Learning Agenda. SRLA’s questions 8, 9, 12, and 13 focus on operational aspects of the agency’s work which influence everything from internal policy, design and procurement processes, program measurement, and staff training.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY20? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies; data-use policies)

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• USAID’s data related investments and efforts are guided by its Information Technology Strategic Plan. This includes support for the Agency’s Development Data Policy, USAID’s open data policy, that provides a framework for systematically collecting Agency-funded data, structuring the data to ensure usability, and making the data public while ensuring rigorous protections for privacy and security. In addition, this policy sets requirements for how USAID data is tagged, submitted, and updated. The Development Data Library (DDL) is the Agency’s repository of USAID-funded, machine readable data, created or collected by the Agency and its implementing partners. The DDL, as a repository of structured and quantitative data, complements the DEC which publishes qualitative reports and information. USAID also participates and leads in global compilations of data across the industry including the Global Innovation Exchange and in response to COVID-19. USAID also has a variety of stakeholder engagement tools available on USAID’s Development Data Library, including: Open Data Community Questions and video tutorials on using DDL.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• Launched in November 2018 as part of the Development Information Solution (DIS), USAID’s public-facing Development Data Library (DDL) provides a comprehensive inventory of data assets available to the Agency. DDL has posted the Enterprise Data Inventory as a json file since 2015. Following the passage of the Foundations for Evidence-Based Policymaking Act, and in preparation for specific guidance expected in the upcoming release of Phase 2 guidance for the Act, USAID will make any necessary changes to its Comprehensive Data Inventory and continue reporting with quarterly updates as required. The DDL’s data catalog is also harvested via JavaScript on an ongoing basis for further distribution on the federal Data.gov website.

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c))

• The USAID Data Services team— located in USAID’s Management Bureau’s Office of the Chief Information Officer (M/CIO)— manages a comprehensive portfolio of data services in support of the Agency’s mission. This includes enhancing the internal and external availability and ease-of use of USAID data and information via technology platforms such as the AidScape platform broadening global awareness of USAID’s data and information services, and bolstering the Agency’s capacity to use data and information via training and the provision of demand-driven analytical services.
• The Data Services Team also manages and develops the Agency’s digital repositories, including the Development Data Library (DDL), the Agency’s central data repository. USAID and external users can search for and access datasets from completed evaluations and program monitoring by country and sector.
• USAID staff also have access to an internal database of over 100 standard foreign assistance program performance indicators and associated baseline, target, and actual data reported globally each year. This database and reporting process, known as the Performance Plan and Report (PPR) promotes evidence building and informs internal learning and decisions related to policy, strategy, budgets, and programs.
• The United States is a signatory to the International Aid Transparency Initiative (IATI)—a voluntary, multi-stakeholder initiative that created a data standard for publishing foreign assistance spending data in machine-readable format. The standard links an activity’s financial data to its evaluations, and in 2019 the Agency tested publishing results indicators for one country. USAID continues to improve and add to its published IATI data, and is looking into ways to utilize these data as best practice—including using it to populate partner country systems, fulfill transparency reporting as part of the U.S. commitment to the Grand Bargain, and make decisions internally, including based on what other development actors are doing by using the Development Cooperation Landscape tool.
• The Landscape tool enables USAID staff to better understand cooperation partners’ priorities and identify potential areas of alignment. This data source is contributing to more robust cooperation strategy development, decision making, and helping USAID to more effectively and efficiently use cooperation resources. USAID created the Global Innovation Exchange that shares information around development innovations with hundreds of other industry partners and governments.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)

• USAID’s Privacy Program directs policies and practices for protecting personally identifiable information and data, while several policy references (ADS303maz and ADS302mbj) provide guidance for protecting information to ensure the health and safety of implementing partners. USAID’s Development Data Policy (ADS Chapter 579) details a data publication process that provides governance for data access and data release in ways that ensure protections for personal and confidential information. As a reference to the Development Data Policy, ADS579maa explains USAID’s foreign assistance data publications and the protection of any sensitive information prior to release. USAID applies statistical disclosure control on all public data before publication or inclusion in the DDL.

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• While specific data on this is limited, USAID does invest in contracts or grants that provide support to build local organizational or governmental capacity in data collection, analysis, and use. In addition, to date, more than 361USAID data assets are available to the public via USAID’s DDL. These assets include microdata related to USAID’s initiatives that provide partner countries and development partners with insight into emerging trends and opportunities for expanding peace and democracy, reducing food insecurity, and strengthening the capacity to deliver quality educational opportunities for children and youth around the globe. Grantees are encouraged to use the data on the DDL, which provides an extensive User Guide to aid in accessing, using, securing and protecting data. The Data Services team conducts communication and outreach to expand the awareness of websites with development data, how to access it, and how to contact the team for support. In addition, the Data Services team has developed a series of videos to show users how to access the data available. The [email protected] mail account responds to requests for assistance and guidance on a range of data services from both within the Agency and from implementing partners and the public.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY20? (Example: What Works Clearinghouses)

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• USAID is developing an agency-level evidence framework to clarify evidence standards for different decisions, including those related to funding.
• USAID’s evidence standards are embedded within its policies and include requirements for the use of evidence in strategic planning, project design, activity design, program monitoring, and evaluation. USAID has a Scientific Research Policy that sets out quality standards for research across the Agency. USAID’s Program Cycle Policy requires the use of evidence and data to assess the development context, challenges, potential solutions, and opportunities in all of USAID’s country strategies. Specific programs, such as the Development Innovation Ventures (DIV) use evaluation criteria related to evidence of cost effectiveness and ability to scale to determine funding decisions to test and scale innovations. As USAID’s flagship open innovation program, DIV helps to test and scale creative solutions to any global development challenge. By investing in breakthrough proven innovations, driven by rigorous evidence and ongoing monitoring, USAID’s DIV program has proven to impact millions of lives at a fraction of the usual cost.
• GAO found in their December 2019 report Evidence-Based Policymaking: EVIDENCE-BASED POLICYMAKING Selected Agencies Coordinate Activities, but Could Enhance Collaboration that USAID reflects leading practices for collaborating when building and assessing evidence.

6.2 Did the agency have a common evidence framework for funding decisions?

• USAID is developing an agency-level evidence framework to clarify evidence standards for different decisions, including those related to funding. In addition, there are specific types of programs at the sub-agency level that do use evidence framework or standards to make funding decisions.
• Development Innovation Ventures (DIV) uses a tiered funding system to test and scale evidence-based innovations, making funding decisions based on its evaluation criteria: evaluation and impact; cost-effectiveness; evidence and evaluation; implementation; sustainability and pathway to scale; and project team (see page 6 in DIV’s most recent Annual Program Statement for the evaluation criteria). DIV’s expectations vary by stage, but every awardee must report against a set of pre-negotiated key performance indicators and nearly all grants are structured in a pay-for-performance model.
• For large scale Stage 2 DIV grants of $500,000 or more, DIV requires evidence of impact that must be causal and rigorous – the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period. There must be significant demonstrated demand for the innovation.

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions? 

• USAID does have an Agency-wide repository for development information (including evaluation reports and other studies) which is available to the public at the Development Experience Clearinghouse. In addition, USAID uses the International Initiative for Impact Evaluations (3ie) database of impact evaluations relevant to development topics (including over 4,500 entries to date), knowledge gap maps, and systematic reviews that pull the most rigorous evidence and data from across international development donors. 3ie also houses a collection of institutional policies and reports that examine findings from its database of impact evaluations on overarching policy questions to help policymakers and development practitioners improve development impact through better evidence. USAID’s Agency Programs and Functions policy designates technical bureaus responsible for being the repository for latest information in the sectors they oversee; prioritizing evidence needs and taking actions to build evidence; and disseminating that evidence throughout the agency for those sectors. Several USAID bureaus and sectors have created user friendly tools to disseminate information on evidence-based solutions. These include, but are not limited to:
  • CLIMATELINKS: A global knowledge portal for climate change and development practitioners
  • EDUCATIONLINKS: Shares innovations and lessons learned on implementation of the USAID Education Policy
  • Natural Resources Management and Development Portal
  • URBANLINKS: USAID’s sharing platform for resources on sustainable urban development
• Finally, USAID led a data-harmony initiative across the industry with other countries called the Global Innovation Exchange which surfaces, validates, and shares a repository of over 16,000 development relevant solutions across all actors, players and locations.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• USAID’s approach to Collaborating, Learning, and Adapting (CLA) helps ensure that evidence from evaluation of USAID programming is shared with and used by staff, partners, and stakeholders in the field. USAID requires a dissemination plan and post-evaluation action plan for each evaluation, and USAID field staff are encouraged to co-create evaluation action plans with key stakeholders based on evaluation evidence. USAID collects examples through the CLA Case Competition, held annually, which recognizes implementers, stakeholders, and USAID staff for their work generating and sharing technical evidence and learning from monitoring and evaluation. It is another way that the Agency encourages evidence-based practices among its stakeholders.
• USAID also periodically holds large learning events with partners and others in the development community around evidence including, but not limited to, Evaluation Summits, engagement around the Self-Reliance Learning Agenda, and Moving the Needle. These gatherings are designed to build interest in USAID’s evidence, build capacity around applying that evidence and learning, and elicit evidence and learning contributions.
• USAID created and led the “Million Lives Club” coalition, with more than 30 partners, which has identified more than 100 social entrepreneurs who have at least a million customers in order to share the learning that this successful cohort has had and better inform how USAID funding can assist more social entrepreneurs to grow successfully and rapidly. This unique learning platform brings donors, funders, governments, and the entrepreneurial community to the table together to learn and iterate on our approaches.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY20? (Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements) 

7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?

• In FY19, USAID appointed a new Chief Innovation Officer to advocate for innovation throughout development and national security strategies across USAID, the U.S. Government, and the international community. The Chief Innovation Officer promotes opportunities for entrepreneurs, implementing partners, universities, donors, and others to test and scale innovative solutions and approaches to development problems around the world. In FY2019, the U.S. Global Development Lab also engaged USAID leadership and Mission staff from around the world at the Mission Directors Conference, the Contracting Officer and Controller Conference, the Foreign Service National Conference, and the Private Sector Engagement Forum.
• For innovations specific to a particular sector, Agency leadership has supported technical staff in surfacing groundbreaking ideas, such as how the Bureau for Global Health’s Center for Innovation and Impact (CII) used open innovation approaches to issue the Saving Lives at Birth Grand Challenge and identify promising, life-saving maternal and newborn health innovations.

 7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?

• In FY2020, USAID released its first Digital Strategy, moving to a “Digital by Default” position and USAID’s innovative approaches have helped get more than 40 million people in the developing world digital access. USAID’s New Partnerships Initiative (NPI) will allow USAID to work with a more diverse range of partners, strengthen existing partner relationships, and provide more entry points for organizations to work with the Agency. The principles behind NPI are outlined in the Agency’s first-ever Acquisition and Assistance (A&A) Strategy.
• USAID and its partners have launched 41 innovative programming approaches including prizes, ventures, challenges, and Grand Challenges for Development since 2011. Across the Grand Challenges portfolio, partners have jointly committed over $535 million ($155 million from USAID) in grants and technical assistance for over 528 innovators in 107 countries. To date, more than $614 million in follow-on funding has been catalyzed from external sources, a key measure of success.
• USAID’s investment in state-of-the-art geo and information intelligence centers mean that any program has the ability to leverage geospatial analysis and critical data sets to drive innovative solutions based on evidence and data. With over 20 programs experimenting with Artificial Intelligence and machine learning, and USAID’s strong work on Digital finance and connectivity, the Agency is using technology to drive our programs farther, faster. USAID has also completed more than 1,500 Global Development Alliances, leveraging private sector in-kind or financial investments.
• In addition, the Center for Innovation and Impact (CII)—the Bureau for Global Health’s dedicated innovation office—takes a business-minded approach to fast-tracking the development, introduction, and scale-up of health innovations that address the world’s most important health challenges, and assessing and adopting cutting-edge approaches (such as using unmanned aerial vehicles and artificial intelligence).
• Feed the Future Partnering for Innovation partners with agribusinesses to help them commercialize and scale new agricultural innovations to help improve the livelihoods of smallholder farmers, increasing their productivity and incomes. To date the program has worked with 59 partners in 20 different countries, investing more than $43 million in new technologies and services, and leveraging nearly $100 million in private sector investment. The program has helped commercialize over 118 innovations, which resulted in an estimated $99 million in sales. It has its own Innovation site that partners can easily see and connect with promising innovations and research.
• Finally, USAID was honored when the co-founder and Scientific Director of USAID’s Development Innovations Venture (DIV) program, Dr. Michael Kremer received the 2019 Nobel prize for economics, along with Dr. Esther Duflo and Dr. Abhijit Banerjee. Some of his work that led to this honor was connected to USAID’s DIV program. DIV values rigorous testing methods such as impact evaluations or robust market tests to measure the impact of USAID innovations. Evidence of clear and measurable outcomes helps demonstrate what is working and what is not. Solutions that demonstrate rigorous evidence of impact can then be scaled to other contexts. Through the DIV program, Dr. Kremer helps USAID use evidence-driven approaches to take small risks, identify what works, and scale those approaches to provide greater impact, which helps partners on the Journey to Self-Reliance. Since 2010, the DIV program has supported over 200 awards to test and scale development focused innovations that have directly affected more than 30 million lives across 46 countries.

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• Within the U.S. Global Development Lab, the MERLIN program works to innovate on traditional approaches to monitoring, evaluation, research and learning. While innovative in themselves, these approaches can also be better suited to evaluating an innovation effort. Two examples include Developmental Evaluation, which aims to provide ongoing feedback to managers on implementation through an embedded evaluator, and Rapid Feedback, which allows implementers to test various methods to reach certain targeted results (more quickly than through traditional midterm or final evaluations).
• Many of the agency’s programs such as Grand Challenges and Development Innovation Ventures (DIV) have been reviewed by formal audit and other performance and impact interventions. DIV is USAID’s tiered, evidence-driven open innovation program. It awards grants for innovative solutions to any development problem, on the basis of rigorous evidence of impact, cost-effectiveness, and a pathway to scale via the public and/or private sectors. The DIV model is designed to source breakthrough solutions, to minimize risk, and maximize impact by funding according to outcomes and milestones, to rigorously evaluate impact and cost-effectiveness, and to scale proven solutions.
• DIV supports innovative solutions across all countries and development sectors in which USAID operates, including education, agriculture, water, energy, and economic development. Over ten years, since 2010, DIV has invested more than $129 million in over 200 innovations in 46 countries, improving the lives of over 55 million beneficiaries.  As of 2017, DIV funded entities have been able to catalyze the USAID DIV investment through private capital at a leveraged rate of $1.59 for every $1.
• It has generated experimental or quasi-experimental evaluation studies of more than a third of those innovations as well as a forthcoming working paper that rigorously assesses the social rate of return of DIV’s early portfolio. Since DIV reopened in fall 2018, approximately 94 percent of its 2,147 applicants were new to USAID. DIV enhances the Agency’s engagement with non-traditional partners by partnering with innovators ranging from local entrepreneurs to researchers to high-growth start-ups to American small businesses trying to take their innovation global.  In total, 53 percent of organizations supported by DIV have been new to USAID. DIV is an important means for for-profit companies––comprising 43 percent of DIV’s portfolio––to engage with the Agency.  Various USAID Missions and Bureaus, including Egypt, Zambia, Bangladesh, Global Health, and the Women’s Global Development and Prosperity (W-GDP) Initiative, and many more have partnered with and funded innovations identified by DIV.

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY20? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring for evidence; Pay for Success provisions)

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• USAID’s top five program accounts based on actual appropriation amounts in FY19 are:
  1. International Disaster Assistance ($4.39 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303);
  2. Economic Support Fund ($3.69 billion ADS 303);
  3. Migration and Refugee Assistance ($3.43 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303);
  4. Global Health (USAID) ($3.15 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303);
  5. Development Assistance ($3 billion; eligible grantees: any U.S. or non-U.S. organization, individual, nonprofit, or for-profit entity that meets the requirements described in ADS 303).

• See the U.S. Foreign Assistance Reference Guide for more information on each of these accounts. More information can also be found in the FY2021 Congressional Budget Justification (page 2 and 3, column 4). USAID generally does not limit eligibility when awarding grants and cooperative agreements; eligibility may be restricted for an individual notice of funding opportunity in accordance with the procedures in ADS 303.

8.2 Did the agency use evidence of effectiveness to allocate funds in its five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• USAID is committed to using evidence of effectiveness in all of its competitive contracts, cooperative agreements, and grants, which comprise the majority of the Agency’s work. USAID’s Program Cycle Policy ensures evidence from monitoring, evaluation and other sources informs funding decisions at all levels, including during strategic planning, project and activity design, procurement and implementation.
• USAID’s Senior Obligation Alignment Review (SOAR) helps to ensure the Agency is using evidence to design and approve funding for innovative approaches to provide long-term sustainable outcomes and provides oversight on the use of grant or contract mechanisms and proposed results.
• USAID includes past performance to comprise 30% of the non-cost evaluation criteria for contracts. As part of determining grant awards, USAID’s policy requires an applicant to provide a list of all its cost-reimbursement contracts, grants, or cooperative agreements involving similar or related programs during the past three years. The grant Selection Committee chair must validate the applicant’s past performance reference information based on existing evaluations to the maximum extent possible, and make a reasonable, good faith effort to contact all references to verify or corroborate how well an applicant performed.
• For assistance, as required by 2 CFR 200, USAID also does a risk assessment to review an organization’s ability to meet the goals and objectives outlined by the agency. Internal procedures for conducting the risk assessment are found in ADS 303.3.9, with guidance on how to look for evidence of effectiveness from potential grantees. Per the ADS, this can be done through reviewing past performance and evaluation/performance reports such as the Contractor Performance Assessment Reporting System (CPARS).
• Even though there is no federal requirement (as there is with CPARS), USAID also assesses grantee past performance for use when making funding decisions (detailed in ADS 303, p. 66). Per USAID’s ADS 303 policy, before making an award of any grant or cooperative agreement the Agreement Officer must state in the memorandum of negotiation that the applicant has a satisfactory record of performance. When making the award, the Agreement Officer may consider withholding authority to proceed to the next phase of a grant until provided evidence of acceptable performance within a given period.
• USAID was recognized by GAO in its recent report published on September 5, 2018, Managing for Results: Government-wide Actions Needed to Improve Agencies’ Use of Performance Information in Decision Making (GAO-18-609SP) as one of four agencies (out of 23 surveyed) with proven practices for using performance information. USAID was also the only CFO Act agency with a statistically significant increase in the Agency Use of Performance Information Index since 2007.

8.3 Did the agency use its 5 largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations) 

• Grantees report on the progress of activities through documentation such as Activity Monitoring, Evaluation, and Learning (MEL) Plans, periodic performance reporting, and external and internal evaluation reports (if applicable). These reports help USAID remain transparent and accountable and also help the Agency build evidence of what does and does not work in its interventions. Any internal evaluation undertaken by a grantee must also be provided to USAID for learning purposes. All datasets compiled under USAID-funded projects, activities, and evaluations are to be submitted by grantees to the USAID Development Data Library. All final evaluation reports must also be submitted to the Agency’s Development Experience Clearinghouse (DEC), unless they receive a waiver to the USAID’s public dissemination requirements. These are rare and require the concurrence of the Director of the Office of Learning, Evaluation, and Research.

8.4 Did the agency use evidence of effectiveness to allocate funds in any competitive grant program?

• USAID is actively engaged in utilizing evidence of effectiveness to allocate funds. For example, Development Innovation Ventures (DIV) invests in innovations that demonstrate evidence of impact, cost-effectiveness, and a viable pathway to scale. DIV provides four types of grants: 1) proof of concept, 2) positioning for scale, 3) scaling proven solutions, and 4) evidence grants.
• The more funding requested (up to $5 million dollars), the more DIV requires in an innovation’s evidence base, the deeper the due diligence process, and the greater the expectation that the applicant will be able to demonstrate development impact and potential to scale. After a decision is made to allocate funding, 98% of all DIV awards are structured as fixed amount pay-for-performance grants, ensuring that awards maximize the impact of U.S. taxpayer dollars. Over the past eight years, DIV has invested $118 million in nearly 200 innovations across 45 countries.

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?  

• No USAID examples.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• USAID’s Program Cycle Policy states that “[f]unding may be dedicated within a project or activity design for implementing partners to engage in an internal evaluation for institutional learning or accountability purposes.”
• USAID’s Development Innovation Ventures (DIV) specifically references evaluations and rigorous evidence in the official solicitation: “Larger scale Stage 2 innovations (over $500,000) must include or test the evidence of impact of an innovation. This evidence of impact must be causal and rigorous—the grantee must either have rigorous underlying evidence already established, use this funding to run an evaluation with an evaluation partner, or run an evaluation with its own funding during the grant period.” More on DIV’s funding framework can be found in its evaluation criteria (see DIV’s most recent Annual Program Statement for the evaluation criteria (p. 6)). 

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY20? (Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

• USAID does not administer non-competitive grant programs (relative score for criteria #8 applied).

In FY20, did the agency shift funds away from or within any practice, policy, or program that consistently failed to achieve desired outcomes? (Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; incentivizing or urging grant applicants to stop using ineffective practices in funding announcements; proposing the elimination of ineffective programs through annual budget requests; incentivizing well-designed trials to fill specific knowledge gaps; supporting low-performing grantees through mentoring, improvement plans, and other forms of assistance; using rigorous evaluation results to shift funds away from a program)

10.1 Did the agency have policy(ies) for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• USAID shifts funds away from ineffective grantees. For example, the Securing Water for Food Grand Challenge is designed with a Technical Assistance Facility to consult and work with grantees to identify specific growth barriers, and then connect them with vetted service providers that bring expertise and capabilities to help these grantees overcome their strategic barriers. The Technical Assistance Facility provides tailored financial and acceleration support to help these grantees improve their market-driven business development, commercial growth, and scaling.
• If a grantee is unable to meet specific performance targets, such as number of customers or product sales, further funding is not granted, and the grantee is re-categorized into the program’s group of unsuccessful alumni. The Securing Water for Food Grand Challenge used milestone-based grants to terminate 15 awards that were not meeting their annual milestones and shifted that money to both grants and technical assistance for the remaining 25 awards in the program.
• Also, USAID’s INVEST program is designed for constant feedback loops around the partner performance. Not only are under-performing partners dropped, but new partners can be added dynamically, based on demand. This greatly increases USAID’s new partner base and increases the performance standard across the board.
• USAID’s Business Ecosystem Project (BEP), implemented by Palladium Group, is designed to increase private sector investment in strengthening domestic supply chains and workforce development in North Macedonia. BEP’s initial strategy was to mobilize corporate social responsibility (CSR) funds from investors and large international corporations toward the project’s goal, but it quickly became evident that such investments would be neither strategic nor sustainable. To achieve a lasting impact on North Macedonia’s business ecosystem, BEP partnered with companies that were better positioned to recognize the link between local economic development and their own business interest. BEP learned from its local partners and adapted its private sector engagement (PSE) strategy to target small, medium, and large enterprises that were more dependent on domestic supply chains and workers. BEP no longer focuses only on foreign direct investment (FDI) companies with CSR budgets, but approaches all companies that have a real economic incentive to invest in local supply chains and workforce development. This approach was more effective and allowed BEP to co-invest in a diverse range of supply chain and workforce development initiatives, first as a proof of concept and later at scale.

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• USAID/Food for Peace’s Sustainable Action for Resilience and Food Security (Sabal) is a five-year program in Nepal, implemented by Save the Children and a consortium of partners. Sabal’s goal is to improve food security and resilience in targeted districts in Nepal by improving livelihoods, health and nutrition, disaster risk reduction and climate change adaptation. Sabal utilized collaborating, learning and adapting (CLA) approaches such as pause and reflect, M&E for learning, and adaptive management to be able to adapt to the changing context. In 2015, there were devastating earthquakes, which necessitated geographic program expansion and then, two years later, there were budget cuts, which meant ending implementation in those expansion areas. At that time, CLA approaches were utilized to identify sustainability strategies, assess the level of self-reliance among community groups, tailor interventions based on the data, and gain consensus and buy-in among internal staff, consortium partners, and the local government. As a result, Sabal registered high-performing community groups with the government and linked these groups with local resources and leaders. At the same time, Sabal identified poor performing groups and built their capacity and self-reliance through targeted trainings and community capacity building
• USAID’s Regional Health Integration to Enhance Services in Eastern Uganda (RHITES-E) Activity (2016-2021), implemented by IntraHealth International and its partners, supports the Government of Uganda’s health “surge” strategy to find new HIV positive patients and enroll them in care and treatment. The data and results from RHITES-E’s first quarter performance review showed the Activity was way behind its target. The Activity leadership and USAID decided to shift from a “business as usual” attitude to applying collaborating, learning and adapting (CLA) approaches to draw on and analyze existing data, from a USAID dashboard, to reflect on findings with key stakeholders and fill identified needs and gaps to improve surge efforts. By the end of the fiscal year 2017, the Activity had improved its surge performance resulting in better results and outcomes and shifted in its culture to be a learning organization. Together with stakeholders, staff identified ineffective approaches such as mass HIV testing and developed and implemented new strategies to include screening of clients before testing for efficient and effective identification and linkage of new HIV positive clients into care and treatment.
• USAID’s Empleando Futuros (Employing Futures) program, an at-risk youth program was launched in Honduras in 2016. During its first year, a pause and reflect event found a significant number of drop-outs and the need to strengthen the program’s response to better meet the needs of youth and the labor market. USAID and its implementing partner, Banyon Global, applied USAID’s Collaborating, Learning, and Adapting (CLA) Framework and tools to establish a framework for strategic pause and reflect events throughout the year, strengthen the program’s performance monitoring system and develop an online platform for tracking program participants’ progress. These changes helped the implementer to revisit the program’s underlying assumptions and theory of change, learn continuously and inform evidence-based decisions. Preliminary findings suggest that the program has fewer dropouts, capacity of local systems and partners has been strengthened, and private sector engagement has improved.