Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY20?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The General Deputy Assistant Secretary of the Office of Policy Development & Research (PD&R) serves as the Department of Housing and Urban Development (HUD) evaluation officer. HUD’s Office of Policy Development & Research (PD&R) is led by an Assistant Secretary and the career General Deputy Assistant Secretary. PD&R comprises six offices, 153 staff including a team of field economists in HUD’s 10 regional offices, and a budget of $98 million in FY20. The Assistant Secretary and Evaluation Officer ensures that evidence informs policy development through frequent personal engagement with other principal staff, the Secretary, and external policy officials including consultation with Congress, speeches to policy audiences, sponsorship of public research briefings, and policy implications memoranda.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• The Chief Technology Officer in the Office of the Chief Information Officer serves as the acting Chief Data Officer for HUD. The FY21 Budget requested funding to stand up the CDO’s office with 13 staff. The PD&R General Deputy Assistant Secretary and Statistical Official are responsible for numerous data infrastructure functions such as the collection and analysis of national housing market data (including survey collaborations with the Census Bureau); developing income limits and factors to support program operations; advising and assisting program offices with the development and analysis of administrative data collections; and supporting data linkages and developing open data products from administrative data, including geospatial data products that are crucial for addressing housing and urban development policy challenges.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, and other related officials in order to inform policy decisions and evaluate the agency’s major programs?

• HUD has engaged and coordinated within the Department its evidence-building efforts, which in FY20 included developing HUD’s learning agenda and conducting the first agency-wide assessment of evidence-building capacity. In FY21, HUD will be focused on establishing an enterprise data governance model, which will include a data governance board consisting of key decision-makers from across the agency, which will include the Evaluation Officer, Chief Data Officer, Statistical Official, and Performance Improvement Officer. HUD’s enterprise data governance model will bring together evaluation, statistical, performance, and data activities and focus on growing the agency’s evidence-based practices in order to improve HUD’s organizational performance.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY20?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• PD&R has published a Program Evaluation Policy that establishes core principles and practices of PD&R’s evaluation and research activities. The six core principles are rigor, relevance, transparency, independence, ethics, and technical innovation.

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• HUD’s learning agendas, called the Research Roadmap, have served as agency-wide evaluation plans that list and describe research and evaluation priorities for a five-year planning period. Annual evaluation plans are developed based on a selection of Roadmap proposals, newly emerging research needs, and incremental funding needs for major ongoing research and are submitted to Congress in association with PD&R’s annual budget requests. Actual research activities are substantially determined by Congressional funding and guidance. Under the Evidence Act, PD&R will prepare public Annual Evaluation Plans informed by the new Research Roadmap to be submitted in conjunction with the Annual Performance Plan.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• HUD’s Research Roadmap has served as the Department’s evidence-building plan and learning agenda for eight years, and a new Roadmap was developed in FY19-20. HUD’s participatory process (see for example pp. 14–16 of Roadmap Update 2017) engages internal and external stakeholders to identify research questions and other evidence-building activities to support effective policy-making. Stakeholders include program partners in state and local governments and the private sector; researchers and academics; policy officials; and members of the general public who frequently access the HUDuser.gov portal. Outreach mechanisms for learning agenda development include email, web forums, conferences and webcasts, and targeted listening sessions. The 2019 roadmapping process added a new public-access conference and webcast. The updated Roadmap provides critical content for developing a learning agenda under the Evidence Act as a component of the next Strategic Plan.

2.4 Did the agency publicly release all completed program evaluations?

• PD&R’s Program Evaluation Policy requires timely publishing and dissemination of all evaluations that meet standards of methodological rigor. Completed evaluations and research reports are posted on PD&R’s website, HUDUSER.gov. Additionally, the policy includes language in research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than six months.

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))

• PD&R is HUD’s independent evaluation office, with scope spanning all the Department’s program operations. In FY20 PD&R is leading the effort to assess the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts, consistent with the values established in HUD’s Evaluation Policy. The forthcoming Research Roadmap covers much of this content, and a formal Capacity Assessment process was designed by evaluation leaders in coordination with the Chief Data Officer and performance management personnel. The initial Capacity Assessment addresses updated content requirements of OMB Circular A-11 (2020) and includes primary data collection through an exploratory key informant survey of senior managers across the Department. The identified weaknesses in evidence-building capacity will become the focus of subsequent in-depth assessments and interventions to be integrated in the Department’s next Strategic Plan.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• For decades, PD&R has been a federal leader in the use of random assignment and other rigorous methods for research and evaluation purposes. Examples of random-assignment program demonstrations found on HUDUSER.gov include landmark research in the Housing Allowance experiment, the Moving to Opportunity Demonstration, and the Family Options Demonstration. Ongoing random-assignment experiments include the Moving to Work Demonstration, Family Self-Sufficiency Demonstration, the First-Time Homebuyer Education and Counseling demonstration, the Rent Reform Demonstration, and the Integrated Wellness in Supportive Housing Demonstration (See the PD&R Biennial Report FY 2017–2018).

Did the agency invest at least 1% of program funds in evaluations in FY20? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, rigorous evaluations, including random assignments)

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY20 budget.

• HUD invested $98 million on evaluations, evaluation technical assistance, and evaluation capacity-building, representing 0.18% of the agency’s $54.195 billion FY20 appropriation.

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• For FY20, Congress appropriated $98 million for the Office of Policy Development and Research’s Research & Technology account. FY20 funding was up $2 million from FY19, reflecting congressional support for the value of PD&R’s research, evaluations, and demonstrations. This budget includes $54 million for core research activities; $14 million for research, evaluations, and demonstrations; and $30 million for technical assistance. The total represents an FY20 investment in evaluations and evidence amounting to 0.18 percent of HUD’s $54.195 billion gross discretionary budget authority, net of salaries and expenses, for FY20. The funding for core research is used primarily for the American Housing Survey, other surveys, data acquisition, and research dissemination that support evaluation of HUD’s mission activities in domains such as affordable housing and housing finance.
• PD&R’s FY20 appropriation of $28 million for Salaries and Expenses, up $2 million from FY19, also supports evidence in the form of PD&R’s in-house research and evaluation program; economic analyses; data linkage initiatives; and management of housing surveys, contract research, and evaluation.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• For FY20, HUD is providing $91 million of technical assistance to equip the Department’s program partners with the knowledge, skills, tools, capacity, and systems to implement HUD programs and policies successfully and to provide effective oversight of federal funding. State and local governments and authorities are among the eligible applicants, with approximately 23 awards expected. Community Compass integrates technical assistance funding from four major HUD program areas to better reflect the cross-cutting nature of housing and community development challenges. Eligible technical assistance activities include training and tool development to help program partners improve program management, evaluation, and performance measurement, and the Community Compass program itself has an increased evidence-based focus for FY20.
• In FY20, HUD is offering a new $3 million Distressed Cities technical assistance program that helps cities that recently experienced disasters build capacity for processes including data collection, analysis, and tracking outcomes. HUD operates a Section 4 Capacity Building grant program that funds national intermediaries and rural jurisdictions in building capacity for functions including assessing needs, planning programs, and evaluation.
• HUD’s Community Development Block Grant (CDBG) program, which provides formula grants to entitlement jurisdictions, increases local evaluation capacity. Specifically, federal regulations (24 CFR570.200) authorize CDBG recipients (including city and state governments) to use up to 20% of their CDBG allocations for administration and planning costs that may include evaluation-capacity building efforts and evaluations of their CDBG-funded interventions (as defined in 570.205 and 570.206).

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY20?
(Example: Performance stat systems, frequent outcomes-focused data-informed meetings)

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• HUD’s FY 2018–2022 Strategic Plan, as amended by HUD’s FY 2021 Annual Performance Plan, defines strategic objectives, priority outcome goals, and program metrics supporting each objective. Progress on program metrics is tracked through the Annual Performance Plan.

4.2 Does the agency use data/evidence to improve outcomes and return on investment?

• HUD uses data and evidence extensively to improve outcomes and return on investment. The primary means are through PD&R’s investments in data collection, program demonstrations and evaluations, and research guided by a multi-year learning agenda; HUD’s extensive use of outcome-oriented performance metrics in the Annual Performance Plan; and senior staff oversight and monitoring of key outcomes and initiatives through the Prescription for HUD, the Advancing Economic Opportunity Task Force, and the Agency-Wide Integrity Task Force, which bring together senior staff for quarterly performance management meetings.
• In 2019, HUD expanded the Standards for Success data collection and reporting framework for discretionary grant programs to cover Resident Opportunities and Self-Sufficiency Service Coordinator (ROSS) grants, Multifamily Housing Service Coordinator grants, and Multifamily Housing Budget-Based Service Coordinator Sites. The framework supports better outcomes by providing a more standardized performance measurement framework, better alignment with Departmental strategies, and more granular reporting to support analytics.

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• HUD’s senior staff support continuous improvement and oversight and monitoring of key outcomes and initiatives through the Prescription for HUD, the regular meetings of the Advancing Economic Opportunity Task Force, and the Agency-Wide Integrity Task Force. These processes are supported by ongoing, significant investments in evidence-building as documented in the Annual Performance Plan and the iterative process of developing the Research Roadmap learning agenda. Monitoring and analysis based on administrative data have a symbiotic and complementary relationship with structured evaluation and program demonstrations.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY20?
(Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies; data-use policies)

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• In FY21, HUD will develop a strategic data plan, which will include an open data policy. Currently, HUD’s open data program includes existing assets including administrative datasets on data.hud.gov, spatially enabled data on the eGIS portal, PD&R datasets for researchers and practitioners, a robust partnership with the Census Bureau, U.S. Postal Service vacancy data, and health data linkages with the National Center for Health Statistics.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• In FY21, HUD will review its existing data inventory and update it accordingly to produce a comprehensive data inventory. HUD will also revisit its data inventory schedule to ensure the agency is performing the activities necessary to develop and maintain a comprehensive data inventory.

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c)) 

• HUD has extensively promoted data access and data linkage, including the following approaches:
  • An updated list of open data assets; numerous PD&R-produced datasets for researchers and practitioners, including tenant public use microdata samples; and an eGIS portal providing geo-identified open data to support public analysis of housing and community development issues using GIS tools.
  • Data linkage agreements with the National Center for Health Statistics and the Census Bureau to enhance major national survey datasets by identifying HUD-assisted households, with updates continuing in FY20; making available major program demonstration datasets in secure environments; and producing special open-access tabulations of census data for HUD’s partners.
  • Engagement in cooperative agreements with research organizations, including both funded Research Partnerships and unfunded Data License Agreements, to support innovative research that leverages HUD’s data assets and informs HUD’s policies and programs. Data licensing protocols ensure that confidential information is protected.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)

• HUD’s Statistical Official supports the Evidence Officer on issues related to protection of confidential data and statistical efficiency. HUD’s Evaluation Policy specifies that HUD protects client privacy by adhering to the Rule of Eleven to prevent disclosure from tabulations with small cell sizes. PD&R’s data licensing protocols ensure that researchers protect confidential information when using HUD’s administrative data or program demonstration datasets.
• The Statistical Official collaborates with statistical agencies to create data linkages and develop data products that are machine-readable and include robust privacy protections. HUD has an interagency agreement with the Census Bureau to conduct the American Housing Survey and collaborates with Census staff to examine disclosure issues for AHS public use files and the potential for “synthetic” public datasets to support researchers in estimating summary statistics with no possibility of reidentifying survey respondents. Another interagency agreement allows the Census Bureau to link data from HUD’s randomized control trials with other administrative data collected under the privacy protections of its Title 13 authority. These RCT datasets are the first intervention data added to Federal Statistical Research Data Centers (RDCs) by any federal agency. Strict RDC protocols and review of all output ensure that confidential information is protected, and the open data and joint support for researchers are currently facilitating seven innovative research projects at minimal cost to HUD.
  

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• HUD has an updated list of open data assets, an open data program, numerous PD&R datasets for researchers and practitioners, and an eGIS portal providing geo-identified data to support public analysis of housing and community development issues related to multiple programs and policy domains using GIS tools. These accessible data assets have privacy protections. Researchers needing detailed microdata can obtain access through data licensing agreements.
• HUDExchange offers numerous resources and training opportunities to help program partners use data assets more effectively. Additional technical assistance is offered through the  program, a $91 million technical assistanceprogram to equip HUD’s customers with the knowledge, skills, tools, capacity, and systems to implement HUD programs and policies successfully and provide effective oversight of federal funding.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY20?
(Example: What Works Clearinghouses)

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• PD&R’s Program Evaluation Policy defines standards that prioritize rigorous methods for research and evaluation covering impact evaluations; implementation of process evaluations; descriptive studies; outcome evaluations; and formative evaluations; and both qualitative and quantitative approaches. It also provides for dissemination of such evidence to stakeholders in a timely fashion.

6.2 Did the agency have a common evidence framework for funding decisions?

• HUD seeks to employ tiered evidence in funding decisions by embedding implementation and impact evaluations in funding requests for program initiatives, including major program demonstrations that employ random assignment methods. These include the Moving To Work Expansion demonstration, the Rental Assistance Demonstration, the Rent Reform Demonstration, the Family Self-Sufficiency Demonstration, the Housing Counseling Demonstration, and the Family Options Demonstration. Such trials provide robust evidence to inform scale-up funding decisions.
• HUD extended its standardized data collection and reporting framework, Standards for Success, to additionaldiscretionary grant programs in FY19. The framework consists of a repository of data elements that participating programs use in their grant reporting, creating common definitions and measures across programs for greater analysis and coordination of services.

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• HUD provides resources and assistance to support community partners in evidence-based practice through the HUD Exchange web portal and through Community Compass technical assistance. PD&R provides the public, policymakers, and practitioners with evidence of what works through the Regulatory Barriers Clearinghouse and HUD USER, which is a portal and web store for program evaluations, case studies, and policy analysis and research. The evaluations of major program demonstrations provide rigorous evidence about effect sizes and variations in effects between key subgroups.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• HUD provides resources and assistance to support community partners in evidence-based practice through the HUD Exchange web portal and through technical assistance. PD&R provides the public, policymakers, and practitioners with evidence of what works primarily through HUD USER, a portal and web store for program evaluations, case studies, and policy analysis and research; the Regulatory Barriers Clearinghouse; and through initiatives such as Innovation of the Day, Sustainable Construction Methods in Indian Country, and the Consumer’s Guide to Energy-Efficient and Healthy Homes. This content is designed to provide current policy information, elevate effective practices, and synthesize data and other evidence in accessible formats such as Evidence Matters. Through these resources, researchers and practitioners can see the full breadth of work on a given topic (e.g., rigorous established evidence, case studies of what has worked in the field, and new innovations currently being explored) to inform their work.
• Community Compass technical assistance for urban, rural, and tribal partners is designed to facilitate understanding of community and housing development issues in a way that cuts across program silos. It supportsthem in evaluation, evidence-building, integrating knowledge management principles, and sharing practices.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY20?
(Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements)

7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?

• HUD extensively engages leadership in innovation efforts through two task forces of principal staff focusing on Advancing Economic Opportunity—with an emphasis on measuring and improving outcomes for assisted tenants—and on the Finance Transformation Initiative, a $20 million program to strengthen people, process, and technology to remediate material weaknesses, strengthen financial management and internal controls, and implement best practices.
• HUD has an Office of Innovation led by a Deputy Assistant Secretary that organized the five-day Innovative Housing Showcase on the national mall with federal and private sector partners in June 2019 to demonstrate new housing technology and discuss innovation barriers and opportunities. The entire local HUD staff was encouraged to attend and view the innovative technologies, and another Showcase is planned for 2021. The Office of Innovation is developing prize competitions to stimulate innovation in housing and HUD policy and programs.
• HUD administers five types of juried Secretary’s Awards to encourage excellence in addressing housing and community development challenges: Public-Philanthropic Partnerships, Opportunity and Empowerment, Healthy Homes, Historic Preservation, and Housing and Community Design. An Innovation in Affordable Housing Competition engages multidisciplinary teams of graduate students in addressing a specific housing problem developed by an actual public housing agency.

7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?

• The Department promotes evidence-based innovation by using program demonstrations to experimentally test potential policy enhancements, which have included eight low-cost, behaviorally informed experiments using interagency data matching and assistance from the GSA Office of Evaluation Sciences. Other innovative research ideas from external stakeholders are supported by the Research Partnerships program. Competitive awards for Healthy Homes Technical Studies generate innovation in the evaluation and control of housing-related health and safety hazards.
• An interagency agreement with the Census Bureau has made datasets from HUD’s randomized control trials available for linkage with census data and administrative datasets. The RCT datasets are the first intervention data added to Federal Statistical Research Data Centers (RDCs) by any federal agency, and joint support is available to help researchers gain access and learn to use the restricted data successfully for innovative research, with seven projects currently underway.
• HUD’s Rental Assistance Demonstration, which restructures the financing of the nation’s public housing to address capital needs backlogs, has the additional innovative feature of providing tenants with a Choice Mobility option. Choice Mobility supports self-sufficiency by offering priority receipt of a Housing Choice Voucher providing freedom to move to neighborhoods with greater economic opportunities or better schools and amenities.
• HUD established the Office of Innovation in 2019 to advance innovation in several domains. The office managed the 2019 Innovative Housing Showcase and is developing a similar Showcase for 2021 and prize competitions to stimulate innovation in housing and HUD policy and programs. FY20 grants fund cooperative agreements for pre-competitive research in homebuilding innovations, with a similar program for Historically Black Colleges and Universities, for more affordable, energy efficient, resilient, and healthier housing
• HUD’s regulation of manufactured housing production is guided by a federal advisory committee, the Manufactured Housing Consensus Committee, to provide increased ability for the industry to produce some of the nation’s most innovative, safe, and affordable housing.
• HUD has a Robotics Process Automation initiative devoted to freeing the workforce from low-value, repetitive work through software robotics solutions. Specialized computer programs known as “bots” automate and standardize repeatable business processes without costly investments in conventional automation. Planned efforts involving payroll, accounts receivable and payable, invoice processing, inventory management, report creation, and data migration have potential to shift over 50,000 hours of employee time from low-value to high-value work.

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• PD&R is conducting numerous random-assignment program demonstrations to test new, innovative program models, as described in PD&R’s biennial report and online: the Family Self-Sufficiency Demonstration, First-Time Homebuyer Education and Counseling Demonstration, Pre-Purchase Homeownership Counseling Demonstration, Support and Services at Home (SASH) Demonstration for elderly households, Supportive Services Demonstrationfor health services in elderly housing, Rent Reform Demonstration, Rental Assistance Demonstration, and the Small Area Fair Market Rent Demonstration.
• HUD also is using random assignment and administrative data linkages to test the impact of education navigatorson rates of application for federal student aid by young residents of public housing.
• The Secretary’s Awards competitions use expert juries who assess quantitative and qualitative information submitted by applicants to identify particularly creative solutions to challenging problems.
• HUD’s Office of Innovation is advancing innovation in several domains. The office managed the 2019 Innovative Housing Showcase and is developing FY20 prize competitions, supported by evaluation, to stimulate innovation in housing and HUD policy and programs.
• In 2019, PD&R published an independent review of building technology innovation policies, programs, and strategies to increase the impact of federal research and development investments.

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY20? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring for evidence; Pay for Success provisions)

8.1 What were the agency’s five  largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY20, HUD’s five largest competitive grant programs are:
  1. Continuum of Care ($2.35 billion; eligible grantees: state and local governments and coalitions)
  2. Lead-Hazard Reduction ($275 million; eligible grantees: local governments)
  3. Choice Neighborhoods Implementation ($182 million; eligible grantees: state and local governments)
  4. Section 202 Service Coordinators ($100 million; eligible grantees: service coordinators/housing providers)
  5. Indian Housing ($91 million; eligible grantees: tribes and tribally designated housing entities).

8.2 Did the agency use evidence of effectiveness to allocate funds in its five  largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• The Continuum of Care program (CoC) provides homelessness assistance awards on the basis of system performance measures focused on outcomes and evidence of effectiveness. This includes up to 56 points (out of 200) for past “performance related to reducing homelessness” and four points for “reallocat[ing] lower performing projects to create new higher performing projects that are based on performance review of existing projects.” Additionally, a precondition for Continuum of Care applicants to be awarded FY19 expansion bonus funding was that they rank homeless assistance projects on the basis of how they improve system performance (p. 34).
• Lead Hazard Reduction Grants require applicants to demonstrate a strategic approach to address low-income neighborhoods having concentrated lead hazards for children. The FY20 grants required the grantees to use evidence-based lead hazard control methods and meet cost-savings, productivity, and grant compliance benchmarks. The application assigned 13 points (out of 100) based on grantees’ past performance. Past research showing large returns on investment supported HUD’s decision to request a 26 percent increase in program funding for FY20, and HUD is funding studies using an implementation science framework to continue improving efficiency and efficacy of lead interventions.
• The Indian Housing competitive grant program was established to address issues of overcrowded and physically inadequate housing identified by a PD&R needs assessment completed in 2017, Housing Needs of American Indians and Alaska Natives in Tribal Areas.

8.3 Did the agency use its five largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• As a condition of grant award, all HUD competitive grantees are required to cooperate (p. 5) in any HUD-sponsored research or evaluation studies.
• The Continuum of Care program is supported by the National Homeless Data Analysis Project, which provides communities with resources to improve data collection and consistent reporting about individuals experiencing homelessness to support national Annual Homeless Assessment Reports.
• HUD Lead Paint grantees are required to integrate evidence into their work by conducting clearance testing of all housing units treated. Technical studies provide evidence to improve lead hazard detection, evaluation, and control technologies, as well as implementation, and rigorous evaluation has demonstrated the large return on investment related to children’s health from controlling lead hazards.
• All HUD-funded programs require recipients to submit, not less than annually, a report documenting achievement of outcomes under the purpose of the program and the work plan in the award agreement for accountability purposes and to build evidence of effective practices in the field.

8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant programs in FY20 (besides its five largest grant programs)?

• HUD’s Housing Counseling Grant Program ($43 million in FY19) provides counseling services to tenants and homeowners. One of the program’s main objectives is to “Distribute federal financial support to housing counseling agencies based on past performance.” As such, the program allocates seven points (out of 100) for past performance based on the “the positive impacts that an Applicant’s housing counseling services had on clients.” HUD scores this item based on its own performance records.
• HUD continues to extend the Standards for Success reporting framework to additional competitive grant programs, establishing a performance outcomes framework that will both drive performance and determine future funding recipients by providing strategically aligned performance metrics that are standardized and sufficiently granular to provide information on relative effectiveness.

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Continuum of Care programs are the nation’s primary structure for assisting people experiencing homelessness. Over more than a decade, increased CoC effectiveness has been supported by Homeless Management Information Systems and evidence-based funding of increased permanent supportive housing. As a result, the estimated numberof chronically homeless individuals declined 27 percent between 2010 and 2016; subsequent increases in unsheltered chronically homeless individuals, however, motivated increases in emergency shelter beds. Following federal criteria, 78 communities and 3 states have effectively ended veteran homelessness.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• HUD operates a centralized evaluation program under the guidance of the evaluation officer. As a condition of grant award, all HUD competitive grantees are required to cooperate in any HUD-sponsored research or evaluation studies and to provide program monitoring data. A number of program statutes do not authorize formal evaluation as an eligible activity for use of program funds. HUD also provides technical assistance to strengthen grantees’ capacity for evaluation and performance management capacity.
• The Continuum of Care FY19 homelessness assistance program NOFA offers one point for applicants who propose to use requested funds to improve their ability to evaluate the outcome of projects funded by the CoC Program and the Emergency Solutions Grant program (p. 39).

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY20?
(Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

9.1 What were the agency’s five largest non-competitive programs and their appropriation amounts (and were city, county, and/or state governments are eligible to receive funds from these programs)?

• In FY20, the five largest non-competitive grant programs are:
  1) Public Housing Operating Fund ($4.55 billion; eligible applicants: Public housing authorities);2) Public Housing Capital Grants ($2.87 billion; Public housing authorities);3) Housing Choice Voucher (HCV) Administrative Fees ($1.98 billion; eligible applicants: Public housing agencies that administer Housing Choice Vouchers);4) Community Development Block Grant Entitlement/Non-Entitlement ($3.43 billion; eligible applicants: entitlement cities and counties and state allocation agencies);5) HOME Investment Partnerships ($1.35 billion; eligible applicants: participating jurisdictions).

9.2 Did the agency use evidence of effectiveness to allocate funds in its five largest non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)

• Although the funding formulas are prescribed in statute, evaluation-based interventions are central to each program. HUD used evidence from a 2015 Administrative Fee study of the costs that high-performing PHAs incur in administering a HCV program to propose a new FY17 approach for funding Administrative Fees while strengthening PHA incentives to improve HCV outcomes by providing tenant mobility counseling.
• HUD’s funding of public housing is being radically shifted through the evidence-based Rental Assistance Demonstration (RAD), which enables accessing private capital to address the $26 billion backlog of capital needs funding. Based on demonstrated success of RAD, for FY20 HUD proposed to transfer $95 million from the Operating Fund and Capital Fund to the Tenant-Based Rental Assistance fund to support RAD conversions. For FY21 HUD is proposing to remove the cap on the number of public housing developments to be converted to Section 8 contracts. HUD is beginning to evaluate RAD’s impacts on children. HUD is also conducting a Rent Reform demonstration and a Moving To Work (MTW) demonstration to test efficiencies of changing rent rules and effects on tenant outcomes.

9.3 Did the agency use its five largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• Evidence-building is central to HUD’s funding approach through the use of prospective program demonstrations. These include the Public Housing Operating Fund’s Rental Assistance Demonstration (RAD), the Public Housing Capital Grants’ Rent Reform demonstration, and the Housing Choice Voucher program’s Moving To Work (MTW) demonstration grants. As Congress moved to expand MTW flexibilities to additional public housing authorities (PHAs), HUD sought authority to randomly assign cohorts of PHAs to provide ability to rigorously test specific program innovations.
• Program funds are provided to operate demonstrations through the HCV account, Tenant-Based Rental Assistance. These include the Tribal HUD-VA Supportive Housing (Tribal HUD-VASH) demonstration of providing permanent supportive housing to Native American veterans and the FSS-Family Unification Program demonstration that tests the effect of providing vouchers to at-risk young adults who are aging out of foster care.

9.4 Did the agency use evidence of effectiveness to allocate funds in any other non-competitive grant programs in FY20 (besides its five largest)?

• HUD-Veterans Affairs Supportive Housing (HUD-VASH) vouchers are allocated in part on the administrative performance of housing agencies as measured by their past utilization of HUD-VASH vouchers in HUD’s Voucher Management System (Notice PIH-2019-15 (HA)). The performance information helps ensure that eligible recipients are actually able to lease units with the vouchers that HUD funds. The HUD-VASH Exit Study documented that 87,864 VASH vouchers were in circulation in April 2017, contributing substantially to the 47-percent decline in the number of homeless Veterans since 2010.
  

9.5 What are the agency’s 1-2 strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• To address a severe backlog of capital needs funding for the nation’s public housing stock, the Rental Assistance Demonstration was authorized in 2011 to convert the properties to project-based Section 8 contracts to attract an infusion of private capital. The 2019 final report on the RAD evaluation showed that conversions successfully raised $12.6 billion of funding, an average of $121,747 per unit to improve physical quality and stabilize project finances. Based on the program’s successes, the limit on the number of public housing conversions was increased to 455,000 units in 2018, nearly half of the stock, and HUD has been proposing to eliminate the cap. Additionally, HUD extended the conversion opportunity to legacy multifamily programs through RAD 2.

9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• Communities receiving HUD block grant funding through Community Development Block Grants, HOME block grants, and other programs are required to consult local stakeholders, conduct housing needs assessments, and develop needs-driven Consolidated Plans to guide their activities. They then provide Consolidated Annual Performance and Evaluation Reports (CAPERs) to document progress toward their Consolidated Plan goals in a way that supports continued community involvement in evaluating program efforts.
• HUD’s Community Development Block Grant program, which provides formula grants to entitlement jurisdictions, increases local evaluation capacity. Specifically, federal regulations (Section 24 CFR 570.200) authorize CDBG recipients (including city and state governments) to use up to 20% of their CDBG allocations for administration and planning costs that may include evaluation-capacity building efforts and evaluations of their CDBG-funded interventions (as defined in 570.205 and 570.206).

In FY20, did the agency shift funds away from or within any practice, policy, or program that consistently failed to achieve desired outcomes?
(Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; incentivizing or urging grant applicants to stop using ineffective practices in funding announcements; proposing the elimination of ineffective programs through annual budget requests; incentivizing well-designed trials to fill specific knowledge gaps; supporting low-performing grantees through mentoring, improvement plans, and other forms of assistance; using rigorous evaluation results to shift funds away from a program)

10.1 Did the agency have policy(ies) for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• The Evaluation of the Housing First model of rehousing chronically homeless individuals with serious mental illness supported a policy shift toward first achieving housing stability to provide a platform for social services. Based on such evidence, HUD continues to encourage the use of more cost-effective rapid rehousing approaches combined with increased permanent supportive housing that is integrated with mainstream services provided by HHS, VA, and others. Additionally, a precondition for Continuum of Care applicants to be awarded FY19 expansion bonus funding was that they rank homeless assistance projects on the basis of how they improve system performance.
• CDBG-DR (Disaster Recovery) is a large and growing program funded by emergency appropriations outside of HUD’s regular budgeting process. In FY18, HUD started promoting mitigation activities for disaster-prone communities, allocating $16 billion of the $28 billion in emergency disaster recovery funds for disaster mitigation in previously disaster-stricken communities. This policy shift was informed by evidence that vulnerability of communities to disasters is increasing even as frequency and severity of severe weather events might also be increasing, such that the National Institute of Building Sciences estimated that society saves $4 in future losses for every dollar spent on mitigation. HUD also drew on the evidence of mitigation pilots through the Hurricane Sandy Rebuild by Design competition and the National Disaster Resilience Competition. By investing in mitigation activities, rather than paying to rebuild existing infrastructure in its previous form, HUD shifted funds in order to help break the cycle of publicly-funded rebuilding and repeated loss.
• HUD grant programs typically provide for recapture of funds that are not committed in a timely fashion, or that remain unexpended after the limits. Effective management by grantees can be especially crucial for timely completion of complex housing development projects, such as with the Capital Fund for public housing and Housing Trust Fund for states. Such funds are reallocated to more effective grantees.
• Preference points used by competitive programs favor grantees that provide evidence of successful outcomes and strategies. The Continuum of Care program awards points that shift funds toward grant applications that have demonstrated better outcomes, that rank and fund better-performing projects, and that take over programs from small and struggling recipients. As noted in the notice of funding: “To encourage CoC mergers and mitigate the potential adverse scoring implications that may occur when a high performing CoC merges with one or more lower performing CoC(s), HUD will award up to 25 bonus points to CoCs that completed a merger…”

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• Through the Community Compass technical assistance program, HUD offers numerous prepared training opportunities as well as in-depth program assistance for grantees or program recipients needing intensive, tailored assistance or long-term capacity-building support to remediate challenges and achieve their potential as HUD program partners.
• HUD has proposed to use Public Housing operating funds set aside for receivership of troubled housing authorities more proactively to address the needs of high-risk PHAs before they go into receivership, including through competitive grants for PHAs that are troubled, substandard, at-risk, or insolvent to help preserve affordable housing for the future. The Real Estate Assessment Center collects extensive data on physical condition, finances, and management to determine PHA status, and field staff have expertise to identify risk factors and useful corrective actions.