Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY20?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The director of the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Behavioral Health Statistics and Quality (CBHSQ) Office of Evaluation serves as the agency’s evaluation lead with key evaluation staff housed in this division. According to the SAMHSA website: “The Office of Evaluation is responsible for providing centralized planning and management of program evaluation across SAMHSA in partnership with program originating Centers.” SAMHSA evaluations are funded from program funds that are used for service grants, technical assistance, and for evaluation activities. Evaluations have also been funded from recycled funds from grants or other contract activities, as described in the FY21 Congressional Justification.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• CBHSQ, led by its Director, designs and carries out special data collection and analytic projects to examine issues for SAMHSA and other federal agencies and is the government’s lead agency for behavioral health statistics, as designated by the Office of Management and Budget.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, and other related officials in order to inform policy decisions and evaluate the agency’s major programs?

• The SAMHSA website states: “The Office of Evaluation is responsible for providing centralized planning and management of program evaluation across SAMHSA in partnership with program originating Centers, providing oversight and management of agency quality improvement and performance management activities and for advancing agency goals and objectives related to program evaluation, performance measurement, and quality improvement.” The Evaluation Office describes 10 areas of support it provides to the Centers, including:
  1. Develops evaluation language for Request for Proposals (RFPs), Request for Applications (RFAs), and other funding announcements to ensure a clear statement of evaluation expectations in the announcements;
  2. Develops and implements standard measures for evaluating program performance and improvement of services;
  3. Manages the design of SAMHSA program evaluations in collaboration with the relevant Center(s);
  4. Monitors evaluation contracts to ensure implementation of planned evaluation and provides early feedback regarding program start-up for use in agency decision-making;
  5. Works collaboratively with the National Mental Health and Substance Use Policy Laboratory to provide support for SAMHSA evaluations;
  6. Oversees the identification of a set of performance indicators to monitor each SAMHSA program in collaboration with program staff and the development of periodic program profiles for use in agency planning, program change, and reporting to departmental and external organizations;
  7. Provides collaboration, guidance, and systematic feedback on SAMHSA’s programmatic investments to support the agency’s policy and program decisions;
  8. Analyzes data in support of agency needs and develops evaluation and performance related reports in response to internal and external request;
  9. Utilizes SAMHSA’s Performance Accountability and Reporting System (SPARS) which serves as a mechanism for the collection of performance data from agency grantees; and
  10. Responds to agency and departmental requests for performance measurement data and information; and conducts a range of analytic and support activities to promote the use of performance data and information in the monitoring and management of agency programs and initiatives

• While evaluation authority, staff, and resources are decentralized and found throughout the agency, SAMHSA is composed of four Centers, the Center for Mental Health Services (CMHS), the Center for Substance Abuse Treatment (CSAT), the Center for Substance Abuse Prevention (CSAP) and the Center for Behavioral Health Statistics and Quality (CBHSQ).
• As such, CMHS, CSAT, and CSAP oversee grantee portfolios and evaluations of those portfolios with the support of the Office of Evaluation. Evaluation decisions within SAMHSA are made within each Center specific to their program priorities and resources. Each of the three program Centers uses their program funds for conducting evaluations of varying types. CBHSQ, SAMHSA’s research arm, provides varying levels of oversight and guidance to the Centers for evaluation activities. CBHSQ also provides technical assistance related to data collection and analysis to assist in the development of evaluation tools and clearance package

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY2o?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• Formerly, SAMHSA had an Evaluation Policy and Procedure (P&P) that provided guidance across the agency regarding all program evaluations. Under “Evaluation Policies,” the SAMHSA website states: “Under [the] Evidence Act, federal agencies are expected to expand their capacity for engaging in program evaluation by designating evaluation officers, developing learning agendas; producing annual evaluation plans, and enabling a workforce to conduct internal evaluations. To this end, SAMHSA seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of its evaluations.”

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• While the Evaluation P&P served as the agency’s formal evaluation plan, a updated, draft evaluation plan is not available.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• As of August 2020, no public learning agenda is available on SAMHSA’s website. However, SAMHSA has posted a National Research Agenda on Homelessness.

2.4 Did the agency publicly release all completed program evaluations?

• As of August 2020, no evaluation reports or summaries are posted on the website, including any ongoing evaluation studies. However, the publications page lists 63 reports, of which nine appears to be evaluation reports. A word search of SAMHSA’s website for the term “evaluation” yielded five results, none of which are evaluation reports.
• The following criteria is used to determine whether an evaluation is significant: (1) whether the evaluation was mandated by Congress; (2) whether there are high priority needs in states and communities; (3) whether the evaluation is for a new or congressionally-mandated program; (4) the extent to which the program is linked to key agency initiatives; (5) the level of funding; (6) the level of interest from internal and external stakeholders; and (7) the potential to inform practice, policy, and/or budgetary decision-making. Results from significant evaluations are made available on SAMHSA’s evaluation website.

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))

• SAMHSA did not describe progress in developing an interim or draft Capacity Assessment. In 2017, SAMHSA formed a new workgroup, the Cross-Center Evaluation Review Board (CCERB). According to the former Evaluation P&P, the CCERB reviews and provides oversight of significant evaluation activities for SAMHSA, from contract planning to evaluation completion and at critical milestones, and is comprised of representatives from each of the centers, and Office of Tribal Affairs and Policy (OTAP) for cultural competency consultation, as necessary. CCERB staff provide support for program-specific and administration-wide evaluations. It is unclear if the CCERB still exists. A word search of the SAMHSA website (August 2020) for “Cross-Center Evaluation Review Board” yielded no results.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• SAMHSA does not list any completed evaluation reports on its evaluation website. Of the nine evaluation reports found on the publications page, none appear to use experimental methods. According to the Evaluation P&P (p. 5): “evaluations should be rigorously designed to the fullest extent possible and include ‘…inferences about cause and effect [that are] well founded (internal validity), […] clarity about the populations, settings, or circumstances to which results can be generalized (external validity); and requires the use of measures that accurately capture the intended information (measurement reliability and validity).’

Did the agency invest at least 1% of program funds in evaluations in FY20? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, rigorous evaluations, including random assignments)

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY20 budget.

• Results for America was unable to determine the amount of resources SAMHSA invested in evaluations in FY20.^[1]

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• Results for America was unable to determine the budget for evaluation at SAMHSA and, thus, any changes from the previous fiscal year. SAMHSA evaluations are funded from program funds that are used for service grants, technical assistance, and for evaluation activities. Each of the three program Centers uses their program funds for conducting evaluations of varying types. Evaluations have also been funded from recycled funds from grants or other contract activities.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• SAMHSA’s Evidence-Based Practices Resource Center aims to provide communities, clinicians, policy-makers and others in the field with the information and tools they need to incorporate evidence-based practices into their communities or clinical settings. The Center lists nine technical assistance projects, two of which appear to provide financial or other resources to help city, county, and state governments or other grantees build evaluation capacity (as of September 2019):
  • The Bringing Recovery Supports to Scale Technical Assistance Center Strategy (BRSS TACS) advances recovery supports and services for people with mental or substance use disorders and their families. The BRSS TACS website indicates it has provided training and technical assistance for building the capacity of peer-run, recovery community, and family organizations through evaluation, among six other topics. 
  • The National Training and Technical Assistance Center for Child, Youth & Family Mental Health (NTTAC)provides states, tribes, and communities with training and technical assistance on children’s behavioral health, with a focus on systems of care. NTTAC’s Training and Technical Assistance activities for clinical best practices, wraparound services, and workforce development focus on evaluation, fidelity assessment, and quality assurance, among nine other topics.

^[1] Results for America was unable to determine the amount of resources SAMHSA invested in evaluations in FY20for criterion #3.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY20?
(Example: Performance stat systems, frequent outcomes-focused data-informed meetings)

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• The SAMHSA Strategic Plan FY2019-FY2023 outlines five priority areas with goals and measurable objectives to carry out the vision and mission of SAMHSA. For each priority area, an overarching goal and series of measurable objectives are described followed by examples of key performance and outcome measures SAMHSA will use to track progress.

4.2 Does the agency use data/evidence to improve outcomes and return on investment?

• According to the SAMHSA website, the Office of Evaluation “oversees the identification of a set of performance indicators to monitor each SAMHSA program in collaboration with program staff and the development of periodic program profiles for use in agency planning, program change, and reporting to departmental and external organizations” and “utilizes SAMHSA’s Performance Accountability and Reporting System (SPARS) which serves as a mechanism for the collection of performance data from agency grantees.”
• According to the FY2019-FY2023 Strategic Plan (pp. 21-22), SAMHSA will modernize the Performance Accountability and Reporting System (SPARS) by 1) capturing real-time data for discretionary grant programs in order to monitor their progress, impact, and effectiveness, and 2) developing benchmarks and disseminating annual Performance Evaluation Reports for all SAMHSA discretionary grant programs.
• The Centers have historically managed internal performance review boards to periodically review grantee performance and provide corrective actions as needed. The SAMHSA website states that the Office of Evaluation is charged with “providing oversight and management of agency quality improvement and performance management activities and for advancing agency goals and objectives related to program evaluation, performance measurement, and quality improvement.”

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• As described on the SAMHSA website, the Office of Evaluation supports continuous improvement and learning in several ways:
  • Analyzes data in support of agency needs and develops evaluation and performance related reports in response to internal and external request;
  • Oversees the identification of a set of performance indicators to monitor each SAMHSA program in collaboration with program staff and the development of periodic program profiles for use in agency planning, program change, and reporting to departmental and external organizations;
  • Utilizes SAMHSA’s Performance Accountability and Reporting System (SPARS) which serves as a mechanism for the collection of performance data from agency grantees; and
  • Responds to agency and departmental requests for performance measurement data and information; and conducts a range of analytic and support activities to promote the use of performance data and information in the monitoring and management of agency programs and initiatives.
•  In 2016, SAMHSA’s Office of Financial Resources (OFR) established a Program Integrity Review Team (PIRT) staffed by representatives from each of its four Centers and managed by OFR. However, information about PIRT is no longer publicly available as of November 2020. rformance and other factors. Program staff present their program self-assessments to the PIRT and receive feedback on, for example, targets of concern.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY20?
(Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies; data-use policies)

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• The SAMHSA Strategic Plan FY2019-FY2023 (pp. 20-23) outlines five priority areas to carry out the vision and mission of SAMHSA, including Priority 4: Improving Data Collection, Analysis, Dissemination, and Program and Policy Evaluation. This Priority includes three objectives: 1) Develop consistent data collection strategies to identify and track mental health and substance use needs across the nation; 2) Ensure that all SAMHSA programs are evaluated in a robust, timely, and high-quality manner; and 3) Promote access to and use of the nation’s substance use and mental health data and conduct program and policy evaluations and use the results to advance the adoption of evidence-based policies, programs, and practices. The SAMHSA website states: “CBHSQ coordinates an integrated data strategy, which includes collecting data each year on the national incidence and prevalence of various forms of mental illness and substance use.”

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• SAMHSA’s Data and Dissemination site identifies eight data collection initiatives: the National Survey on Drug Use and Health (NSDUH): Treatment Episode Data Set (TEDS); National Survey of Substance Abuse Treatment Services (N-SSATS); the National Mental Health Services Survey (N-MHSS); Drug Abuse Warning Network (DAWN); Mental Health Client-Level Data (MH-CLD); Uniform Reporting System (URS); and Substance Abuse and Mental Health Data Archive (SAMHDA). SAMHSA has made numerous administrative and survey datasets publicly available for secondary use. 
  

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c)) 

• The Center for Behavioral Health Statistics and Quality (CBHSQ) oversees data collection initiatives and provides publicly available datasets so that some data can be shared with researchers and other stakeholders while preserving client confidentiality and privacy. 
• SAMHSA’s Substance Abuse and Mental Health Data Archive (SAMHDA) contains substance use disorder and mental illness research data available for restricted and public use. SAMHDA promotes the access and use of SAMHSA’s substance abuse and mental health data by providing public-use data files and documentation for download and online analysis tools to support a better understanding of this critical area of public health.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)

• SAMHSA’s Performance and Accountability and Reporting System (SPARS) hosts the data entry, technical assistance request, and training system for grantees to report performance data to SAMHSA. SPARS serves as the data repository for the Administration’s three centers, Center for Substance Abuse and Prevention (CSAP), Center for Mental Health Services (CMHS), and Center for Substance Abuse Treatment (CSAT). In order to safeguard confidentiality and privacy, the current data transfer agreement limits the use of grantee data to internal reports so that data collected by SAMHSA grantees will not be available to share with researchers or stakeholders beyond SAMHSA, and publications based on grantee data will not be permitted.

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• The Center of Excellence for Protected Health Information (CoE for PHI) is a SAMHSA funded technical assistance project designed to develop and increase access to simple, clear, and actionable educational resources, training, and technical assistance for consumers and their families, state agencies, and communities to promote patient care while protecting confidentiality. According to the SAMHSA website: “the National Survey on Drug Use and Health (NSDUH) data are available (1) as pre-published estimates, (2) via online analyses systems, and (3) as microdata files.” A description of NSDUH products can be found under the NSDUH landing page.
• Through SAMHSA’s Substance Abuse and Mental Health Data Archive (SAMHDA) SAMHSA has partnered with the National Center for Health Statistics (NCHS) to host restricted-use National Survey on Drug Use and Health (NSDUH) data at their Federal Statistical Research Data Centers (RDCs). RDCs are secure facilities that provide access to a range of restricted-use microdata for statistical purposes.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY20?
(Example: What Works Clearinghouses)

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• There is great diversity across SAMHSA programming, ranging from community-level prevention activities to residential programs for pregnant and postpartum women with substance misuse issues. While this diversity allows SAMHSA to be responsive to a wide set of vulnerable populations, it limits the utility of a common evidence framework for the entire agency. Within Centers (the Center for Substance Abuse Prevention, the Center for Substance Abuse Treatment, and the Center for Mental Health Services), consistent evidence frameworks are in use and help to shape the process of grant-making (e.g., Center staff are familiar with the pertinent evidence base for their particular portfolios).
• In 2011, based on the model of the National Quality Strategy, SAMHSA developed the National Behavioral Health Quality Framework (NBHQF). With the NBHQF, SAMHSA proposes a set of core measures to be used in a variety of settings and programs, as well as in evaluation and quality assurance efforts. The proposed measures are not intended to be a complete or total set of measures a payer, system, practitioner, or program may want to use to monitor the quality of its overall system or the care or activities it provides. SAMHSA encourages such entities to utilize these basic measures as appropriate as a consistent set of indicators of quality in behavioral health prevention, promotion, treatment, and recovery support efforts across the nation.

6.2 Did the agency have a common evidence framework for funding decisions?

• SAMHSA has universal language about using evidence-based practices (EBPs) that is included in its Funding Opportunity Announcements (FOAs) (entitled Using Evidence-Based Practices (EBPs)). This language includes acknowledgement that, “EBPs have not been developed for all populations and/or service settings” thus encouraging applicants to “provide other forms of evidence” that a proposed practice is appropriate for the intended population. Specifically, the language states that applicants should: (1) document that the EBPs chosen are appropriate for intended outcomes; (2) explain how the practice meets SAMHSA’s goals for the grant program; (3) describe any modifications or adaptations needed for the practice to meet the goals of the project; (4) explain why the EBP was selected; (5) justify the use of multiple EBPs, if applicable; and (6) discuss training needs or plans to ensure successful implementation. Lastly, the language includes resources the applicant can use to understand EBPs. Federal grants officers work in collaboration with the SAMHSA Office of Financial Resources to ensure that grantee funding announcements clearly describe the evidence standard necessary to meet funding requirements.
• SAMHSA developed a manual, Developing a Competitive SAMHSA Grant Application, which explains information applicants will likely need for each section of the grant application. The manual has two sections devoted to evidence-based practices (p. 8, p. 26), including: 1) A description of the EBPs applicants plan to implement; 2) Specific information about any modifications applicants plan to make to the EBPs and a justification for making them; and 3) How applicants plan to monitor the implementation of the EBPs. In addition, if applicants plan to implement services or practices that are not evidence-based, they must show that these services/practices are effective.

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• Until 2018, SAMHSA regarded the National Registry of Evidence-based Programs and Practices (NREPP) as the primary online user-friendly tool for identifying evidence-based programs for grantee implementation. In January 2018, SAMHSA announced that it was “moving to EBP [evidence-based practice] implementation efforts through targeted technical assistance and training that makes use of local and national experts and will assist programs with actually implementing services….” NREPP was taken offline in August 2018. In August 2019, the Pew-MacArthur Results First Initiative announced it had restored users’ access to this information, which can be found in the Results First Clearinghouse Database. The Evidence-Based Practices Resource Center “provides communities, clinicians, policy-makers and others with the information and tools to incorporate evidence-based practices into their communities or clinical settings.” As of August 2020, the EBP Resource Center included 149 items.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• In April 2018, SAMHSA launched the Evidence-Based Practices Resource Center (Resource Center) that aims to provide communities, clinicians, policy-makers and others in the field with the information and tools they need to incorporate evidence-based practices into their communities or clinical settings. The Resource Center contains a collection of science-based resources, including Treatment Improvement Protocols, toolkits, resource guides, and clinical practice guidelines, for a broad range of audiences. As of August 2020, the Resource Center includes 149 items, including 15 data reports, 24 toolkits, 24 fact sheets, and 96 practice guides.
• The Mental Health Technology Transfer Center (MHTTC) Network works with organizations and treatment practitioners involved in the delivery of mental health services to strengthen their capacity to deliver effective evidence-based practices to individuals, including the full continuum of services spanning mental illness prevention, treatment, and recovery support. The State Targeted Response Technical Assistance (STR-TA), known as the Opioid Response Network, was created to support efforts to address opioid use disorder prevention, treatment, and recovery, and to provide education and training at the local level in evidence-based practices.
• To date SAMHSA has produced 11 Evidence-Based Practice Knowledge Informing Transformation (KIT) guides to help move the latest information available on effective behavioral health practices into community-based service delivery. The KITs contain information sheets, introductory videos, practice demonstration videos, and training manuals. Each KIT outlines the essential components of the evidence-based practice and provides suggestions collected from those who have successfully implemented them.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY20?
(Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements)

7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?

• SAMHSA participates in collaborations with other HHS agencies to promote innovative uses of data, technology and innovation across HHS to create a more effective government and improve the health of the nation, via the HHS IDEA Lab. SAMHSA has co-developed and submitted several innovative data utilization project proposals to the Ignite Accelerator of the HHS IDEA Lab, such as Rapid Opioid Alert and Response (ROAR), a project to monitor and prevent opioid overdoses by linking heroin users to resources and information.

7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?

• Pursuant to the 21st Century Cures Act, SAMHSA established the National Mental Health and Substance Use Policy Laboratory (NMHSUPL) as an office, led by a Director. The NMHSUPL promotes evidence-based practices and service delivery models through evaluating models that would benefit from further development and through expanding, replicating, or scaling evidence-based programs across a wider area. Specifically, according to the SAMHSA website, NMHSUPL:
  • Identifies, coordinates, and facilitates the implementation of policy changes likely to have a significant effect on mental health, mental illness (especially severe mental illnesses such as schizophrenia and schizoaffective disorders), recovery supports, and the prevention and treatment of substance use disorder services;
  • Works with CBHSQ to collect information from grantees under programs operated by the Administration in order to evaluate and disseminate information on evidence-based practices, including culturally and linguistically appropriate services, as appropriate, and service delivery models; and
• Carries out other activities as deemed necessary to continue to encourage innovation and disseminate evidence-based programs and practices.
• The SAMHSA Program Portal, a collection of technical assistance and training resources provided by the agency, provides behavioral health professionals with education and collaboration opportunities, and ample tools and technical assistance resources that promote innovation in practice and program improvement. Located within the Knowledge Network are groups such as the Center for Financing Reform and Innovation, which works with states and territories, local policy makers, providers, consumers, and other stakeholders to promote innovative financing and delivery system reforms.

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• SAMHSA does not list any completed evaluation reports on its evaluation website. Of the nine evaluation reports found on the publications page, none appear to use experimental methods.

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY20? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring for evidence; Pay for Success provisions)

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

In FY20, the five largest competitive grant programs are:

1. State Opioid Response Grants ($1.5 billion; eligible applicants: states);
2. Children Mental Health Services ($1.25 billion; eligible applicants: States, Tribes, Communities, Territories);
3. Strategic Prevention Framework ($119.5 million; eligible applicants: public and private nonprofit entities);
4. Targeted Capacity Expansion – General ($100.2 million; eligible applicants: domestic public and private nonprofit entities);
5. Project AWARE ($92 million; eligible applicants: State education agencies).

8.2 Did the agency use evidence of effectiveness to allocate funds in its five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• The FY20 State Opioid Response Grants application required states to use evidence-based practices to address opioid use disorder (p. 19), as 1 of 5 evaluation criteria; however the application did not allot points for the various criteria.
• The FY20 Strategic Prevention Framework Grants application states that applicants are expected to use evidence-based practices (p. 8), but this does not factor in the evaluation of applications (pp. 14-16).
• The FY20 Project AWARE State Education Agency Grants application gave applicants 25 out of 100 points for the following: “Identify the Evidence-Based Practice(s) (EBPs) that will be used in each of the three LEAS [local educational agencies]. Discuss how each EBP chosen is appropriate for your population(s) of focus and the outcomes you want to achieve. Describe any modifications that will be made to the EBP(s) and the reason the modifications are necessary” (p. 21).
• The FY19 Targeted Capacity Expansion Grants application gave applicants 25 out of 100 points for proposing evidence-based services or practices (p. 19).

8.3 Did the agency use its five largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• The FY20 Strategic Prevention Framework Grants application states that SAMHSA may negotiate additional terms and conditions with applicants prior to grant award, including “requirements relating to participation in a cross-site evaluation” (p. 51).
• The FY20 Project AWARE State Education Agency Grants application states that SAMHSA may negotiate additional terms and conditions with applicants prior to grant award, including “requirements relating to participation in a cross-site evaluation” (p. 58).
• The FY19 Targeted Capacity Expansion Grants application stated that SAMHSA may negotiate additional terms and conditions with applicants prior to grant award, including “requirements relating to participation in a cross-site evaluation” (p. 57).

8.4 Did the agency use evidence of effectiveness to allocate funds in any other competitive grant program (besides its five largest grant programs)?

• No examples available.

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• No examples available.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• No examples available.

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY20?
(Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

9.1 What were the agency’s 5 largest non-competitive programs and their appropriation amounts (and were city, county, and/or state governments are eligible to receive funds from these programs)?

• In FY20, the five largest non-competitive grant programs are:
  1. Substance Abuse Prevention and Treatment Block Grant Program ($1.86 billion; eligible grantees: states);
  2. Community Mental Health Block Grant Program ($722.5 million; eligible grantees: states);
  3. Projects for Assistance in Transition from Homelessness (PATH) Program ($64.6 million; eligible grantees: states); and
  4. Protection and Advocacy for Individuals with Mental Illness (PAIMI) Program ($36.1 million; eligible grantees: states).

9.2 Did the agency use evidence of effectiveness to allocate funds in its five largest non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)

• In FY20, Congress maintained the 10 percent set-aside for evidence-based programs in SAMHSA’s Mental Health Grant Block (MHBG) grant to address the needs of individuals with early serious mental illness, including psychotic disorders, regardless of the age of the individual at onset (see p. 48 of the FY20-FY21 Block Grant Application). In the FY21 budget request (p. 348), SAMHSA expressed its desire to reduce the set-aside in half to 5%.
• The FY20-FY21 Block Grant Application requires states seeking Mental Health Block Grant (MHBG) and Substance Abuse and Treatment Prevention Block Grant (SAGB) funds to identify specific priorities. For each priority, states must identify the relevant goals, measurable objectives, and at least one-performance indicator for each objective, which must include strategies to deliver evidence-based individualized treatment plans (p. 21); evidence-based interventions for substance use or dependence (p. 21); building provider capacity to deliver evidence-based, trauma-specific interventions (p. 22); evidence-based programs, policies, and practices in prevention efforts (p. 22); evidence-based models to prevent substance misuse (p. 23). 

9.3 Did the agency use its five largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• The FY20-FY21 Block Grant Application requires states applying for Substance Abuse Prevention and Treatment funds to create an evaluation plan, which must include at least five specified evaluation elements. Additionally, the application specifies that SAMHSA will work with the National Institute of Mental Health (NIMH) to plan for program evaluation and data collection related to demonstrating program effectiveness of the Mental Health Block Grant.

9.4 Did the agency use evidence of effectiveness to allocate funds in any other non-competitive grant program (besides its five largest grant programs)?

• No examples available.

9.5 What are the agency’s 1-2 strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• No examples available.

9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• The FY20-FY21 Block Grant Application clarified that “Section 1921 of the PHS [Public Health Services] Act (42 U.S.C.§ 300x-21) authorizes the States to obligate and expend SABG [Substance Abuse and Treatment Prevention Block Grant] funds to plan, carry out and evaluate activities and services designed to prevent and treat substance use disorders” (p. 16). The Application further clarifies that states “may utilize SABG funds to train personnel to conduct fidelity assessments of evidence-based practices” (p. 35).

In FY20, did the agency shift funds away from or within any practice, policy, or program that consistently failed to achieve desired outcomes?
(Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; incentivizing or urging grant applicants to stop using ineffective practices in funding announcements; proposing the elimination of ineffective programs through annual budget requests; incentivizing well-designed trials to fill specific knowledge gaps; supporting low-performing grantees through mentoring, improvement plans, and other forms of assistance; using rigorous evaluation results to shift funds away from a program)  

10.1 Did the agency have policy(ies) for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• As a matter of policy, SAMHSA uses the term “restricted status” to describe grant recipients that are financially unstable, have inadequate financial management systems, or are poor programmatic performers. Grants placed on restricted status require additional monitoring and have additional award conditions that must be met before funds can be drawn. SAMHSA adheres to HHS Grants Policy Statement, including the policy on suspension or termination, which states: “If a recipient has failed to materially comply with the terms and conditions of award, the OPDIV [Grant-Awarding Operating Division] may suspend the grant, pending corrective action, or may terminate the grant for cause” (p. II-89).
• The FY18 State Opioid Response Grants program required states and subgrantees to only use evidence-based treatments, practices, and interventions. As such, SAMHSA disallowed the use of medical withdrawal (detoxification) in isolation since it “is not the standard of care for OUD, is associated with a very high relapse rate, and significantly increases an individual’s risk for opioid overdose and death if opioid use is resumed” (p. 6). And SAMHSA clarified: “SAMHSA will monitor use of these funds to assure that they are being used to support evidence-based treatment and recovery supports and will not permit use of these funds for non-evidence-based approaches” (p. 7). Further, under Standard Funding Restrictions, SAMHSA included: “non-evidence-based treatment approaches” (p. 54).
• In January 2018, SAMHSA announced it would shift resources away from the National Registry of Evidence-based Programs and Practices (NREPP) toward targeted technical assistance and training for implementing evidence-based practices. The reasoning was that NREPP had flawed and skewed presentation of evidence-based interventions, which “did not address the spectrum of needs of those living with serious mental illness and substance use disorders.”

10. 2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• Results for America was unable to identify any examples.