Criteria for Evidence of Effectiveness To Be Applied to Projects Identified for Inclusion in the What Works Clearinghouse of Proven and Promising Projects To Move Welfare Recipients Into Work, 26290-26293 [2018-12160]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Administration for Children and Families

[Federal Register Volume 83, Number 109 (Wednesday, June 6, 2018)]
[Notices]
[Pages 26290-26293]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2018-12160]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Criteria for Evidence of Effectiveness To Be Applied to Projects 
Identified for Inclusion in the What Works Clearinghouse of Proven and 
Promising Projects To Move Welfare Recipients Into Work

AGENCY: Administration for Children and Families (ACF), U.S. Department 
of Health and Human Services (HHS).

ACTION: Request for public comment.

-----------------------------------------------------------------------

SUMMARY: The Administration for Children and Families, HHS, solicits 
comments by August 5, 2018 on the criteria for evidence of 
effectiveness for the What Works Clearinghouse of Proven and Promising 
Projects to Move Welfare Recipients into Work. Final criteria for 
evidence of effectiveness will be used to develop the clearinghouse.

SUPPLEMENTARY INFORMATION: 
    Invitation to Comment: HHS invites comments regarding this notice 
on the proposed criteria for HHS's systematic review of the evidence. 
To ensure that your comments are clearly stated, please identify the 
specific criterion or other section of this notice that your comments 
address.

1.0 Background

1.1 Legislative Context

    The Consolidated Appropriations Act of 2017 (Pub. L. 115-31 
(https://www.congress.gov/115/plaws/publ31/PLAW-115publ31.pdf)) directs 
the U.S. Department of Health and Human Services (HHS) to create a 
database of projects that have used a proven or promising approach to 
move welfare recipients into work, based on independent, rigorous 
evaluations of the projects, and to create a What Works Clearinghouse 
of Proven and Promising Projects to Move Welfare Recipients into Work. 
As stated in the statute, the database shall additionally ``include a 
separate listing of projects that used a developmental approach in 
delivering services and a further separate listing of the projects with 
no or negative effects.'' The statute requires HHS to establish 
criteria for evidence of effectiveness.

1.2 The Legislation's Direction for Establishing the Criteria for 
Evidence of Effectiveness

    Section 413(g)(2) of Public Law 115-31 charges the Secretary of 
Health and Human Services with establishing the criteria of 
effectiveness. The statute further stipulated that the (B) process for 
establishing the criteria--
    (i) is transparent;
    (ii) is consistent across agencies;
    (iii) provides opportunity for public comment; and
    (iv) takes into account efforts of Federal agencies to identify and 
publicize effective interventions, including efforts at the Department 
of Health and Human Services, the Department of Education, and the 
Department of Justice.

1.3 The Employment Strategies for Low-Income Adults Evidence Review

    Prior to the enactment of Public Law 115-31, the Office of 
Planning, Research, and Evaluation (OPRE) at the Administration for 
Children and Families (ACF) at HHS had developed the Employment 
Strategies for Low-Income Adults Evidence Review (ESER). The new 
statute aligns with and extends the work of ESER. HHS proposes building 
on this existing work to develop the new Clearinghouse.
    The Employment Strategies for Low-Income Adults Evidence Review 
(ESER) is a systematic review of the evaluation research published 
between 1990 and 2014 on employment and training programs for low-
income adults. It culminated in a searchable, public database (https://employmentstrategies.acf.hhs.gov/). The review was supplemented with 
briefs synthesizing the results of the review and highlighting 
strategies that appeared to be promising, as identified by the review. 
To identify the programs and strategies--or interventions-- that appear 
to be most effective in helping low-income adults gain and retain 
employment, ESER systematically identified, assessed, and synthesized 
evidence from the existing evaluation research literature. A core 
component of ESER's review, as with other federal evidence reviews, 
involved assessing the quality of the research evidence on different 
interventions.
    To assess the quality of the evidence, ESER reviewed each study's 
methods to

[[Page 26291]]

determine if they were rigorous enough to ensure that the study's 
findings could be considered reliable. ESER assessed whether the 
study's methods reliably supported the conclusion that an 
intervention's impacts were caused by the intervention and not by 
something else. The standards for assessing studies' methods were 
defined based on consultation with federal experts on evidence reviews 
and researchers with expertise in evaluation methodology. To 
differentiate among different levels of the strength of evidence, ESER 
assigned a High, Moderate, or Low rating to each study reviewed. 246 of 
the 314 studies included in the review earned a High rating and 1 study 
earned a Moderate rating. The remaining 67 studies received a Low 
rating.
    Through this review, ESER was able to identify interventions whose 
findings could be considered most reliable. Studies' ratings reflect 
the rigor of their study methods, independent of whether the findings 
were positive or negative. As a result, a study could be rated High or 
Moderate even if the intervention studied did not improve the outcomes 
for low-income adults. While the vast majority of studies included in 
ESER achieved a High rating (and, therefore, are considered to provide 
reliable, or strong, evidence), the review also found that, overall, 
null impacts were more prevalent than statistically significant 
impacts.
    While ESER did not assess the effectiveness of the interventions 
reviewed, ESER conducted a number of preliminary steps necessary for 
assessing effectiveness. This included categorizing each study's 
findings according to whether it found positive, negative or null 
impacts for the interventions studied. In addition, through a number of 
synthesis briefs (published on the website), ESER qualitatively and 
quantitatively summarized the direction of impacts for different 
interventions and highlighted interventions associated with the 
greatest number of positive impacts.
    To be included in ESER, studies had to--
     Quantitatively measure the effectiveness of a program or 
strategy
     Be published between 1990 and 2014
     Study an employment program or strategy-- an 
intervention-- that
    [cir] had a primary aim of improving employment-related outcomes
    [cir] primarily targeted low-income adults
    [cir] took place in the United States, Canada, or the United 
Kingdom
    To identify studies eligible for review, ESER issued a call for 
papers, conducted literature searches, and consulted with experts in 
workforce development programs that serve low-income adults.
    ESER looked at the effects of the interventions on four domains, or 
outcome areas:
     Employment
     Earnings
     Public benefit receipt
     Education/training
    Outcomes were examined for short and longer-term impacts (longer-
term was measured as being more than 18 months after the intervention 
was implemented).
    The ESER website (https://employmentstrategies.acf.hhs.gov/) 
reports key results for all eligible studies. The website also allows 
users to search for results by program studied, target population, 
outcome(s) of interest, service strategies, intervention setting, year 
of study publication, and whether favorable impacts were found.
    While ESER's overall population of interest was low-income adults, 
a majority of the studies in ESER examined welfare populations. Because 
studies of interventions in a welfare setting typically include both 
recipients and applicants, ESER does not include any studies that 
solely focused on welfare recipients. ESER does, however, include 
interventions targeted to low-income populations understood to share 
important characteristics with welfare recipients, such as other public 
benefit recipients, and those considered hard to employ, including 
those who have been homeless or formerly incarcerated.

2.0 Process for Establishing the Criteria of Effectiveness for the New 
What Works Clearinghouse

    In fall 2017 and early winter 2018, OPRE engaged in a series of 
systematic consultations with federal and non-federal technical experts 
on evidence reviews. In addition to representation from the Department 
of Labor (DOL) and the Department of Education (ED) in these 
consultations, federal representation included the Department of 
Justice (DOJ) and a number of HHS agencies/offices including the Office 
of Family Assistance (OFA), the Office of Planning, Research, and 
Evaluation (OPRE), the Office of the Assistant Secretary for Planning 
and Evaluation (ASPE) and the Agency for Healthcare Research and 
Quality (AHRQ).
    The objective of these consultations was to help HHS:
    (1) Develop criteria for categorizing interventions in the new 
Clearinghouse as proven, promising, developmental, or ineffective,
    (2) develop these criteria through a process that
    a. involved consultation with the Department of Labor (DOL), the 
Department of Education (ED), and other entities with experience 
evaluating relevant effectiveness research,
    b. allowed HHS to better understand other Federal evidence reviews' 
standards and processes and determine where it would make sense for the 
new Clearinghouse to be consistent with these standards and processes, 
and
    (3) learn best practices from other Federal evidence reviews for 
identifying and publicizing effective interventions

2.1 Transparent

    To ensure that the Clearinghouse's procedures and standards, 
including the criteria for evidence of effectiveness, are transparent, 
HHS intends to implement the following practices:
     Post the procedures and standards and information about 
the process on the Clearinghouse website.
     Provide the public a means of contacting the 
Clearinghouse, for example, by establishing a help desk to respond to 
email inquiries.

2.2 Consistent Across Agencies

    To ensure that the Clearinghouse is as consistent as possible with 
other federal evidence reviews in its processes and standards, HHS 
intends to implement the following practices:
     Adopt the standards and methods for reviewing studies from 
OPRE's existing Employment Strategies Evidence Review (ESER) (https://www.acf.hhs.gov/opre/resource/employment-strategies-for-low-income-adults-evidence-review-standards-and-methods), which are broadly 
consistent with other federal Clearinghouses. ESER's standards and 
methods (e.g., author queries; number and training of reviewers; 
choices about reporting effect sizes) were developed by considering 
both the choices made by other federal and non-federal Clearinghouses 
and the standards of research in the employment and training 
intervention field. Other existing federal Clearinghouses have followed 
this same approach (considering both the choices made by other 
clearinghouses and the norms of research within their fields of focus).
     In any instances where the new Clearinghouse's ratings of 
a project's strength of evidence or effectiveness differ from another 
federal evidence review that rates projects according to the same 
outcomes (such as the Department of Labor's (DOL's) Clearinghouse for 
Labor Evaluation and

[[Page 26292]]

Research (CLEAR)), annotate the findings to explain the reason for the 
difference.

2.3 Provides Opportunity for Public Comment

    To provide an opportunity for public comment on the criteria for 
effectiveness, ACF is publishing this Federal Register Notice.

2.4 Takes Into Account Efforts of Federal Agencies To Identify and 
Publicize Effective Interventions

    To ensure the Clearinghouse reflects the learning of other Federal 
agencies about how to identify and publicize effective interventions, 
HHS intends to implement the following practices:
     Use some of the methods adopted by other clearinghouses to 
create multiple products tailored to different audiences and use 
graphic design and other user-friendly dissemination elements to help 
users digest evidence quickly.
     Include information on the Clearinghouse website that is 
especially useful to practitioners, such as summary information about 
projects and approaches.
     Develop and incorporate alternative media for the 
Clearinghouse such as videos that will tailor communication to various 
groups.
     Ensure that information is effectively conveyed on the 
Clearinghouse website by soliciting feedback from various stakeholders 
who can represent key target audiences. Key among these would be state 
or county Temporary Assistance for Needy Families (TANF) and Workforce 
Development practitioners, as well as evaluation researchers.

3.0 Proposed Criteria for Evidence of Effectiveness

3.1 Criteria for Well-Designed, Rigorous Impact Research

    HHS intends to employ the criteria established by OPRE's Employment 
Strategies for Low-Income Adults Evidence Review (ESER) to assess the 
quality of study design and to assess the strength of the evidence 
resulting from studies. These criteria (referred to as ``standards and 
methods'') are available in ESER's Standards and Methods report https://www.acf.hhs.gov/opre/resource/employment-strategies-for-low-income-adults-evidence-review-standards-and-methods.

3.2 Proposed Criteria for Evidence of Effectiveness for Projects 
Included in the Clearinghouse

3.2.1 Definition of Project and Approach
    The legislation requires that ratings, or categorizations, of 
evidence of effectiveness be applied to projects and approaches. To 
standardize definitions for these terms, HHS intends to define a 
project and an approach as follows:
     Define project to be a specific bundle of services and/or 
policies implemented in a given context.
     Project will be the unit that receives an effectiveness 
rating (i.e. proven, promising, developmental, or ineffective).
     Define approach to be the guiding framework of specific 
services (e.g., career pathways).
     Approaches will not be rated as proven, promising, 
developmental, or ineffective, but the Clearinghouse will include 
narrative summaries related to different approaches.
     While the legislation does not require HHS to define or 
evaluate the effectiveness of program components, there is interest in 
the field in examining program components. Thus, HHS intends that the 
Clearinghouse include meta-analyses of specific components of projects 
(such as ``case management'' or ``job search assistance'') whenever 
appropriate and feasible.
3.2.2 Parameters Guiding the Application of Evidence of Effectiveness 
Ratings
    Before a project can be categorized as being proven, promising, 
developmental, or ineffective, a number of preliminary definitions, or 
parameters, must be established to guide decision making. These include 
the outcomes for which a project's effectiveness will be evaluated, how 
a favorable or unfavorable effect will be measured, and how an 
effectiveness rating will be applied to a project.
3.2.2.1 Outcomes
    HHS intends that the new Clearinghouse will review the following 
outcomes:
    [cir] Employment (short and longer-term),
    [cir] earnings (short and longer-term),
    [cir] educational attainment, and
    [cir] public benefit receipt.
3.2.2.2 Definition of Favorable and Unfavorable Effects
    HHS intends that the Clearinghouse consider only statistically 
significant findings (p <.05) as evidence of favorable or unfavorable 
effects.
3.2.2.3 Pre-Defining Criteria for Selecting Among Multiple Outcome 
Measures
    HHS intends to reduce the likelihood for reporting a false positive 
rate for outcomes--an issue that can occur when studies use multiple 
measures or multiple outcomes to assess impacts in the same domain 
(e.g., short-term earnings)--by relying on the decision rules ESER 
developed to address the potential for multiple comparisons. https://www.acf.hhs.gov/opre/resource/employment-strategies-for-low-income-adults-evidence-review-standards-and-methods).
3.2.2.4 Application of Evidence of Effectiveness Ratings
    HHS intends that evidence of effectiveness ratings will be applied 
within outcome domains; each project will receive ratings of 
effectiveness on each outcome domain (e.g., a project may be found 
promising for short-term employment but ineffective for long-term 
employment). There will be no overall rating for the project.
3.2.3 Definition of Proven
    The legislation directs HHS to categorize projects as Proven, 
Promising, Developmental, or Ineffective.
    HHS intends that for a project to be considered proven, the 
following conditions must be met:
     There must be at least two separate studies of the same 
project that meet evidence standards and meet criteria for a promising 
rating.
    [cir] Studies are considered to be separate studies of the same 
project if they use non-overlapping samples to examine distinct 
implementations of the project.
     There must be only favorable or null impacts within a 
given outcome domain. Thus, no studies that meet evidence standards for 
a given outcome domain can show an unfavorable impact within that 
domain.
     Projects that have both favorable and unfavorable impacts 
in a given domain will be categorized as mixed.
     A project has a limited number, or proportion, of null 
findings in a given domain.
    HHS is soliciting comments on how to best determine the ceiling for 
the number, or proportion, of null to positive findings in a given 
domain.
    If subsequent studies or replications result in only null findings 
in a given domain, the review will establish procedures for revisiting 
a project's rating of proven.

[[Page 26293]]

3.2.4 Definition of Promising
    HHS intends that for a project to be considered promising, the 
following conditions must be met:
     One study of a project must meet evidence standards.
     That study must find only favorable or null impacts within 
a given outcome domain. Thus no studies that meet evidence standards 
for an outcome domain can show an unfavorable impact within the domain.
    [cir] If the review examines more than one measure to identify 
impacts on a particular domain (e.g., Unemployment Insurance data and 
participant survey data), as long as one measure (among those selected 
according to 3.2.2.3 above) finds favorable impacts for that outcome, 
the intervention can receive a Promising rating for that outcome.
     Projects that have both favorable and unfavorable impacts 
in a given domain will be categorized as mixed.
3.2.5 Definition of Ineffective
    HHS intends that for a project to be considered ineffective, the 
following conditions must be met:
     One or more studies of a project must meet evidence 
standards.
     There must be only findings of unfavorable or null effects 
in a given domain.
     For studies finding null effect in a given domain, the 
review will include a measure of statistical precision--so that small, 
under-powered studies do not drive the effectiveness rating. If an 
intervention has been evaluated using only small studies, a lack of 
detectable effects could reflect either ineffectiveness of the 
intervention or the lack of statistical power to detect effects. It 
would be misleading to characterize this latter scenario as an 
ineffective project.
3.2.6 Definition of Developmental
    HHS intends that for a project to be considered developmental, the 
following conditions must be met:
     There must be at least one current, ongoing evaluation of 
the project that uses a study design that meets evidence standards but 
has not yet produced impact findings.
3.2.7 Additional Category of Mixed and Definition of Mixed
    HHS intends that there be an additional category for categorizing 
evidence of effectiveness called mixed. HHS proposes that for a project 
to be considered mixed, the following conditions must be met:
     One or more studies of a project must meet evidence 
standards.
     The studies find both favorable and unfavorable impact 
estimates within the same domain.
    3.2.8 HHS intends that narrative descriptions of rated projects, 
narrative descriptions of approaches, and information on case studies 
be provided to users of the Clearinghouse to facilitate a fuller 
understanding of the field of welfare-to-work interventions.

4.0 Submission of Comments

    Comments may be submitted until August 5, 2018 by email to 
[email protected].

Naomi Goldstein,
Deputy Assistant Secretary for Planning, Research, and Evaluation.
[FR Doc. 2018-12160 Filed 6-5-18; 8:45 am]
 BILLING CODE 4184-09-P