Criteria for Evidence of Effectiveness To Be Applied to Projects Identified for Inclusion in the What Works Clearinghouse of Proven and Promising Projects To Move Welfare Recipients Into Work, 26290-26293 [2018-12160]

Download as PDF 26290 Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices the Act Against AIDS National Testing’’. The information collected from these data collections was used to evaluate a specific AAA campaign phase. We are requesting the same amount of time to continue surveying AAA target audiences as new phases are developed. Through this extension, we plan to reach the remaining approved 6,445 respondents. To obtain the remaining respondents, we anticipate screening approximately 32,220 individuals. Depending on the target audience for the campaign phase, the study screener will vary. The study screener may address one or more of the following items: Race/ethnicity, sexual behavior, sexual orientation, gender identity, HIV testing history, HIV status, and injection drug use. Each survey will have a core set of items asked in all rounds, as well as a module of questions relating to specific AAA phases and activities. Respondents will be recruited through national opt-in email lists, the internet, and external partnerships with community-based and membership organizations that work with or represent individuals from targeted populations (e.g., National Urban League, the National Medical Association). Respondents will selfadminister the survey at home on personal computers. There is no cost to the respondents other than their time. ESTIMATED ANNUALIZED BURDEN HOURS Number of responses per respondent Number of respondents Average burden per response (in hours) Total burden (in hours) Type of respondents Form name Individuals (male and female) aged 18 years and older. Study Screener ............. Survey .......................... 10,740 2,148 1 1 2/60 30/60 358 1,074 Total ............................................................... ....................................... ........................ ........................ ........................ 1,432 Jeffrey M. Zirger, Acting Chief, Information Collection Review Office, Office of Scientific Integrity, Office of the Associate Director for Science, Office of the Director, Centers for Disease Control and Prevention. section of this notice that your comments address. 1.0 Background Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS). ACTION: Request for public comment. 1.1 Legislative Context The Consolidated Appropriations Act of 2017 (Pub. L. 115–31 (https:// www.congress.gov/115/plaws/publ31/ PLAW-115publ31.pdf)) directs the U.S. Department of Health and Human Services (HHS) to create a database of projects that have used a proven or promising approach to move welfare recipients into work, based on independent, rigorous evaluations of the projects, and to create a What Works Clearinghouse of Proven and Promising Projects to Move Welfare Recipients into Work. As stated in the statute, the database shall additionally ‘‘include a separate listing of projects that used a developmental approach in delivering services and a further separate listing of the projects with no or negative effects.’’ The statute requires HHS to establish criteria for evidence of effectiveness. The Administration for Children and Families, HHS, solicits comments by August 5, 2018 on the criteria for evidence of effectiveness for the What Works Clearinghouse of Proven and Promising Projects to Move Welfare Recipients into Work. Final criteria for evidence of effectiveness will be used to develop the clearinghouse. SUPPLEMENTARY INFORMATION: Invitation to Comment: HHS invites comments regarding this notice on the proposed criteria for HHS’s systematic review of the evidence. To ensure that your comments are clearly stated, please identify the specific criterion or other 1.2 The Legislation’s Direction for Establishing the Criteria for Evidence of Effectiveness Section 413(g)(2) of Public Law 115– 31 charges the Secretary of Health and Human Services with establishing the criteria of effectiveness. The statute further stipulated that the (B) process for establishing the criteria— (i) is transparent; (ii) is consistent across agencies; (iii) provides opportunity for public comment; and (iv) takes into account efforts of Federal agencies to identify and publicize effective interventions, including efforts at the Department of [FR Doc. 2018–12082 Filed 6–5–18; 8:45 am] BILLING CODE 4163–18–P DEPARTMENT OF HEALTH AND HUMAN SERVICES Administration for Children and Families Criteria for Evidence of Effectiveness To Be Applied to Projects Identified for Inclusion in the What Works Clearinghouse of Proven and Promising Projects To Move Welfare Recipients Into Work AGENCY: daltland on DSKBBV9HB2PROD with NOTICES SUMMARY: VerDate Sep<11>2014 17:35 Jun 05, 2018 Jkt 244001 PO 00000 Frm 00039 Fmt 4703 Sfmt 4703 Health and Human Services, the Department of Education, and the Department of Justice. 1.3 The Employment Strategies for LowIncome Adults Evidence Review Prior to the enactment of Public Law 115–31, the Office of Planning, Research, and Evaluation (OPRE) at the Administration for Children and Families (ACF) at HHS had developed the Employment Strategies for LowIncome Adults Evidence Review (ESER). The new statute aligns with and extends the work of ESER. HHS proposes building on this existing work to develop the new Clearinghouse. The Employment Strategies for LowIncome Adults Evidence Review (ESER) is a systematic review of the evaluation research published between 1990 and 2014 on employment and training programs for low-income adults. It culminated in a searchable, public database (https:// employmentstrategies.acf.hhs.gov/). The review was supplemented with briefs synthesizing the results of the review and highlighting strategies that appeared to be promising, as identified by the review. To identify the programs and strategies—or interventions— that appear to be most effective in helping low-income adults gain and retain employment, ESER systematically identified, assessed, and synthesized evidence from the existing evaluation research literature. A core component of ESER’s review, as with other federal evidence reviews, involved assessing the quality of the research evidence on different interventions. To assess the quality of the evidence, ESER reviewed each study’s methods to E:\FR\FM\06JNN1.SGM 06JNN1 daltland on DSKBBV9HB2PROD with NOTICES Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices determine if they were rigorous enough to ensure that the study’s findings could be considered reliable. ESER assessed whether the study’s methods reliably supported the conclusion that an intervention’s impacts were caused by the intervention and not by something else. The standards for assessing studies’ methods were defined based on consultation with federal experts on evidence reviews and researchers with expertise in evaluation methodology. To differentiate among different levels of the strength of evidence, ESER assigned a High, Moderate, or Low rating to each study reviewed. 246 of the 314 studies included in the review earned a High rating and 1 study earned a Moderate rating. The remaining 67 studies received a Low rating. Through this review, ESER was able to identify interventions whose findings could be considered most reliable. Studies’ ratings reflect the rigor of their study methods, independent of whether the findings were positive or negative. As a result, a study could be rated High or Moderate even if the intervention studied did not improve the outcomes for low-income adults. While the vast majority of studies included in ESER achieved a High rating (and, therefore, are considered to provide reliable, or strong, evidence), the review also found that, overall, null impacts were more prevalent than statistically significant impacts. While ESER did not assess the effectiveness of the interventions reviewed, ESER conducted a number of preliminary steps necessary for assessing effectiveness. This included categorizing each study’s findings according to whether it found positive, negative or null impacts for the interventions studied. In addition, through a number of synthesis briefs (published on the website), ESER qualitatively and quantitatively summarized the direction of impacts for different interventions and highlighted interventions associated with the greatest number of positive impacts. To be included in ESER, studies had to— • Quantitatively measure the effectiveness of a program or strategy • Be published between 1990 and 2014 • Study an employment program or strategy— an intervention— that Æ had a primary aim of improving employment-related outcomes Æ primarily targeted low-income adults Æ took place in the United States, Canada, or the United Kingdom To identify studies eligible for review, ESER issued a call for papers, VerDate Sep<11>2014 17:35 Jun 05, 2018 Jkt 244001 conducted literature searches, and consulted with experts in workforce development programs that serve lowincome adults. ESER looked at the effects of the interventions on four domains, or outcome areas: • Employment • Earnings • Public benefit receipt • Education/training Outcomes were examined for short and longer-term impacts (longer-term was measured as being more than 18 months after the intervention was implemented). The ESER website (https:// employmentstrategies.acf.hhs.gov/) reports key results for all eligible studies. The website also allows users to search for results by program studied, target population, outcome(s) of interest, service strategies, intervention setting, year of study publication, and whether favorable impacts were found. While ESER’s overall population of interest was low-income adults, a majority of the studies in ESER examined welfare populations. Because studies of interventions in a welfare setting typically include both recipients and applicants, ESER does not include any studies that solely focused on welfare recipients. ESER does, however, include interventions targeted to lowincome populations understood to share important characteristics with welfare recipients, such as other public benefit recipients, and those considered hard to employ, including those who have been homeless or formerly incarcerated. 2.0 Process for Establishing the Criteria of Effectiveness for the New What Works Clearinghouse In fall 2017 and early winter 2018, OPRE engaged in a series of systematic consultations with federal and nonfederal technical experts on evidence reviews. In addition to representation from the Department of Labor (DOL) and the Department of Education (ED) in these consultations, federal representation included the Department of Justice (DOJ) and a number of HHS agencies/offices including the Office of Family Assistance (OFA), the Office of Planning, Research, and Evaluation (OPRE), the Office of the Assistant Secretary for Planning and Evaluation (ASPE) and the Agency for Healthcare Research and Quality (AHRQ). The objective of these consultations was to help HHS: (1) Develop criteria for categorizing interventions in the new Clearinghouse as proven, promising, developmental, or ineffective, PO 00000 Frm 00040 Fmt 4703 Sfmt 4703 26291 (2) develop these criteria through a process that a. involved consultation with the Department of Labor (DOL), the Department of Education (ED), and other entities with experience evaluating relevant effectiveness research, b. allowed HHS to better understand other Federal evidence reviews’ standards and processes and determine where it would make sense for the new Clearinghouse to be consistent with these standards and processes, and (3) learn best practices from other Federal evidence reviews for identifying and publicizing effective interventions 2.1 Transparent To ensure that the Clearinghouse’s procedures and standards, including the criteria for evidence of effectiveness, are transparent, HHS intends to implement the following practices: • Post the procedures and standards and information about the process on the Clearinghouse website. • Provide the public a means of contacting the Clearinghouse, for example, by establishing a help desk to respond to email inquiries. 2.2 Consistent Across Agencies To ensure that the Clearinghouse is as consistent as possible with other federal evidence reviews in its processes and standards, HHS intends to implement the following practices: • Adopt the standards and methods for reviewing studies from OPRE’s existing Employment Strategies Evidence Review (ESER) (https:// www.acf.hhs.gov/opre/resource/ employment-strategies-for-low-incomeadults-evidence-review-standards-andmethods), which are broadly consistent with other federal Clearinghouses. ESER’s standards and methods (e.g., author queries; number and training of reviewers; choices about reporting effect sizes) were developed by considering both the choices made by other federal and non-federal Clearinghouses and the standards of research in the employment and training intervention field. Other existing federal Clearinghouses have followed this same approach (considering both the choices made by other clearinghouses and the norms of research within their fields of focus). • In any instances where the new Clearinghouse’s ratings of a project’s strength of evidence or effectiveness differ from another federal evidence review that rates projects according to the same outcomes (such as the Department of Labor’s (DOL’s) Clearinghouse for Labor Evaluation and E:\FR\FM\06JNN1.SGM 06JNN1 26292 Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices Research (CLEAR)), annotate the findings to explain the reason for the difference. 3.2 Proposed Criteria for Evidence of Effectiveness for Projects Included in the Clearinghouse 3.2.2.3 Pre-Defining Criteria for Selecting Among Multiple Outcome Measures 2.3 Provides Opportunity for Public Comment 3.2.1 Definition of Project and Approach The legislation requires that ratings, or categorizations, of evidence of effectiveness be applied to projects and approaches. To standardize definitions for these terms, HHS intends to define a project and an approach as follows: • Define project to be a specific bundle of services and/or policies implemented in a given context. • Project will be the unit that receives an effectiveness rating (i.e. proven, promising, developmental, or ineffective). • Define approach to be the guiding framework of specific services (e.g., career pathways). • Approaches will not be rated as proven, promising, developmental, or ineffective, but the Clearinghouse will include narrative summaries related to different approaches. • While the legislation does not require HHS to define or evaluate the effectiveness of program components, there is interest in the field in examining program components. Thus, HHS intends that the Clearinghouse include meta-analyses of specific components of projects (such as ‘‘case management’’ or ‘‘job search assistance’’) whenever appropriate and feasible. HHS intends to reduce the likelihood for reporting a false positive rate for outcomes—an issue that can occur when studies use multiple measures or multiple outcomes to assess impacts in the same domain (e.g., short-term earnings)—by relying on the decision rules ESER developed to address the potential for multiple comparisons. https://www.acf.hhs.gov/opre/resource/ employment-strategies-for-low-incomeadults-evidence-review-standards-andmethods). To provide an opportunity for public comment on the criteria for effectiveness, ACF is publishing this Federal Register Notice. 2.4 Takes Into Account Efforts of Federal Agencies To Identify and Publicize Effective Interventions To ensure the Clearinghouse reflects the learning of other Federal agencies about how to identify and publicize effective interventions, HHS intends to implement the following practices: • Use some of the methods adopted by other clearinghouses to create multiple products tailored to different audiences and use graphic design and other user-friendly dissemination elements to help users digest evidence quickly. • Include information on the Clearinghouse website that is especially useful to practitioners, such as summary information about projects and approaches. • Develop and incorporate alternative media for the Clearinghouse such as videos that will tailor communication to various groups. • Ensure that information is effectively conveyed on the Clearinghouse website by soliciting feedback from various stakeholders who can represent key target audiences. Key among these would be state or county Temporary Assistance for Needy Families (TANF) and Workforce Development practitioners, as well as evaluation researchers. 3.0 Proposed Criteria for Evidence of Effectiveness daltland on DSKBBV9HB2PROD with NOTICES 3.1 Criteria for Well-Designed, Rigorous Impact Research HHS intends to employ the criteria established by OPRE’s Employment Strategies for Low-Income Adults Evidence Review (ESER) to assess the quality of study design and to assess the strength of the evidence resulting from studies. These criteria (referred to as ‘‘standards and methods’’) are available in ESER’s Standards and Methods report https://www.acf.hhs.gov/opre/resource/ employment-strategies-for-low-incomeadults-evidence-review-standards-andmethods. VerDate Sep<11>2014 17:35 Jun 05, 2018 Jkt 244001 3.2.2 Parameters Guiding the Application of Evidence of Effectiveness Ratings Before a project can be categorized as being proven, promising, developmental, or ineffective, a number of preliminary definitions, or parameters, must be established to guide decision making. These include the outcomes for which a project’s effectiveness will be evaluated, how a favorable or unfavorable effect will be measured, and how an effectiveness rating will be applied to a project. 3.2.2.1 Outcomes HHS intends that the new Clearinghouse will review the following outcomes: Æ Employment (short and longerterm), Æ earnings (short and longer-term), Æ educational attainment, and Æ public benefit receipt. 3.2.2.2 Definition of Favorable and Unfavorable Effects HHS intends that the Clearinghouse consider only statistically significant findings (p <.05) as evidence of favorable or unfavorable effects. PO 00000 Frm 00041 Fmt 4703 Sfmt 4703 3.2.2.4 Application of Evidence of Effectiveness Ratings HHS intends that evidence of effectiveness ratings will be applied within outcome domains; each project will receive ratings of effectiveness on each outcome domain (e.g., a project may be found promising for short-term employment but ineffective for longterm employment). There will be no overall rating for the project. 3.2.3 Definition of Proven The legislation directs HHS to categorize projects as Proven, Promising, Developmental, or Ineffective. HHS intends that for a project to be considered proven, the following conditions must be met: • There must be at least two separate studies of the same project that meet evidence standards and meet criteria for a promising rating. Æ Studies are considered to be separate studies of the same project if they use non-overlapping samples to examine distinct implementations of the project. • There must be only favorable or null impacts within a given outcome domain. Thus, no studies that meet evidence standards for a given outcome domain can show an unfavorable impact within that domain. • Projects that have both favorable and unfavorable impacts in a given domain will be categorized as mixed. • A project has a limited number, or proportion, of null findings in a given domain. HHS is soliciting comments on how to best determine the ceiling for the number, or proportion, of null to positive findings in a given domain. If subsequent studies or replications result in only null findings in a given domain, the review will establish procedures for revisiting a project’s rating of proven. E:\FR\FM\06JNN1.SGM 06JNN1 Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices 3.2.4 Definition of Promising HHS intends that for a project to be considered promising, the following conditions must be met: • One study of a project must meet evidence standards. • That study must find only favorable or null impacts within a given outcome domain. Thus no studies that meet evidence standards for an outcome domain can show an unfavorable impact within the domain. Æ If the review examines more than one measure to identify impacts on a particular domain (e.g., Unemployment Insurance data and participant survey data), as long as one measure (among those selected according to 3.2.2.3 above) finds favorable impacts for that outcome, the intervention can receive a Promising rating for that outcome. • Projects that have both favorable and unfavorable impacts in a given domain will be categorized as mixed. 3.2.5 Definition of Ineffective HHS intends that for a project to be considered ineffective, the following conditions must be met: • One or more studies of a project must meet evidence standards. • There must be only findings of unfavorable or null effects in a given domain. • For studies finding null effect in a given domain, the review will include a measure of statistical precision—so that small, under-powered studies do not drive the effectiveness rating. If an intervention has been evaluated using only small studies, a lack of detectable effects could reflect either ineffectiveness of the intervention or the lack of statistical power to detect effects. It would be misleading to characterize this latter scenario as an ineffective project. 3.2.6 Definition of Developmental HHS intends that for a project to be considered developmental, the following conditions must be met: • There must be at least one current, ongoing evaluation of the project that uses a study design that meets evidence standards but has not yet produced impact findings. daltland on DSKBBV9HB2PROD with NOTICES 3.2.7 Additional Category of Mixed and Definition of Mixed HHS intends that there be an additional category for categorizing evidence of effectiveness called mixed. HHS proposes that for a project to be considered mixed, the following conditions must be met: • One or more studies of a project must meet evidence standards. VerDate Sep<11>2014 17:35 Jun 05, 2018 Jkt 244001 • The studies find both favorable and unfavorable impact estimates within the same domain. 3.2.8 HHS intends that narrative descriptions of rated projects, narrative descriptions of approaches, and information on case studies be provided to users of the Clearinghouse to facilitate a fuller understanding of the field of welfare-to-work interventions. 4.0 Submission of Comments Comments may be submitted until August 5, 2018 by email to OPREinfocollection@acf.hhs.gov. Naomi Goldstein, Deputy Assistant Secretary for Planning, Research, and Evaluation. [FR Doc. 2018–12160 Filed 6–5–18; 8:45 am] BILLING CODE 4184–09–P DEPARTMENT OF HEALTH AND HUMAN SERVICES Health Resources and Services Administration Notice of Single Source Award Based on Non-Statutory Earmark to the Delta Region Community Health Systems Development Program Health Resources and Services Administration (HRSA), Department of Health and Human Services. ACTION: Notice. AGENCY: The purpose of the Delta Region Community Health Systems Development Program is to support collaboration with and input from the Delta Regional Authority to develop a pilot program to help underserved rural communities in the Delta region identify and better address their health care needs and to help small rural hospitals improve their financial and operational performance. HRSA received an additional $2,000,000 in FY 2018 to support the Delta Region Community Health Systems Development Program, increasing the total FY 2018 resources from $2,000,000 to $4,000,000. The single award recipient, the Rural Health Resource Center has a need for additional funds to support activities performed within the scope of this program. The center will use a multipronged approach to deliver phased-in technical assistance (TA) to all eight Delta Region communities. ADDRESSES: Further information on the Delta Region Community Health Systems Development Program is available at: https://www.hrsa.gov/ ruralhealth/programopportunities/ fundingopportunities/?id=8d869eff0bca-4703-a821-88a9f0433b73. SUMMARY: PO 00000 Frm 00042 Fmt 4703 Sfmt 9990 26293 FOR FURTHER INFORMATION CONTACT: Rachel Moscato, Program Coordinator, Delta Region Community Health Systems Development, Federal Office of Rural Health Policy, HRSA, RMoscato@ hrsa.gov. Background The Delta Region Community Health Systems Development program is authorized by Section 711(b) of the Social Security Act, (42 U.S.C. 912 (b)), as amended. HRSA established the Delta Region Community Health Systems Development Program in FY 2017, under announcement HRSA–17–117, providing up to $2,000,000 per year to one awardee, the Rural Health Resource Center for a three-year project period: September 30, 2017 through September 29, 2020. The FY 2018 House Report 115–244 and Senate Report 115–150 Division H of the Consolidated Appropriations Act of 2018 (Pub. L. 115–141) provided direction that an additional $2,000,000 included in the appropriation to be used to support the Delta Program. HRSA plans to increase the maximum funding per year for the Delta Region Community Health Systems Development Program to $4,000,000 for one award recipient in FY 2018, as well as in subsequent budget periods within the three-year project period, should funds become available. Conclusion HRSA will provide $2,000,000 in additional resources to the current award recipient, the Rural Health Resource Center in FY 2018 to support additional activities within the scope of the Delta Region Community Health Systems Development Program. The recipient will utilize its existing infrastructure and a multipronged approach to deliver intensive assistance to all eight Delta Region communities, including onsite assessments in financial, operational performance, and quality improvement in the areas of population health, social services, emergency medical services, and telehealth. Please direct any questions or concerns to RMoscato@hrsa.gov. Dated: May 31, 2018. George Sigounas, Administrator. [FR Doc. 2018–12141 Filed 6–5–18; 8:45 am] BILLING CODE 4165–15–P E:\FR\FM\06JNN1.SGM 06JNN1

Agencies

[Federal Register Volume 83, Number 109 (Wednesday, June 6, 2018)]
[Notices]
[Pages 26290-26293]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2018-12160]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Criteria for Evidence of Effectiveness To Be Applied to Projects 
Identified for Inclusion in the What Works Clearinghouse of Proven and 
Promising Projects To Move Welfare Recipients Into Work

AGENCY: Administration for Children and Families (ACF), U.S. Department 
of Health and Human Services (HHS).

ACTION: Request for public comment.

-----------------------------------------------------------------------

SUMMARY: The Administration for Children and Families, HHS, solicits 
comments by August 5, 2018 on the criteria for evidence of 
effectiveness for the What Works Clearinghouse of Proven and Promising 
Projects to Move Welfare Recipients into Work. Final criteria for 
evidence of effectiveness will be used to develop the clearinghouse.

SUPPLEMENTARY INFORMATION: 
    Invitation to Comment: HHS invites comments regarding this notice 
on the proposed criteria for HHS's systematic review of the evidence. 
To ensure that your comments are clearly stated, please identify the 
specific criterion or other section of this notice that your comments 
address.

1.0 Background

1.1 Legislative Context

    The Consolidated Appropriations Act of 2017 (Pub. L. 115-31 
(https://www.congress.gov/115/plaws/publ31/PLAW-115publ31.pdf)) directs 
the U.S. Department of Health and Human Services (HHS) to create a 
database of projects that have used a proven or promising approach to 
move welfare recipients into work, based on independent, rigorous 
evaluations of the projects, and to create a What Works Clearinghouse 
of Proven and Promising Projects to Move Welfare Recipients into Work. 
As stated in the statute, the database shall additionally ``include a 
separate listing of projects that used a developmental approach in 
delivering services and a further separate listing of the projects with 
no or negative effects.'' The statute requires HHS to establish 
criteria for evidence of effectiveness.

1.2 The Legislation's Direction for Establishing the Criteria for 
Evidence of Effectiveness

    Section 413(g)(2) of Public Law 115-31 charges the Secretary of 
Health and Human Services with establishing the criteria of 
effectiveness. The statute further stipulated that the (B) process for 
establishing the criteria--
    (i) is transparent;
    (ii) is consistent across agencies;
    (iii) provides opportunity for public comment; and
    (iv) takes into account efforts of Federal agencies to identify and 
publicize effective interventions, including efforts at the Department 
of Health and Human Services, the Department of Education, and the 
Department of Justice.

1.3 The Employment Strategies for Low-Income Adults Evidence Review

    Prior to the enactment of Public Law 115-31, the Office of 
Planning, Research, and Evaluation (OPRE) at the Administration for 
Children and Families (ACF) at HHS had developed the Employment 
Strategies for Low-Income Adults Evidence Review (ESER). The new 
statute aligns with and extends the work of ESER. HHS proposes building 
on this existing work to develop the new Clearinghouse.
    The Employment Strategies for Low-Income Adults Evidence Review 
(ESER) is a systematic review of the evaluation research published 
between 1990 and 2014 on employment and training programs for low-
income adults. It culminated in a searchable, public database (https://employmentstrategies.acf.hhs.gov/). The review was supplemented with 
briefs synthesizing the results of the review and highlighting 
strategies that appeared to be promising, as identified by the review. 
To identify the programs and strategies--or interventions-- that appear 
to be most effective in helping low-income adults gain and retain 
employment, ESER systematically identified, assessed, and synthesized 
evidence from the existing evaluation research literature. A core 
component of ESER's review, as with other federal evidence reviews, 
involved assessing the quality of the research evidence on different 
interventions.
    To assess the quality of the evidence, ESER reviewed each study's 
methods to

[[Page 26291]]

determine if they were rigorous enough to ensure that the study's 
findings could be considered reliable. ESER assessed whether the 
study's methods reliably supported the conclusion that an 
intervention's impacts were caused by the intervention and not by 
something else. The standards for assessing studies' methods were 
defined based on consultation with federal experts on evidence reviews 
and researchers with expertise in evaluation methodology. To 
differentiate among different levels of the strength of evidence, ESER 
assigned a High, Moderate, or Low rating to each study reviewed. 246 of 
the 314 studies included in the review earned a High rating and 1 study 
earned a Moderate rating. The remaining 67 studies received a Low 
rating.
    Through this review, ESER was able to identify interventions whose 
findings could be considered most reliable. Studies' ratings reflect 
the rigor of their study methods, independent of whether the findings 
were positive or negative. As a result, a study could be rated High or 
Moderate even if the intervention studied did not improve the outcomes 
for low-income adults. While the vast majority of studies included in 
ESER achieved a High rating (and, therefore, are considered to provide 
reliable, or strong, evidence), the review also found that, overall, 
null impacts were more prevalent than statistically significant 
impacts.
    While ESER did not assess the effectiveness of the interventions 
reviewed, ESER conducted a number of preliminary steps necessary for 
assessing effectiveness. This included categorizing each study's 
findings according to whether it found positive, negative or null 
impacts for the interventions studied. In addition, through a number of 
synthesis briefs (published on the website), ESER qualitatively and 
quantitatively summarized the direction of impacts for different 
interventions and highlighted interventions associated with the 
greatest number of positive impacts.
    To be included in ESER, studies had to--
     Quantitatively measure the effectiveness of a program or 
strategy
     Be published between 1990 and 2014
     Study an employment program or strategy-- an 
intervention-- that
    [cir] had a primary aim of improving employment-related outcomes
    [cir] primarily targeted low-income adults
    [cir] took place in the United States, Canada, or the United 
Kingdom
    To identify studies eligible for review, ESER issued a call for 
papers, conducted literature searches, and consulted with experts in 
workforce development programs that serve low-income adults.
    ESER looked at the effects of the interventions on four domains, or 
outcome areas:
     Employment
     Earnings
     Public benefit receipt
     Education/training
    Outcomes were examined for short and longer-term impacts (longer-
term was measured as being more than 18 months after the intervention 
was implemented).
    The ESER website (https://employmentstrategies.acf.hhs.gov/) 
reports key results for all eligible studies. The website also allows 
users to search for results by program studied, target population, 
outcome(s) of interest, service strategies, intervention setting, year 
of study publication, and whether favorable impacts were found.
    While ESER's overall population of interest was low-income adults, 
a majority of the studies in ESER examined welfare populations. Because 
studies of interventions in a welfare setting typically include both 
recipients and applicants, ESER does not include any studies that 
solely focused on welfare recipients. ESER does, however, include 
interventions targeted to low-income populations understood to share 
important characteristics with welfare recipients, such as other public 
benefit recipients, and those considered hard to employ, including 
those who have been homeless or formerly incarcerated.

2.0 Process for Establishing the Criteria of Effectiveness for the New 
What Works Clearinghouse

    In fall 2017 and early winter 2018, OPRE engaged in a series of 
systematic consultations with federal and non-federal technical experts 
on evidence reviews. In addition to representation from the Department 
of Labor (DOL) and the Department of Education (ED) in these 
consultations, federal representation included the Department of 
Justice (DOJ) and a number of HHS agencies/offices including the Office 
of Family Assistance (OFA), the Office of Planning, Research, and 
Evaluation (OPRE), the Office of the Assistant Secretary for Planning 
and Evaluation (ASPE) and the Agency for Healthcare Research and 
Quality (AHRQ).
    The objective of these consultations was to help HHS:
    (1) Develop criteria for categorizing interventions in the new 
Clearinghouse as proven, promising, developmental, or ineffective,
    (2) develop these criteria through a process that
    a. involved consultation with the Department of Labor (DOL), the 
Department of Education (ED), and other entities with experience 
evaluating relevant effectiveness research,
    b. allowed HHS to better understand other Federal evidence reviews' 
standards and processes and determine where it would make sense for the 
new Clearinghouse to be consistent with these standards and processes, 
and
    (3) learn best practices from other Federal evidence reviews for 
identifying and publicizing effective interventions

2.1 Transparent

    To ensure that the Clearinghouse's procedures and standards, 
including the criteria for evidence of effectiveness, are transparent, 
HHS intends to implement the following practices:
     Post the procedures and standards and information about 
the process on the Clearinghouse website.
     Provide the public a means of contacting the 
Clearinghouse, for example, by establishing a help desk to respond to 
email inquiries.

2.2 Consistent Across Agencies

    To ensure that the Clearinghouse is as consistent as possible with 
other federal evidence reviews in its processes and standards, HHS 
intends to implement the following practices:
     Adopt the standards and methods for reviewing studies from 
OPRE's existing Employment Strategies Evidence Review (ESER) (https://www.acf.hhs.gov/opre/resource/employment-strategies-for-low-income-adults-evidence-review-standards-and-methods), which are broadly 
consistent with other federal Clearinghouses. ESER's standards and 
methods (e.g., author queries; number and training of reviewers; 
choices about reporting effect sizes) were developed by considering 
both the choices made by other federal and non-federal Clearinghouses 
and the standards of research in the employment and training 
intervention field. Other existing federal Clearinghouses have followed 
this same approach (considering both the choices made by other 
clearinghouses and the norms of research within their fields of focus).
     In any instances where the new Clearinghouse's ratings of 
a project's strength of evidence or effectiveness differ from another 
federal evidence review that rates projects according to the same 
outcomes (such as the Department of Labor's (DOL's) Clearinghouse for 
Labor Evaluation and

[[Page 26292]]

Research (CLEAR)), annotate the findings to explain the reason for the 
difference.

2.3 Provides Opportunity for Public Comment

    To provide an opportunity for public comment on the criteria for 
effectiveness, ACF is publishing this Federal Register Notice.

2.4 Takes Into Account Efforts of Federal Agencies To Identify and 
Publicize Effective Interventions

    To ensure the Clearinghouse reflects the learning of other Federal 
agencies about how to identify and publicize effective interventions, 
HHS intends to implement the following practices:
     Use some of the methods adopted by other clearinghouses to 
create multiple products tailored to different audiences and use 
graphic design and other user-friendly dissemination elements to help 
users digest evidence quickly.
     Include information on the Clearinghouse website that is 
especially useful to practitioners, such as summary information about 
projects and approaches.
     Develop and incorporate alternative media for the 
Clearinghouse such as videos that will tailor communication to various 
groups.
     Ensure that information is effectively conveyed on the 
Clearinghouse website by soliciting feedback from various stakeholders 
who can represent key target audiences. Key among these would be state 
or county Temporary Assistance for Needy Families (TANF) and Workforce 
Development practitioners, as well as evaluation researchers.

3.0 Proposed Criteria for Evidence of Effectiveness

3.1 Criteria for Well-Designed, Rigorous Impact Research

    HHS intends to employ the criteria established by OPRE's Employment 
Strategies for Low-Income Adults Evidence Review (ESER) to assess the 
quality of study design and to assess the strength of the evidence 
resulting from studies. These criteria (referred to as ``standards and 
methods'') are available in ESER's Standards and Methods report https://www.acf.hhs.gov/opre/resource/employment-strategies-for-low-income-adults-evidence-review-standards-and-methods.

3.2 Proposed Criteria for Evidence of Effectiveness for Projects 
Included in the Clearinghouse

3.2.1 Definition of Project and Approach
    The legislation requires that ratings, or categorizations, of 
evidence of effectiveness be applied to projects and approaches. To 
standardize definitions for these terms, HHS intends to define a 
project and an approach as follows:
     Define project to be a specific bundle of services and/or 
policies implemented in a given context.
     Project will be the unit that receives an effectiveness 
rating (i.e. proven, promising, developmental, or ineffective).
     Define approach to be the guiding framework of specific 
services (e.g., career pathways).
     Approaches will not be rated as proven, promising, 
developmental, or ineffective, but the Clearinghouse will include 
narrative summaries related to different approaches.
     While the legislation does not require HHS to define or 
evaluate the effectiveness of program components, there is interest in 
the field in examining program components. Thus, HHS intends that the 
Clearinghouse include meta-analyses of specific components of projects 
(such as ``case management'' or ``job search assistance'') whenever 
appropriate and feasible.
3.2.2 Parameters Guiding the Application of Evidence of Effectiveness 
Ratings
    Before a project can be categorized as being proven, promising, 
developmental, or ineffective, a number of preliminary definitions, or 
parameters, must be established to guide decision making. These include 
the outcomes for which a project's effectiveness will be evaluated, how 
a favorable or unfavorable effect will be measured, and how an 
effectiveness rating will be applied to a project.
3.2.2.1 Outcomes
    HHS intends that the new Clearinghouse will review the following 
outcomes:
    [cir] Employment (short and longer-term),
    [cir] earnings (short and longer-term),
    [cir] educational attainment, and
    [cir] public benefit receipt.
3.2.2.2 Definition of Favorable and Unfavorable Effects
    HHS intends that the Clearinghouse consider only statistically 
significant findings (p <.05) as evidence of favorable or unfavorable 
effects.
3.2.2.3 Pre-Defining Criteria for Selecting Among Multiple Outcome 
Measures
    HHS intends to reduce the likelihood for reporting a false positive 
rate for outcomes--an issue that can occur when studies use multiple 
measures or multiple outcomes to assess impacts in the same domain 
(e.g., short-term earnings)--by relying on the decision rules ESER 
developed to address the potential for multiple comparisons. https://www.acf.hhs.gov/opre/resource/employment-strategies-for-low-income-adults-evidence-review-standards-and-methods).
3.2.2.4 Application of Evidence of Effectiveness Ratings
    HHS intends that evidence of effectiveness ratings will be applied 
within outcome domains; each project will receive ratings of 
effectiveness on each outcome domain (e.g., a project may be found 
promising for short-term employment but ineffective for long-term 
employment). There will be no overall rating for the project.
3.2.3 Definition of Proven
    The legislation directs HHS to categorize projects as Proven, 
Promising, Developmental, or Ineffective.
    HHS intends that for a project to be considered proven, the 
following conditions must be met:
     There must be at least two separate studies of the same 
project that meet evidence standards and meet criteria for a promising 
rating.
    [cir] Studies are considered to be separate studies of the same 
project if they use non-overlapping samples to examine distinct 
implementations of the project.
     There must be only favorable or null impacts within a 
given outcome domain. Thus, no studies that meet evidence standards for 
a given outcome domain can show an unfavorable impact within that 
domain.
     Projects that have both favorable and unfavorable impacts 
in a given domain will be categorized as mixed.
     A project has a limited number, or proportion, of null 
findings in a given domain.
    HHS is soliciting comments on how to best determine the ceiling for 
the number, or proportion, of null to positive findings in a given 
domain.
    If subsequent studies or replications result in only null findings 
in a given domain, the review will establish procedures for revisiting 
a project's rating of proven.

[[Page 26293]]

3.2.4 Definition of Promising
    HHS intends that for a project to be considered promising, the 
following conditions must be met:
     One study of a project must meet evidence standards.
     That study must find only favorable or null impacts within 
a given outcome domain. Thus no studies that meet evidence standards 
for an outcome domain can show an unfavorable impact within the domain.
    [cir] If the review examines more than one measure to identify 
impacts on a particular domain (e.g., Unemployment Insurance data and 
participant survey data), as long as one measure (among those selected 
according to 3.2.2.3 above) finds favorable impacts for that outcome, 
the intervention can receive a Promising rating for that outcome.
     Projects that have both favorable and unfavorable impacts 
in a given domain will be categorized as mixed.
3.2.5 Definition of Ineffective
    HHS intends that for a project to be considered ineffective, the 
following conditions must be met:
     One or more studies of a project must meet evidence 
standards.
     There must be only findings of unfavorable or null effects 
in a given domain.
     For studies finding null effect in a given domain, the 
review will include a measure of statistical precision--so that small, 
under-powered studies do not drive the effectiveness rating. If an 
intervention has been evaluated using only small studies, a lack of 
detectable effects could reflect either ineffectiveness of the 
intervention or the lack of statistical power to detect effects. It 
would be misleading to characterize this latter scenario as an 
ineffective project.
3.2.6 Definition of Developmental
    HHS intends that for a project to be considered developmental, the 
following conditions must be met:
     There must be at least one current, ongoing evaluation of 
the project that uses a study design that meets evidence standards but 
has not yet produced impact findings.
3.2.7 Additional Category of Mixed and Definition of Mixed
    HHS intends that there be an additional category for categorizing 
evidence of effectiveness called mixed. HHS proposes that for a project 
to be considered mixed, the following conditions must be met:
     One or more studies of a project must meet evidence 
standards.
     The studies find both favorable and unfavorable impact 
estimates within the same domain.
    3.2.8 HHS intends that narrative descriptions of rated projects, 
narrative descriptions of approaches, and information on case studies 
be provided to users of the Clearinghouse to facilitate a fuller 
understanding of the field of welfare-to-work interventions.

4.0 Submission of Comments

    Comments may be submitted until August 5, 2018 by email to 
[email protected].

Naomi Goldstein,
Deputy Assistant Secretary for Planning, Research, and Evaluation.
[FR Doc. 2018-12160 Filed 6-5-18; 8:45 am]
 BILLING CODE 4184-09-P