Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures, Draft Version 2.0, 73021-73029 [2023-23391]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Administration for Children and Families

[Federal Register Volume 88, Number 204 (Tuesday, October 24, 2023)]
[Notices]
[Pages 73021-73029]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2023-23391]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Title IV-E Prevention Services Clearinghouse Handbook of 
Standards and Procedures, Draft Version 2.0

AGENCY: Administration for Children and Families, Department of Health 
and Human Services.

ACTION: Request for public comment.

-----------------------------------------------------------------------

SUMMARY: The Administration for Children and Families (ACF), within the 
U.S. Department of Health and Human Services (HHS), oversees the Title 
IV-E Prevention Services Clearinghouse. ACF seeks comments on proposed 
changes and clarifications to existing standards and procedures in the 
Handbook of Standards and Procedures, Version 2.0.

DATES: The deadline for comments on this notice is November 24, 2023.

ADDRESSES: Interested parties may submit written questions, comments, 
and supplementary documents by email to [email protected] 
with ``Title IV-E Prevention Services Clearinghouse FRN comment'' in 
the subject line. To ensure that your comments have maximum effect, 
please identify clearly the section of the draft Handbook of Standards 
and Procedures, Version 2.0 that your comments address.
    Readers are referred to the full version of the draft Handbook of 
Standards and Procedures, Version 2.0 on the Clearinghouse website 
(https://preventionservices.acf.hhs.gov/resources/comment-draft-handbook).

SUPPLEMENTARY INFORMATION:

1.0 Background and Legislative Context

    The Family First Prevention Services Act (FFPSA) was signed into 
law as part of the Bipartisan Budget Act (H.R. 1892) on February 9, 
2018. FFPSA amended the Social Security Act (the Act) to enable use of 
Federal funds available under parts B and E of title IV of the Social 
Security Act to provide enhanced support to children and families and 
prevent foster care placements through the provision of evidence-based 
``mental health and substance abuse prevention and treatment services, 
in-home parent skill-based programs, and kinship navigator services.'' 
As described in the statutory language, these services and programs are 
intended ``for children who are candidates for foster care or who are 
pregnant or parenting foster youth and the parents or kin caregivers of 
the children.'' The Act requires an independent systematic review of 
evidence to designate programs and services as ``promising,'' 
``supported,'' and ``well-supported'' practices.
    In order to meet these requirements, ACF established the Title IV-E 
Prevention Services Clearinghouse (the Clearinghouse). The 
Clearinghouse carries out a systematic review process implemented by 
trained reviewers using consistent, transparent standards and 
procedures. The Handbook of Standards and Procedures, Version 1.0 
(https://preventionservices.acf.hhs.gov/review-process) provides a 
detailed description of the standards used to identify and review 
programs and services for the Clearinghouse and the procedures followed 
by the Clearinghouse staff. The Handbook of Standards and Procedures, 
Version 1.0 was informed by public comments submitted in response to 
Federal Register Notice 83 FR 29122 (https://www.federalregister.gov/documents/2018/06/22/2018-13420/decisions-related-to-the-development-of-a-clearinghouse-of-evidence-based-practices-in-accordance), 
consultations with research and practice experts, and the review 
processes developed and used by other prominent evidence 
clearinghouses.

2.0 Overview of 2021 Request for Public Comment on Title IV-E 
Prevention Services Clearinghouse Handbook of Standards and Procedures, 
Version 1.0

    ACF solicited feedback on the Prevention Services Clearinghouse 
Handbook of Standards and Procedures, Version 1.0 (subsequently 
referred to as Handbook Version 1.0) through a Federal Register Notice 
86 FR 37332 (https://www.federalregister.gov/documents/2021/07/15/2021-15065/title-iv-e-prevention-services-clearinghouse-handbook-of-standards-and-procedures) published on July 15, 2021. This comment 
period was open for 30 days and closed on August 16, 2021. One hundred 
four unique commenters submitted feedback, including 10 commenters from 
state and local child welfare agencies. Commenters included state and 
local government administrators, program and service developers, 
Federal staff, researchers and evaluators, foundation and non-profit 
organization staff, and other interested parties. ACF ensured the 
careful review and consideration of all of the comments in developing 
the draft Handbook of Standards and Procedures, Version 2.0 
(subsequently referred to as Handbook Version 2.0). Comments were 
considered within the context of the statutory requirements of FFSPA, 
the necessity to conduct a systematic, objective, and transparent 
evidence review, and resource considerations. The public comments 
informed discussions with a large number of experts whose comments were 
also considered in developing the proposed revisions.
    Summary of Comments. Comments highlighted how the standards and 
procedures specified in Handbook Version 1.0 might be revised to better 
reflect the goals and requirements of the Executive Order on Advancing 
Racial Equity and Support for Underserved Communities Through the 
Federal Government. For example, commenters recommended prioritizing 
the review of programs and services that have been implemented and/or 
studied with diverse populations (Section 2.2). Commenters also 
recommended engaging diverse individuals and those with lived 
experience to inform the systematic review process and allowing greater 
flexibility for culturally adapted programs and services. Commenters

[[Page 73022]]

recommended providing additional detail to clarify the existing 
standards and procedures. For example, comments requested technical 
clarification regarding the definition of an available written 
protocol, manual, or other documentation (Section 2.1.2), determination 
of the length of time after the end of treatment (Section 6.2.3), 
determination of whether program or service or study adaptations are 
substantial (Section 4.1.6), and calculations of effect size and 
statistical significance (Section 5.1.0). Commenters recommended 
broadening the definitions of the program or service areas (Section 
2.1.2) to be more inclusive regarding the types of programs and 
services that may be eligible for review. Commenters recommended 
broadening the definition of eligible comparison conditions (Section 
4.1.4) and making the design and execution standards (Chapter 5), 
particularly those related to baseline equivalence (Section 5.7), more 
flexible. Finally, commenters provided recommendations to ACF that did 
not pertain to the Clearinghouse. For example, comments recommended ACF 
provide further support and investment in building evidence, 
particularly of programs and services designed to serve communities of 
color and others disproportionally represented in the child welfare 
system as well as for kinship navigator programs.
    Summary of Proposed Revisions. The draft Handbook Version 2.0 aims 
to be responsive to the diversity of comments received, to enhance the 
transparency of the systematic review process, and to support efforts 
to advance equity in accordance with the Executive Order on Advancing 
Racial Equity and Support for Underserved Communities Through the 
Federal Government. For example, revised program or service area 
definitions (Section 2.1) are inclusive of a broader range of programs 
and services, new program or service prioritization criteria have been 
added to consider the child welfare relevance and diversity of 
populations served (Section 2.2) with similar criteria also added for 
study prioritization (Section 2.3), and the range of eligible 
comparison conditions for studies has been expanded to include studies 
that compare one intervention to another intervention (Section 4.1.7). 
Additional clarification and guidance are now provided on program or 
service and study adaptations, including new examples of how standards 
are applied to culturally adapted programs and services (Sections 2.3.2 
and 4.1.9). Clarification is also provided that eligible outcomes and 
outcome measures may be defined differently across studies to reflect 
the different ages, backgrounds, cultures, locations, and contexts of 
the study participants, with examples provided (Section 4.1.8). 
Formulae used in effect size and statistical significance calculations 
are now provided directly in the Handbook (Chapter 6) and additional 
guidance and clarification is provided on design confounds, including 
clarification that studies with a single provider unit shared across 
the intervention and comparison conditions are not considered a 
confound (Section 5.9.3). A broader range of options is provided for 
establishing baseline equivalence and low attrition randomized group 
design contrasts are no longer assessed for baseline equivalence 
(Section 5.7). The Handbook now provides additional information on how 
the risk of harm assessment is conducted, with additional 
considerations for cases where the comparison group receives another 
intervention (Section 7.2.1). Further, additional clarification on how 
time since the end of treatment is calculated is provided (Section 
7.2.3). The Handbook now clearly specifies how any member of the public 
can submit recommendations of programs or services for review or 
information about studies of those recommended programs and services to 
the Clearinghouse at any time (Chapters 1 and 3).
    Additional Relevant Activities. The Clearinghouse also intends to 
conduct additional activities to be responsive to public comments and 
to support efforts to advance equity in accordance with the Executive 
Order on Advancing Racial Equity and Support for Underserved 
Communities Through the Federal Government. First, the Clearinghouse is 
planning to display study participant characteristics on the program or 
service page of the Clearinghouse website. Display of participant 
characteristics is intended to promote transparency on the extent to 
which diverse populations are represented in research reviewed by the 
Clearinghouse. Second, the Clearinghouse plans to develop two new 
reports focused on equity. These two reports are intended to provide 
additional information about diverse populations included in studies of 
the programs and services that have been reviewed by the Clearinghouse 
and identify gaps in evidence. Third, enhanced activities are planned 
for future public calls for program and service recommendations in 
order to comprehensively identify culturally adapted and culturally 
grounded programs and services that may be eligible for review. The 
Clearinghouse plans to conduct targeted outreach to providers of 
culturally adapted and culturally grounded programs and services and 
community-based organizations serving diverse populations to improve 
engagement. The Clearinghouse also plans to clearly communicate in 
future public calls how the public, including community-based 
organizations and providers of culturally adapted and culturally 
grounded programs and services, can recommend programs and services and 
submit relevant studies of programs and services to the Clearinghouse. 
Further, the Clearinghouse plans to make future public call materials 
available in both English and Spanish. Fourth, the Clearinghouse 
intends to revise its author Reporting Guide to clarify recommended 
reporting related to culturally adapted and culturally grounded 
programs and services and the characteristics of their participants. 
Fifth, the Clearinghouse intends to revise existing resources for 
Clearinghouse users, such as its Frequently Asked Questions (FAQ) 
website section and fact sheet resources, with person-centered design 
principles to ensure information about the Clearinghouse and its 
standards and procedures are accessible. Sixth, the Clearinghouse plans 
to publicly post all programs and services that have been recommended 
for review and will continue to explore additional ways to improve 
transparency such as through data sharing.
    A comprehensive list of specific revisions and clarifications to 
the Clearinghouse's Standards and Procedures is provided in the 
following section. Subsequent chapter and section numbers all refer to 
the chapter and section numbering for the draft Handbook Version 2.0 
unless the text explicitly indicates a reference to Handbook Version 
1.0 chapter and section numbering.

3.0 Revisions and Clarifications to the Clearinghouse's Standards and 
Procedures in the Draft Handbook Version 2.0

3.1 Introduction

    The revised introduction includes a description of the 
Clearinghouse website and resources available on the website. This 
includes reference to the FAQ section that includes information on how 
members of the public can submit a program or service recommendation 
and how to provide information about studies to the Clearinghouse.

[[Page 73023]]

3.2 Chapter 1. Identify Programs and Services

    Revisions clarify that all program and service recommendations are 
retained for consideration, including those submitted during public 
calls and ad hoc recommendations submitted to the Prevention Services 
Clearinghouse inbox. Revisions also clarify that any member of the 
public may submit a program or service recommendation at any time to 
the Clearinghouse via email and that suggested information to include 
as part of a program or service recommendation can be found on the FAQ 
section of the Clearinghouse website. Additionally, this section now 
indicates that all programs and services identified as potential 
candidates for review will be posted on the Clearinghouse website.

3.3 Chapter 2. Prioritize and Select Programs and Services

3.3.1 Revisions and Clarifications to Program or Service Area 
Definitions (Section 2.1.1)

    Based on FRN feedback and consultation with experts in the fields 
of mental health, substance use, parenting and parent skill-based 
programs and services, kinship navigator programs, and child welfare, 
the draft Handbook Version 2.0 revised and clarified the in-home 
parent-skill based and substance use prevention and treatment program 
or service area definitions, as noted below.
     In-home parent skill-based programs and services. The 
revised definition is more flexible and now indicates that eligible 
programs and services involve direct intervention with a parent or 
caregiver and target parenting skills or other skills that can be 
applied to where the child resides, including in the home. The revised 
definition also clarifies that delivery of programs and services can 
occur in the home or other settings and defines necessary content for a 
program or services to be considered ``skill-based.''
    Revised examples of eligible and ineligible in-home skill-based 
programs and services are provided in Exhibit 2.3.
     Substance use prevention and treatment programs and 
services. The revised definition clarifies that programs or services:
     targeting recovery from substance use (as well as those 
targeting prevention, treatment, remediation, elimination and/or 
reduction of substance use or misuse) are eligible; and
     without client-oriented substance use prevention or 
treatment components, such as mass communications/media campaigns or 
interventions that solely target broader community-level or policy 
systems, remain not eligible.
    Revised examples of eligible and not eligible programs and services 
are now provided in Exhibit 2.2. Specifically, one new example 
clarifies that programs or services targeting parents or caregivers 
aiming to prevent substance use among children and youth are eligible.
    Minor wording changes were made to the kinship navigator program or 
service area definition for clarification purposes. Experts did not 
suggest any changes to eligible outcomes for kinship navigator programs 
and services.
    No changes were made to the mental health prevention and treatment 
programs and services definition. New examples of eligible and 
ineligible programs and services are provided in Exhibit 2.1.

3.3.2 Clarifications to Available Protocols, Manuals, or Other 
Documentation (Section 2.1.2)

    To be eligible for review by the Prevention Services Clearinghouse, 
programs and services must be clearly defined and replicable. To meet 
this criterion, programs and services must have available written or 
recorded protocols, manuals, or other documentation that describes how 
to implement or administer the practice (referred to subsequently in 
this notice as a ``manual'' for brevity). Revisions to this section 
clarify that materials to satisfy this requirement may be presented in 
a web-based format and that ``manual'' can include recorded videos or 
online learning systems if these materials describe how to implement or 
administer the practice. The Clearinghouse notes that, consistent with 
Handbook Version 1.0, there are no language requirements for manual 
eligibility.

3.3.3 Revisions and Clarifications to Program or Service Prioritization 
(Section 2.2)

    As of July 2023, the Prevention Services Clearinghouse has reviewed 
148 programs and services. Yet there remains a high volume of 
potentially eligible programs and services identified for review. As a 
result, the Prevention Services Clearinghouse must continue to 
prioritize programs and services for review. The draft Handbook Version 
2.0 continues to highlight the prioritization of programs and services 
with available evidence of eligibility and programs and services in 
active use (Section 2.2). New to this section is further clarification 
about additional prioritization considerations. These additional 
prioritization criteria were informed by recommendations from public 
comments and consultation with experts. Listed below are the additional 
prioritization criteria included in the draft Handbook Version 2.0.
     Number and source of program or service recommendations 
received;
     Child welfare relevance;
     Population(s) served;
     Previous evaluations and studies; and
     Implementation supports.
    The Clearinghouse continues to prioritize programs and services in 
a way that ensures representation across the four program and service 
areas. Additional clarification is provided in draft Handbook Version 
2.0 noting that the Clearinghouse assesses prioritization criteria by 
examining publicly available information, other clearinghouses' 
websites, and materials submitted with program or service 
recommendations.

3.3.4 Clarifications on Program or Service Selection (Section 2.3.1)

    Given the large volume of programs and services identified, 
resource considerations mean that not all programs and services can be 
selected for review at once. To help clarify the distinction between 
the prioritization and reviewing process, the draft Handbook Version 
2.0 adds a new section on selection of a program or service for review 
(Section 2.3.1). Based on the prioritization process, specific programs 
and services are selected for review at a given time, as indicated by 
publication on the working list of programs and services planned for 
review available on the Prevention Services Clearinghouse website. The 
final eligibility of a program or service for review by the 
Clearinghouse is determined after a program or service is selected for 
the working list.

3.3.5 Revisions to Program or Service Adaptations Criteria (Section 
2.3.2)

    Multiple public comments requested clarification regarding the 
program or service adaptation standards specified in Handbook Version 
1.0 (found in Section 4.1.6 of this version) and recommended increased 
inclusivity, particularly with respect to cultural adaptations. The 
Prevention Services Clearinghouse sought input from a range of experts 
specifically focusing on program or service adaptations, including 
those with expertise in cultural adaptations designed to serve 
historically underserved communities. Underserved communities, as 
articulated in the Executive Order on Advancing Racial Equity and 
Support

[[Page 73024]]

for Underserved Communities Through the Federal Government, include 
Black, Latino, and Indigenous and Native American persons, Asian 
Americans and Pacific Islanders and other persons of color; members of 
religious minorities; lesbian, gay, bisexual, transgender, and queer 
(LGBTQ+) persons; persons with disabilities; persons who live in rural 
areas; and persons otherwise adversely affected by persistent poverty 
or inequality.
    To meet the eligibility criteria of being clearly defined and 
replicable, a program or service must have publicly available written 
or recorded protocols, manuals, or the documentation (hereafter 
referred to as ``manuals'') that describe how to implement the practice 
(Section 2.1.2). A new section (2.3.2) clarifies the procedures used to 
identify and review relevant manuals for a program or service. This 
includes procedures for identifying a primary manual for review and 
addressing cases with multiple potential manuals.
    Many programs and services have multiple manuals, including manual 
editions (e.g., editions of a manual as a program or service evolves 
over time or expands) and manual variants (e.g., adaptations of a 
program or service or a manual to address new issues, different 
populations, or alternative approaches to delivering the program or 
service). This section clarifies the standard process by which the 
Prevention Services Clearinghouse assesses whether alternative manual 
editions or variants have any substantial adaptations, compared to the 
primary manual identified. This process consists of the following 
steps, followed as needed based on the nature of the program or 
service:
     Step 1: Determining whether the adaptation is explicitly 
prohibited in the primary program or service manual under review or is 
the result of adding another separate program or service to the 
existing program or service (i.e., ``bundling'');
     Step 2: Determining whether the adaptation is explicitly 
allowed by the primary program or service manual under review;
     Step 3: Determining whether the adaptation substantially 
changes a program element in the primary program or service manual 
under review;
     Step 4: Gathering additional information and consulting 
with senior content experts on the Clearinghouse.
    A revised table (Exhibit 2.4) classifies program elements and gives 
examples of acceptable and substantial adaptations--including expanded 
examples of adaptations that may be made in the process of culturally 
adapting a program or service. (These criteria and procedures are 
aligned with those used to assess any program or service adaptations 
identified in studies during the study eligibility process, described 
in Section 4.1.9). Manuals that are substantially adapted from a 
primary manual may be considered as a separate program or service when 
reviewing studies. Studies with these substantial adaptations would be 
ineligible in a review based on the primary manual identified for a 
particular program or service. Alternatively, manuals without 
substantial adaptations may be considered the same program or service 
when reviewing studies. Studies without substantial adaptations would 
be included in a review based on the primary manual.

3.4 Chapter 3. Literature Search

    To help ensure identification of studies conducted with American 
Indian and Alaska Native populations, the draft Handbook Version 2.0 
adds Healthy Native Youth to its list of clearinghouses used to 
identify relevant research. The list of bibliographic databases has 
been trimmed for efficiency and resource considerations. Some databases 
in Handbook Version 1.0 were largely providing duplicative results. 
This section clarifies that any publicly available research from 
program or service websites is incorporated into the search. 
Clarification is also provided on procedures for incorporation of 
research that is submitted to the Prevention Services Clearinghouse 
inbox ad hoc or during public calls.

3.5 Chapter 4. Study Eligibility Screening and Prioritization

3.5.1 Revision to Study Definition (Section 4.1)

    In alignment with other Federal evidence clearinghouses, the 
Prevention Services Clearinghouse intends to focus on degree of sample 
overlap in applying its definition of a study as ``one research 
investigation of a defined subject sample, and the interventions, 
measures, and statistical analyses applied to that sample.'' Additional 
study definition criteria (based on the What Works Clearinghouse v4.0 
study definition) in Handbook Version 1.0 have been dropped in the 
draft Handbook Version 2.0.

3.5.2 Clarifications on Source of Publication Criteria (Section 4.1.2), 
Language of Publication (Section 4.1.3) and Location of Study (Section 
4.1.4)

    The draft Handbook Version 2.0 clarifies the definition of 
``publicly available'' and ``published'' for the source of publication 
standard (Section 4.1.2), in response to public comments. 
Dissertations, theses, and conference papers remain ineligible. Given 
the priority of reviewing a large number of programs and services, the 
Prevention Services Clearinghouse intends to continue to exclude such 
sources in the interests of efficiency.
    Some public comments indicated confusion about whether studies 
conducted outside of the United States or those conducted in non-
English-speaking countries are eligible. The draft Handbook Version 2.0 
clarifies that the standard from Handbook Version 1.0 that studies must 
be available in English (Section 4.1.3) is inclusive of studies 
originally published in another language that have published English 
language translations available. The draft Handbook Version 2.0 
explicitly clarifies that studies conducted in any country are eligible 
(Section 4.1.4), as they were under Handbook Version 1.0.

3.5.3 Revisions to Study Design and Intervention Condition Criteria 
(Sections 4.1.5, 4.1.6)

    The draft Handbook Version 2.0 provides clarification on 
definitions for randomized group designs and quasi-experimental group 
designs with respect to eligible study designs (Section 4.1.5). It 
clarifies that single-group pretest-posttest designs and interrupted 
time series designs without comparison groups are not eligible. It also 
clarifies that group assignment must be exclusive for an outcome 
measured at a given point in time--that is, participants cannot be 
counted in both the intervention and comparison condition. The 
criterion for eligible intervention conditions--that the intervention 
group is offered an eligible program or service that is essentially the 
same for all participants in the group--remains the same as in Handbook 
Version 1.0, with minor clarifications, but is presented as a distinct 
subsection in the draft Handbook Version 2.0 (Section 4.1.6) for 
clarity.

3.5.4 Revisions to Eligible Comparison Conditions (Section 4.1.7)

    Many public comments requested expansion of eligible study 
comparison conditions beyond no or minimal treatment and treatment as 
usual to

[[Page 73025]]

include more active comparison conditions. Many experts also 
recommended that the Prevention Services Clearinghouse consider 
including active comparison conditions. One consideration voiced by 
multiple experts consulted is that active comparison conditions are 
increasingly recommended, particularly if there are other available 
interventions considered to be efficacious. Revision to this standard 
was considered in the context of the FFSPA legislative criterion that a 
program or service must be demonstrated as being superior to an 
appropriate comparison practice.
    The draft Handbook Version 2.0 allows for five types of eligible 
comparison conditions:
     No intervention or wait list--offered no services or 
services at a later date (clarifying that outcomes measured after a 
wait list group is offered the intervention are not eligible).
     Minimal intervention--including informational materials or 
psychoeducation, referrals to available services, or similar nominal 
services.
     Placebo or attention control--conditions designed to 
account for nonactive effects of treatment, such as participants' 
expectations, contact time with an interventionist, or the relationship 
between interventionist and participants; includes psychological or 
pharmacological placebos, attention placebos, and nonspecific therapy 
in which participants receive the same or similar amount of attention 
or contact as the participants in the intervention condition.
     Treatment as usual--The draft Handbook Version 2.0 
clarifies that both ``usual or typical services'' (i.e., individuals do 
not receive anything they would not have been able to receive outside 
the context of the study) or ``services consistent with usual or 
typical services'' (i.e., services as part of the study that are not 
offered in the community but are clearly described as consistent with 
the usual or typical services that would be received by individuals or 
families similar to those in the study) are considered eligible under 
treatment as usual. Therapeutic or pharmacological interventions that 
meet the definition of treatment as usual are eligible.
     Head-to-head comparisons--assigned to another intervention 
that is not a variant of the program or service under review (may also 
be referred to as alternative interventions, active interventions, or 
comparator interventions); excluded are comparisons to pharmacological 
interventions that do not meet the definition of treatment as usual 
above.
    The draft Handbook Version 2.0 indicates three types of comparison 
conditions that are explicitly not eligible for review and provides a 
rationale for each:
     Intervention variants--assigned to an intervention that is 
a variation of the intervention under review. Examples include 
dismantling studies (e.g., full version of intervention compared to one 
lacking one or more components); bundled intervention studies (e.g., 
full version of intervention compared to a version with a second 
intervention added); studies comparing different delivery modes, 
providers, dosage, or fidelity levels for the same intervention; 
sequencing studies (e.g., both conditions receive the same 
interventions, but in a different order).
     Population-level data or benchmarks--constructed from 
population norms or statistics derived from other studies, surveys, 
censuses, or similar sources.
     Comprised only of intervention refusers or dropouts--
composed entirely of individuals who were offered the intervention 
condition but refused the offer or dropped out of the intervention 
after being offered the intervention.

3.5.5 Revisions to Outcomes (Section 4.1.8)

    Definitions of outcome domain, outcome, and outcome measurement 
have been provided for clarity. Clarifications have been included 
regarding eligible outcomes within the child safety and child 
permanency outcome domains and family functioning outcomes within the 
adult well-being outcome domain. The clarifications to the child safety 
and child permanency outcomes were previously described in the FAQ 
section of the Prevention Services Clearinghouse website. Additionally, 
eligible educational achievement and attainment outcomes in the child 
well-being outcome domain have been expanded to include school 
attendance and absenteeism as eligible outcomes. These outcomes, though 
not direct measures of educational achievement and attainment, are 
viewed as closely related and relevant outcomes. Clarification is 
provided that outcomes that are composites of one or more eligible 
outcomes within the eligible outcome domains are eligible; those that 
are composites of eligible and ineligible outcomes are not eligible. 
Clarification is also provided that eligible outcomes and outcome 
measures may be defined differently across studies to reflect the 
different ages, backgrounds, cultures, locations, and contexts of the 
study participants, with examples provided.
    The Prevention Services Clearinghouse currently does not have 
measurement standards for assessing the validity or reliability of 
biomarker measures (i.e., a physiological measure used as an indicator 
of a physical, psychological or emotional state), such as the use of 
cortisol as a measure of psychological stress. Expert consultations on 
biomarkers did not indicate a clear set of standards that could be 
broadly applied for review of such measures. As a result, the draft 
Handbook Version 2.0 indicates that biomarker measures are not 
currently eligible for review as child well-being or adult well-being 
outcomes.

3.5.6 Revisions to Study Program or Service Adaptations Criteria 
(Section 4.1.9)

    Consistent with Handbook Version 1.0, the draft Handbook Version 
2.0 indicates that, to be eligible for review, studies of a program or 
service must all represent similar implementations of the program or 
service selected for review. Revisions in the draft Handbook Version 
2.0 clarify that the process of assessing program or service 
adaptations for study eligibility is based on having identified a 
particular manual (or set of manuals) of the program or service under 
review (see Sections 2.3.1, 2.3.2).
    The standard process used to identify whether program or service 
adaptations are present in the studies being screened for eligibility 
is clarified. The procedures and criteria for assessing whether 
adaptations identified in studies are acceptable or substantial mirror 
those specified in Section 2.3.2 for adaptations found in manual 
editions or variants. The end result of these procedures is the 
determination of study eligibility for the particular program or 
service under review (in Section 2.3.2, the end determination is 
whether two manuals are substantively similar or represent different 
programs or services). Studies with any substantial adaptations are 
ineligible for review as a study of the program or service under review 
(such studies may be eligible for review as a study of different 
program or service and its associated manual). Studies with only minor 
adaptations may potentially be eligible if all other study eligibility 
criteria are met.

3.5.7 Revisions to Study Review Prioritization Criteria (Section 4.2)

    The Prevention Services Clearinghouse notes that study 
prioritization criteria are distinct from study eligibility criteria. 
When a

[[Page 73026]]

program or service has more than 15 studies eligible for review, study 
prioritization criteria are applied to order the review of eligible 
studies. The study prioritization process ensures efficiencies in the 
reviewing process to review a large number of programs and services.
    The Prevention Services Clearinghouse notes that only 12 of the 148 
programs and services reviewed as of July 2023 had more than 15 
eligible studies identified, requiring the use of study prioritization 
criteria in these reviews to prioritize the first 15 eligible studies 
for review using the design and execution standards. Of these 12 
programs and services, nine had 16 to 25 eligible studies, with a few 
having a much larger number of eligible studies (e.g., 75 or 90). All 
other programs and services reviewed had 15 or fewer eligible studies, 
with all eligible studies reviewed using the design and execution 
standards. Therefore, as in Handbook Version 1.0, the study 
prioritization criteria continue to apply only when there are 15 or 
more eligible studies of a program or service in the draft Handbook 
Version 2.0.
    Three modifications have been made to the process of assigning 
prioritization points for identifying the order in which studies are 
reviewed in the draft Handbook Version 2.0. First, given that programs 
or services must demonstrate sustained favorable effects 6 or 12 months 
beyond the end of treatment (Section 7.2.3) to receive a rating of 
supported or well-supported, the Prevention Services Clearinghouse 
intends to increase the prioritization points given to studies that 
include outcomes measured 6 or 12 months beyond the end of treatment to 
ensure that these studies are reviewed earlier when present, increasing 
the prioritization points for such studies to 3 and 6 points, 
respectively (compared to 1 and 2 points, respectively, in Handbook 
Version 1.0). Second, some public commenters and experts consulted 
noted the importance of statistical power for being able to detect 
intervention effects. The draft Handbook Version 2.0 adds one 
prioritization score point for studies that report an analysis of 
statistical power. Third, many public comments recommended that points 
be awarded to studies based on populations served. The draft Handbook 
Version 2.0 intends to add one prioritization score point for the child 
welfare relevance of populations served and two prioritization points 
for studies with samples from underserved communities. Prioritization 
points for studies with outcomes in multiple outcome domains have been 
decreased from a maximum of three to a maximum of one. The draft 
Handbook Version 2.0 provides procedural details clarifying how ties in 
prioritization scores are resolved in cases where more than 15 eligible 
studies are identified.
    The draft Handbook Version 2.0 includes efficiency enhancements 
based on the study prioritization process for programs and services 
where more than 15 eligible studies are identified. If, after review of 
the first 15 eligible studies prioritized for review, a program or 
service has not achieved a rating of well-supported, additional studies 
are reviewed using the design and execution standards in their 
prioritized order until either no eligible studies remain that could 
result in further improvement to the program or service rating or all 
eligible studies have been reviewed. Determination of potential for 
program or service ratings to improve upon review of additional 
eligible studies is based on (1) the program rating from studies 
already reviewed using the design and execution standards and (2) the 
duration of effects examined in the remaining studies (as assessed 
according to study review prioritization criteria). Detailed examples 
of the application of this policy are described in Section 4.2. The 
draft Handbook Version 2.0 retains the policy from Handbook Version 1.0 
of reviewing all studies against design and execution standards when 15 
or fewer eligible studies are identified. All eligible studies are 
reviewed for risk of harm.

3.6 Chapter 5. Evidence Review Using the Design and Execution Standards

3.6.1 Revisions and Clarifications to Contrasts Rated, Design and 
Execution Rating Categories, Method of Assignment, and Integrity of 
Random Assignment (Sections 5.1 to 5.5)

    The draft Handbook Version 2.0 indicates that contrasts from all 
eligible comparison conditions (Section 5.1) will be rated, whereas 
under Handbook Version 1.0, only contrasts from the least-intensive 
eligible comparison condition for a particular contrast were rated if 
multiple comparison conditions were eligible for review (Handbook 
Version 1.0, Section 4.1.4). Given the priority of reviewing a large 
number of programs and services, the draft Handbook Version 2.0 retains 
the policy from Handbook Version 1.0 of only reviewing full-sample 
analyses and not reviewing subgroup or sensitivity analyses due to 
resource considerations. For any studies that receive a moderate or 
high design and execution rating and report subgroup analyses, the 
Clearinghouse intends to indicate whether subgroup analyses were 
conducted for informational purposes only. New and revised examples are 
provided to clarify integrity of randomization standards for individual 
and cluster-assignment designs.

3.6.2 Revisions and Clarifications to Attrition, Baseline Equivalence, 
and Pretest Standards (Sections 5.6 to 5.8)

    Based on expert feedback, and in alignment with other Federal 
clearinghouses (in particular, the What Works Clearinghouse and Home 
Visiting Evidence of Effectiveness [HomVEE]), the draft Handbook 
Version 2.0 no longer requires baseline equivalence to be established 
for a contrast from a low attrition randomized group design to receive 
a ``High'' support of causal evidence rating.
    Public comments expressed a desire for greater flexibility 
regarding options for demonstrating baseline equivalence and 
reconsideration of participant sociodemographic characteristics that 
could be used to establish baseline equivalence when a pretest 
alternative is not available. Informed by expert consultations, the 
draft Handbook Version 2.0 maintains a general preference for using the 
same (or nearly the same) measure as the outcome (i.e., a ``direct 
pretest'') for baseline equivalence but now allows any eligible outcome 
measure demonstrated to be correlated with the outcome at a threshold 
of 0.60 or higher to be used to establish baseline equivalence (here 
referred to as a ``correlated pretest measure''). Also informed by 
expert feedback, when a correlated pretest measure or pretest 
alternative is not available, the draft Handbook Version 2.0 provides 
greater flexibility in the form of two options for establishing 
equivalence on sociodemographic characteristics, allowing an expanded 
set of individual characteristics and the use of a set of neighborhood 
characteristics if only one individual characteristic is available. 
Option 1 requires demonstration of baseline equivalence on at least two 
of the following individual characteristics: race or ethnicity, 
socioeconomic status, household composition, or age. If only one of the 
four individual characteristics from Option 1 is available, baseline 
equivalence can still be established under Option 2 if equivalence is 
demonstrated on a measure of each of the following neighborhood 
characteristics: race or ethnicity, socioeconomic status, and household 
composition. When sociodemographics are used to establish

[[Page 73027]]

baseline equivalence, a new requirement indicates that study authors 
must clearly describe all criteria used to create the intervention and 
comparison groups and affirmatively indicate that the same or similar 
criteria were used to create each group.
    Binary measures have different statistical properties than 
continuous measures that can potentially reduce their reliability as 
indicators of baseline equivalence--particularly when events are rare 
or in smaller samples. To address this, the draft Handbook Version 2.0 
indicates a preference for continuous correlated pretests over direct 
pretests when establishing baseline equivalence for a binary outcome. 
It also permits use of continuous pretest alternative measures when 
outcomes are binary, even if it was feasible to measure a direct 
pretest. Specifically, continuous measures that meet the correlated 
pretest measure or pretest alternative criteria are preferred over a 
direct pretest of the binary measure, when available.

3.6.3 Revisions and Clarifications to Statistical Model Standards 
(Section 5.9)

    The statistical model standards (Section 5.9.1) have been revised 
in the draft Handbook Version 2.0 to clarify procedures used when 
statistical models do not meet standards and alternative statistical 
models are not available or do not meet standards. In such cases, the 
Prevention Services Clearinghouse will seek to review the contrast 
based on unadjusted means and standard deviations and the statistical 
significance test procedures specified in Chapter 6.
    The measurement reliability standard for inter-rater reliability in 
Handbook Version 1.0 was revised in the draft Handbook Version 2.0 
(Section 5.9.2), with specific thresholds for inter-rater reliability 
(correlation), inter-rater agreement on the basis of percentage 
agreement (0.80 or higher), and inter-rater agreement based on kappa 
(0.60 or higher). These revised standards are aligned with current What 
Works Clearinghouse standards.
    Some public comments expressed concern that confound standards 
prevent inclusion of studies conducted in rural or underserved areas 
where only a single service provider is available may not be able to 
meet standards. The draft Handbook Version 2.0 clarifies that studies 
with a single person or administrative unit are not automatically 
confounded, with detailed clarifying examples added to this section. 
Specifically, if a single provider (or a single administrative unit) 
provides treatment or services to at least some participants in both 
the intervention and comparison condition, a design confound is not 
considered to be present. Expert feedback indicated that the confound 
standards in Handbook Version 1.0 were appropriate causal evidence 
standards, informing the retention of these confound standards in the 
draft Handbook Version 2.0.

3.7 Chapter 6. Record and Characterize Impact Estimates

    Public comments requested additional information about the formulae 
used for computing effect sizes and procedures used for determining 
statistical significance. The draft Handbook Version 2.0 provides all 
standard formulae used in computing effect sizes reported and for 
computing statistical significance. For models that meet statistical 
model standards in the design and execution requirements (Section 5.9), 
the draft Handbook Version 2.0 indicates that author-reported 
statistical significance is preferred in covariate-adjusted models and 
certain models for which the Prevention Services Clearinghouse does not 
currently have standards for computing statistical significance (e.g., 
time-to-event models). When such models are not available or do not 
meet statistical model standards, the formulae provided are used to 
conduct a post-hoc statistical significance test based on the natural 
metric of the outcome reported (e.g., continuous, binary, count, or 
time-to-event).
    Clarification is provided on information needed and procedures used 
to compute effect sizes and statistical significance for repeated 
measures models (e.g., growth curve models). In alignment with other 
Federal clearinghouses (in particular, What Works Clearinghouse, 
HomVEE), point-in-time estimates for each measurement time period are 
required. If such information is not reported, unadjusted means and 
standard deviations for each point in time are used (or requested if 
not reported), with appropriate post-hoc significance tests performed 
based on the natural metric of the outcome.

3.8 Chapter 7. Program or Service Ratings

3.8.1 Revisions and Clarifications to Program or Service Ratings 
(Section 7.1) and Risk of Harm (Section 7.2.1)

    No changes were made to the criteria for promising, supported, or 
well-supported program or service ratings in the draft Handbook Version 
2.0 (Section 7.1). This section clarifies that intention of the 
Prevention Services Clearinghouse is for program or service ratings 
from reviews conducted under Handbook Version 1.0 to be retained until 
such time that a program or service is re-reviewed under Handbook 
Version 2.0 (see Section 8.5.1 below regarding re-review procedures).
    A new standard specified in the risk of harm section (Section 
7.2.1) of the draft Handbook Version 2.0 is that contrasts in head-to-
head comparison conditions or placebo or attention control comparison 
conditions where the comparison condition has any evidence for risk of 
harm cannot contribute to a promising, supported, or well-supported 
rating. If risk of harm is present in these kinds of comparison 
conditions, impact estimates are not clearly interpretable as evidence 
of intervention effectiveness--as it is possible that both the 
intervention and comparison condition could be made worse off than if 
they had not participated in the study at all. When risk of harm is not 
present in the comparison condition, favorable impacts can be 
interpreted as the intervention group being at least better off than 
they would have been if no treatment had been offered at all and can 
potentially contribute as evidence of effectiveness. Standard 
procedures for identifying potential risk of harm in comparison 
conditions are detailed in this section.

3.8.2 Revisions and Clarifications to Usual Care or Practice Settings 
Definition (Section 7.2.2)

    The definition of usual care or practice settings (Section 7.2.2) 
in the draft Handbook Version 2.0 has been clarified to indicate that 
community settings, such as schools, with embedded service providers 
that may provide eligible programs or services as part of their typical 
operations (e.g., school counselors), are also considered usual care or 
practice settings. It clarifies that clinics that provide services 
solely for participants in research studies or clinical trials (i.e., 
that do not provide any services to persons not participating in 
research studies as part of their typical operations) do not constitute 
usual care or practice settings.

3.8.3 Revisions and Clarifications to Beyond the End of Treatment 
(Section 7.2.3)

    Some public comments requested clarification on how the Prevention 
Services Clearinghouse assesses the duration of sustained effects, 
particularly in cases where the end of

[[Page 73028]]

treatment is flexible across participants. Section 7.2.3 of the draft 
Handbook Version 2.0 includes revisions to clarify the order of 
preference for information that may be provided in studies about the 
end of treatment and procedures for computing the duration of sustained 
effects when the duration of treatment is fixed, when the duration of 
treatment is defined and varies across participants, and when the 
duration of treatment is undefined. Treatment of boosters in computing 
the duration of sustained effects is now explicitly addressed. Detailed 
procedures and examples can be found in Section 7.2.3 of the draft 
Handbook Version 2.0.

3.9 Chapter 8. Prevention Services Clearinghouse Procedures

    The draft Handbook Version 2.0 represents the first update to the 
Handbook of Standards and Procedures since the beginning of the Title 
IV-E Prevention Services Clearinghouse in 2018. The basic procedures 
for identifying eligible studies (Section 8.3) and reviewing studies 
against the design and execution standards (Section 8.4) remain 
essentially the same, with minor clarifications to operational 
procedures. Author query policies (Section 8.4.2) have been clarified; 
new content has been added clarifying the reasons that the Prevention 
Services Clearinghouse may query program and service developers for 
information about programs or services (Section 8.4.3). New content and 
more substantive revisions are described below.

3.9.1 Selection of Handbook of Standards and Procedures Version To Use 
in Reviews (Section 8.2)

    The intention of the Prevention Services Clearinghouse is to 
conduct reviews of any program or service not previously reviewed under 
Handbook Version 1.0 solely under the standards and procedures 
specified in Handbook Version 2.0 once it is finalized. Programs or 
services that are included on the working list prior to when Handbook 
Version 2.0 is finalized may be reviewed under Handbook Version 1.0 or 
Handbook Version 2.0. The version of the handbook used to conduct a 
review (or re-review) of a program or service will be clearly stated on 
the working list and on the program or service's review page on the 
Prevention Services Clearinghouse website.

3.9.2 Program and Service Re-Reviews and Study Re-Reviews (Sections 
8.5.1, 8.5.2)

    The Prevention Services Clearinghouse intends to conduct program 
and service re-reviews solely under Handbook Version 2.0 after it is 
finalized (Section 8.5.1). The intention of the Prevention Services 
Clearinghouse is that all existing program and service ratings 
determined under Handbook Version 1.0 will remain in effect until such 
time that a program or service re-review is conducted of a program or 
service.
    Programs and services reviewed by the Prevention Services 
Clearinghouse under Handbook Version 1.0 may be considered for re-
review under Handbook Version 2.0 if a re-review has the potential to 
change the program or service rating (Section 8.5.1). Program or 
service ratings could potentially change due to application of Handbook 
Version 2.0 standards to studies already identified in a prior review 
(e.g., studies previously ineligible now being eligible; studies being 
able to demonstrate baseline equivalence under revised standards) or 
the emergence of new evidence since the original review. The intention 
of the Prevention Services Clearinghouse is that the rating of a re-
reviewed program or service would be based solely on the standards and 
procedures in Handbook Version 2.0 (i.e., the previously assigned 
rating would no longer be in effect).
    The intention of the Prevention Service Clearinghouse is to conduct 
study re-reviews (i.e., due to missing information or errors in the 
currently published review of an individual study) under the version of 
the handbook used to review the program or service (Section 8.5.2). 
That is, for a program or service reviewed under Handbook Version 1.0 
where the program or service has not been re-reviewed under Handbook 
Version 2.0, a study re-review would be conducted under Handbook 
Version 1.0. For a program or service where a program or service rating 
has been assigned using Handbook Version 2.0, study re-reviews would be 
conducted using Handbook Version 2.0. This policy is consistent with 
other Federal evidence clearinghouses with multiple handbook versions 
(e.g., HomVEE). The Prevention Services Clearinghouse's intention is 
that the emergence of substantial new evidence that has the potential 
to change program or service ratings (e.g., a newly published study) 
should be addressed through a program or service re-review. Similarly, 
cases where application of Handbook Version 2.0 standards to a study 
reviewed under Handbook Version 1.0 could affect the program or service 
rating are intended to be addressed through a program or service re-
review. Study re-reviews are intended to be limited solely to 
addressing missing information or errors in studies already reviewed.

3.9.3 Manual Citation Updates (Section 8.5.3)

    The Prevention Services Clearinghouse recognizes that program or 
service manuals may be updated in the course of time after a review of 
a program or service has been published. Should a new manual edition 
(as defined in Section 2.3.2) be published, the public may request 
consideration of an update to the manual citation used for the program 
or service as outlined in Section 8.5.3 of the draft Handbook Version 
2.0. If updated manual editions do not have substantive modifications 
or adaptations from the manual reviewed (per the criteria specified in 
Section 2.3), a manual citation may be updated to reflect that a newer 
manual edition is in active use that is substantively similar to the 
original primary manual selected for the review of the program or 
service. In considering whether an update to a manual citation is 
warranted, the Prevention Services Clearinghouse must have sufficient 
information available to be able to apply the procedures specified in 
Section 2.3 for determining whether any substantive adaptations are 
present in the newer manual edition compared to the original edition 
reviewed. If the manual citation is updated, the original manual 
citation used to conduct the review of evidence for the program or 
service will also be noted for clarity.

4.0 Timeline for the Clearinghouse To Apply New Standards and 
Procedures

    The Prevention Services Clearinghouse proposes to apply the 
standards and procedures upon publication of a final Handbook Version 
2.0. The public will be clearly notified on the Prevention Services 
Clearinghouse website and via other avenues (e.g., email to subscribers 
to the Prevention Services Clearinghouse email list) when the final 
published Handbook Version 2.0 will go into effect for reviewing 
programs and services.
    Per the procedures in Chapters 7 and 8 of the draft Handbook 
Version 2.0, all existing program and service ratings established under 
Handbook Version 1.0 will remain in effect until such time that a 
program or service re-review is conducted of a program or service under 
Handbook Version 2.0.

5.0 Request for Information (RFI)

    To facilitate the review of submissions, please identify the 
chapter, section, and/or page number of the draft Handbook of Standards 
and Procedures, Version 2.0 (https://

[[Page 73029]]

preventionservices.acf.hhs.gov/resources/comment-draft-handbook) that 
your comments address. This RFI is for information and planning 
purposes only and should not be construed as a solicitation or as an 
obligation on the part of ACF or HHS. For more information about the 
Prevention Services Clearinghouse, visit: https://preventionservices.acf.hhs.gov.

Lauren Supplee,
Deputy Assistant Secretary for Planning, Research, and Evaluation.
[FR Doc. 2023-23391 Filed 10-23-23; 8:45 am]
BILLING CODE 4184-01-P