Revised Procedures and Standards: Home Visiting Evidence of Effectiveness (HomVEE) Review, 47376-47384 [2020-17001]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Administration for Children and Families

[Federal Register Volume 85, Number 151 (Wednesday, August 5, 2020)]
[Notices]
[Pages 47376-47384]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2020-17001]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Revised Procedures and Standards: Home Visiting Evidence of 
Effectiveness (HomVEE) Review

AGENCY: Administration for Children and Families, U.S. Department of 
Health and Human Services.

ACTION: Request for public comment.

-----------------------------------------------------------------------

SUMMARY: The Administration for Children and Families (ACF), within the 
U.S. Department of Health and Human Services (HHS), oversees the Home 
Visiting Evidence of Effectiveness

[[Page 47377]]

(HomVEE) review, which is proposing to revise the procedures and 
standards that guide its work. The revised procedures and standards 
will be presented in two separate Federal Register notices. The current 
Federal Register notice seeks comments on proposed changes and 
clarifications to several procedural topics and on the standards for 
assessing the quality of impact study designs. Readers are referred to 
the full version of the HomVEE Draft Version 2 Handbook on the HomVEE 
website (https://homvee.acf.hhs.gov) for more details. Another Federal 
Register notice summarizes updated definitions, rules, and procedures 
related to handling home visiting model versions (commonly referred to 
in the home visiting research literature as adaptations) in the review.

DATES: Send comments on or before September 1, 2020.

ADDRESSES: Submit questions, comments, and supplementary documents to 
[email protected] with ``HomVEE procedures and standards FRN comment'' 
in the subject line.

SUPPLEMENTARY INFORMATION: 
    Invitation to Comment: HHS invites comments regarding this notice. 
To ensure that your comments are clearly stated, please identify the 
section of this notice or the chapter and section of the HomVEE Draft 
Version 2 Handbook that your comments address.

1.0 Background

    To help policymakers, program administrators, model developers, 
researchers, and the public identify rigorous research and understand 
which early childhood home visiting models are effective, ACF's Office 
of Planning, Research, and Evaluation within HHS oversees the HomVEE 
review. HomVEE's mission is to conduct a thorough and transparent 
review of the research literature on home visiting for families with 
pregnant women and children from birth to kindergarten entry. The 
review team identifies well-designed research within that pool and 
extracts and summarizes the findings from that research.
    One critical use of HomVEE's results is to determine which home 
visiting models meet the HHS criteria for an ``evidence-based early 
childhood home visiting service delivery model'' (see Exhibit II.11 in 
the HomVEE Draft Version 2 Handbook), a key requirement of eligibility 
for implementation with the Maternal, Infant, and Early Childhood Home 
Visiting (MIECHV) Program funding. The MIECHV Program is administered 
by the Health Resources and Services Administration in partnership with 
ACF. Created in 2010, the MIECHV Program provides funding to states, 
territories, and tribal entities to implement home visiting models. 
MIECHV awardees have the flexibility to tailor the program to serve the 
specific needs of their communities. Through a needs assessment, 
awardees identify at-risk communities and select home visiting service 
delivery models that best meet state and/or local needs. As per 
MIECHV's authorizing statute, state and territory awardees must spend 
the majority of their MIECHV Program grants to implement evidence-based 
home visiting models, with up to 25 percent of funding available to 
implement promising approaches that will undergo rigorous evaluation.
    For the first time since its inception in 2009, HomVEE is proposing 
to revise the procedures and standards that guide the systematic 
review. The proposed revisions include (1) clarifying and updating 
standards and procedures for rating the quality of impact studies that 
are used to determine which home visiting models meet HHS criteria for 
an ``evidence-based early childhood home visiting service delivery 
model'' and (2) clarifying definitions, rules, and procedures for 
handling model versions (commonly referred to in the home visiting 
research literature as adaptations) in the review (presented in a 
separate Federal Register notice). The current Federal Register Notice 
focuses on the former set of revisions.
    The proposed changes to HomVEE procedures and standards generally 
bring the review into alignment with procedures and standards for other 
federally sponsored systematic evidence reviews. The proposed revisions 
also specify standards for research designs that are becoming 
increasingly common in home visiting studies. Over the course of the 
past 2 years, HomVEE consulted with methodological experts and other 
federal evidence reviews to refine and update the procedures and 
standards.
    Through this Federal Register notice, HomVEE seeks to provide a 
transparent account of how the review operates and to gather 
stakeholder input on draft changes. The sections below summarize the 
main changes to the procedures and standards. A preliminary version of 
these revisions, the HomVEE Draft Version 2 Handbook, is available 
during the public comment period on the HomVEE website at (https://homvee.acf.hhs.gov).
    After a period of public comment (including close consultation with 
selected methods experts outside of HomVEE), HomVEE will release a 
final Version 2 Handbook.

2.0 Changes and Clarifications to HomVEE Procedures

    It is natural in the course of a systematic review for issues to 
arise that cannot be addressed by existing procedures. These issues 
require the HomVEE team to develop internal guidance to guide the 
review.
    Clarifications to the procedures added to the HomVEE Draft Version 
2 Handbook, summarized in Sections 2.1 through 2.7 below, represent 
HomVEE's attempt to formalize internal guidance generated over the 
course of conducting reviews so that the procedures are applied 
systematically.
    Similarly, as reviews evolve, it is necessary for ongoing 
systematic reviews to change their procedures to meet the needs of the 
field. Changes are new procedures proposed to align with best practices 
in systematic reviews and keep the HomVEE review current.

2.1 Clarify Definitions of Research Terms

    Recognizing the importance of clear communication and consistent 
terminology when applying systematic review rules, HomVEE clarifies the 
definitions of important research terms in the HomVEE Draft Version 2 
Handbook. Exhibit I.3 of the HomVEE Draft Version 2 Handbook presents 
these, as listed below.
     A study evaluates a distinct implementation of an 
intervention (that is, with a distinct sample, enrolled into the 
research investigation at a defined time and place, by a specific 
researcher or research team). HomVEE reviews eligible manuscripts about 
studies that examine the impact of an early childhood home visiting 
model by comparing an intervention condition (in which study 
participants are offered the home visiting model under study) and a 
comparison condition (in which study participants are not offered that 
model). This includes eligible manuscripts about studies on model 
replications, iterations, and versions. See Chapter III, Section A.1.b 
of the HomVEE Draft Version 2 Handbook, including Exhibit II.4, for 
more information on how HomVEE screens research for eligibility.
    [cir] A sample encompasses both the entire intervention group and 
the entire comparison group of participants included in a study.
    [cir] A subgroup is a subset of the sample examined in a study 
(that is, an analytic subgroup). For example, researchers may examine 
how a home visiting model affects teenage mothers when there are 
mothers with a range of

[[Page 47378]]

ages in their study; hence, teenage mothers would be an analytic 
subgroup. Sometimes, researchers present subgroup findings in a 
manuscript alongside findings for the overall sample, and sometimes 
researchers prepare a manuscript based exclusively on subgroup findings 
from a broader study. (For HomVEE, results from teenage mothers would 
not be considered an analytic subgroup analysis when the overall study 
only enrolled teenage mothers.) See 2.7 below for more details on 
HomVEE's clarified subgroup definition.
     Manuscripts describe study results. Manuscripts may be 
published or unpublished research, such as journal articles, book 
chapters, or working papers. A single study may produce one, or many, 
manuscripts. Typically, one manuscript reports on only one study, 
although in rare cases one manuscript may include several studies, if 
it describes evaluations of multiple interventions or the same 
intervention evaluated in multiple distinct (non-overlapping) samples.
     Findings summarize the effect of a home visiting model on 
a specific sample or subgroup, on a specific eligible outcome measure 
(see Chapter III, Section A.4.a of the HomVEE Draft Version 2 
Handbook), at a specific time point, from a specific analysis. A 
manuscript typically includes multiple findings.
    HomVEE rates findings (according to standards proposed in Chapter 
III of the HomVEE Draft Version 2 Handbook) and sorts manuscripts 
according to the highest-rated finding in the manuscript (see Chapter 
II, Section B.2.b of the HomVEE Draft Version 2 Handbook). When 
determining which models meet HHS criteria for an ``evidence-based 
early childhood home visiting service-delivery model,'' HomVEE 
considers both whether the research that calculated the findings was 
well designed, and whether the findings come from different studies 
(with distinct samples). See Exhibit II.11 of the HomVEE Draft Version 
2 Handbook for details.

2.2 Establish a 20-Year Moving Search Window for Reviewing Most 
Manuscripts

    Searches in HomVEE's first 11 annual reviews were for manuscripts 
published in or after 1989. Generally, the HomVEE Draft Version 2 
Handbook updates the description of how HomVEE identifies research.
    Now, HomVEE proposes to implement a 20-year moving window for 
previously unreviewed manuscripts to be eligible for review. Beginning 
with the 2021 review, to keep the review current, HomVEE proposes 
establishing a 20-year moving search window for previously unreviewed 
manuscripts to be eligible for review. For example, for the 2021 
review, HomVEE would consider manuscripts released or published in 2001 
through 2020. However, HomVEE proposes that two categories of older 
research remain eligible for review (1) older research that HomVEE has 
already reviewed and (2) research submitted at any time (that is, since 
HomVEE's inception and moving forward) through the call for research. 
This change is described in Chapter II, Section A.1.a.1 of the HomVEE 
Draft Version 2 Handbook.

2.3 Adopt the PRESS Method for Systematic Searching

    Professional librarians have always conducted annual HomVEE 
literature searches using a transparent process in which the databases 
and search terms are published on the HomVEE website. Beginning with 
the 2021 review, in recognition of accepted practice in the library 
science field, HomVEE proposes to use a modified Peer Review of 
Electronic Search Strategies (PRESS) method to refine the search terms 
(McGowan et al. 2016). This approach includes adjusting search terms 
and search databases in keeping with the recommendations of 
professional librarians. This change is described in more depth in 
Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.

2.4 Add New ``Grey Literature'' Databases

    To better capture research that is not published in academic 
journals, HomVEE proposes to expand its annual search, beginning in 
2021, to include two new databases to identify this ``grey 
literature'': Google Scholar and the Harvard Think Tank Search. See 
Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.

2.5 Establish Rule for Accepting Supplemental Information

    HomVEE will continue, as it always has, to accept submissions from 
the public, both during its call for research and at other times, and 
to consider those annually as part of prioritizing research for review. 
The HomVEE Draft Version 2 Handbook proposes a clarification stating 
that, if authors submit unpublished work to the HomVEE call for 
research, HomVEE will consider only unpublished full manuscripts with 
sufficient text describing the study's procedure, analysis approach, 
and findings.
    As part of the HomVEE Draft Version 2 Handbook (Chapter II, Section 
B.1.b), HomVEE proposes establishing a new rule about accepting 
supplemental information from stakeholders. Under the new rule, HomVEE 
would accept supplemental information only under specific 
circumstances. HomVEE must maintain a strict review schedule for the 
annual review to ensure results are released on time.
    Supplemental information can take two forms (1) new information 
about a study's methods or procedures, or (2) new research that 
supplements what HomVEE had on hand at the close of that year's call 
for research, such as additional findings or new analyses of research 
in a previously reviewed manuscript, or an entirely new set of 
findings.

2.5.1 HomVEE Rule About New Information

    HomVEE proposes to incorporate new information about methods and 
procedures into the initial review of a manuscript only if (1) it is 
provided in direct response to an author query and (2) authors submit 
it in time for reviewers to examine it during the same annual review 
cycle in which HomVEE issued the query. Otherwise, HomVEE intends to 
require authors wait until HomVEE releases its annual review results 
for the model described in the manuscript in question. Then, authors 
could follow the process for requesting a reconsideration of evidence 
to ask HomVEE to examine supplemental information that authors provide, 
through the appeals process, about methods or procedure.

2.5.2 HomVEE Rule About New Research

    HomVEE proposes to treat all new research as a submission to the 
following year's call for research, unless it consists of new analyses 
conducted at the explicit request of the HomVEE review team (see 
Sections 3.3.1 and 3.3.2, below on repeated measures studies and 
structural equation models, respectively).

2.6 Define Contrasts in Impact Research That Are Ineligible for Review 
by HomVEE

    To date, HomVEE has placed no restrictions on services offered to 
the comparison condition in the impact studies it reviews. Beginning 
with the 2021 review, HomVEE proposes that the review generally exclude 
research that isolates the impact of model features. Research on 
specific features does not answer HomVEE's core question of whether an 
early childhood home

[[Page 47379]]

visiting model is effective. Specifically, HomVEE proposes not to 
review studies about the impact of model features. However, studies 
isolating the impact of a curriculum module may be treated as evidence 
for an independent model if all of the following criteria are met:
     The curriculum module satisfies the definition of an early 
childhood home visiting model;
     The treatment group does not receive any other curriculum 
modules from the base model; and
     The curriculum module has a manual and implementation 
infrastructure independent from that of the base model.
    This change is described in Chapter III, Section A.2 of the HomVEE 
Draft Version 2 Handbook.

2.7 Defining Subgroups and Protocol for Reporting Subgroup Analyses

    In the HomVEE Draft Version 2 Handbook, HomVEE defines a subgroup 
as a subset of the sample that researchers choose to examine in a 
study; that is, an analytic subgroup.
    Subgroup research is important for HomVEE because a model can meet 
the HHS criteria for ``evidence-based early childhood home visiting 
service delivery models'' based on findings from subgroups. The HHS 
criteria for an ``evidence-based early childhood home visiting service 
delivery model'' include special rules about subgroup findings: If 
favorable results that could form the evidence base for a model ``[are] 
found for subgroups but not for the full sample for the study, [the 
findings must] be replicated in the same domain in two or more studies 
using non-overlapping analytic study samples.'' (See Exhibit II.11 in 
the HomVEE Draft Version 2 Handbook.) Therefore, HomVEE exercises care 
in identifying subgroup research and understanding how the subgroup 
relates to the overall study sample.
    HomVEE defines a subgroup as a subset of the overall sample 
examined in a study--that is, an analytic subgroup (see Section 2.1, 
above). Notably, this is different from defining subgroup as a subset 
of the overall population. Although researchers may examine an analytic 
subgroup in hopes of making inferences about a subset of the 
population, the goal of the HHS criteria is to ensure that program 
impacts are replicated consistently for an outcome domain. Such 
replication is what gives HomVEE confidence that evidence of 
effectiveness is not due simply to chance. Thus, if a model meets HHS 
criteria for evidence of effectiveness based on subgroup findings, this 
means that research in which that subgroup was similarly defined in 
relation to the broader sample had consistent, favorable (statistically 
significant) findings in distinct study samples.
    Subgroup results may be nested within a manuscript (for example, 
results from teenage mothers when the overall results in the manuscript 
are from mothers with a range of ages), or they may be the main focus 
of a manuscript (for example, a manuscript focusing on results from 
teenage mothers when the overall study sample included mothers with a 
range of ages). HomVEE treats both of those as analytic subgroup 
analyses. HomVEE's definition means that not all analyses restricted to 
a certain characteristic are subgroup analyses. For example, results 
from teenage mothers are not an analytic subgroup analysis when the 
overall study only enrolled teenage mothers, even though teenage 
mothers are a subgroup of the population of mothers as a whole.
    Because HomVEE's mission is to identify which models are effective 
according to the HHS criteria, and to use project resources 
judiciously, HomVEE proposes to only review research on replicable 
subgroups (if it meets other eligibility criteria defined in Chapter 
II, Section A of the HomVEE Draft Version 2 Handbook), and to only 
report review results for replicated subgroups. HomVEE proposes the 
following definitions for those terms:
     Replicable subgroups are defined by a characteristic that 
a different study could replicate with a non-overlapping sample. Most 
subgroups are replicable, in theory. However, HomVEE does not consider 
subgroups defined by cohort or time (for example, a subgroup of mothers 
enrolled in 1995 in a study that included mothers enrolled across 
several years) to be replicable in subsequent studies, and therefore 
does not review time-based subgroups. Similarly, HomVEE will only 
consider a subgroup defined by location to be replicable if the 
location was selected based on defined characteristics (for example, 
county with the highest teen birth rate in the state in a study 
conducted in several counties). Location-based subgroups defined by a 
location name (for example, Adams County in a study conducted in 
several counties) will not be reviewed because the HomVEE team cannot 
confidently verify whether the subgroup sample in a subsequent study in 
that county overlaps with the first study when the team applies HHS 
criteria.
    HomVEE will report subgroup results only from a replicated 
subgroup, one that has an identical definition in two non-overlapping 
research samples. For example, a study examining a subgroup of 
primiparous teenagers is not replicated by a study examining 
primiparous women of all ages. This approach is consistent with the HHS 
criteria's emphasis on observing effects across independent samples.

2.8 Clarify HomVEE's Approach to Operationalizing the HHS Criteria for 
Randomized Controlled Trials

    As specified in the statute that authorized the MIECHV Program and 
required HHS to establish criteria for evidence of effectiveness of 
home visiting models, the HHS criteria for an ``evidence-based early 
childhood home visiting service delivery model'' state that additional 
criteria apply when the research on home visiting models comes from 
randomized controlled trials (see Exhibit II.11 in the HomVEE Draft 
Version 2 Handbook). Specifically, one or more favorable impacts must 
be sustained for at least 1 year after program enrollment, and one or 
more favorable impacts must be reported in a peer-reviewed journal. The 
HomVEE Draft Version 2 Handbook clarifies the way that HomVEE has 
operationalized the additional criteria for randomized controlled 
trials (RCTs). Specifically, these two requirements can be satisfied by 
findings from different studies, provided the quality of these findings 
is rated as moderate or high.

3.0 Clarifications and Changes to HomVEE Standards

    HomVEE proposes several updates to its standards for reviewing 
manuscripts about impact studies, including both clarifications and 
changes.
    It is natural in the course of a systematic review for issues to 
arise that cannot be addressed by existing standards and rules. These 
issues require the HomVEE team to develop internal guidance to guide 
the review.
    Clarifications to the standards added to the HomVEE Draft Version 2 
Handbook, summarized in Section 3.1 below, represent HomVEE's attempt 
to formalize internal guidance generated over the course of conducting 
reviews so that the procedures are applied systematically.
    Similarly, as research methods evolve, it is necessary for ongoing 
systematic reviews to change their standards to meet the needs of the 
field. Changes are HomVEE's attempt both to align with aspects of other 
ongoing, federally sponsored systematic evidence reviews (Section 3.2) 
and to specify standards for research designs that are becoming 
increasingly common in home visiting studies (Section 3.3).

[[Page 47380]]

3.1 Clarifications

3.1.1 Changes to Terminology Used

    HomVEE reviews manuscripts about research that uses any of three 
types of quasi-experimental designs (QEDs)--regression discontinuity 
designs (RDDs), single-case designs (SCDs), and non-experimental group 
designs (NEDs). Previously, HomVEE used QED to refer only to NEDs. 
Other designs, including SCD and RDD, are also quasi-experimental, so 
HomVEE proposes labeling this category of research more precisely, as 
NED (see Chapter I, Section C of the HomVEE Draft Version 2 Handbook). 
HomVEE intends to use this new terminology to more accurately reflect 
the fact that HomVEE does not have (and does not propose to implement) 
requirements about statistical matching in NED designs.

3.1.2 Ineligible and Preferred Analyses

    To date, as long as the underlying study design is an RCT or QED, 
HomVEE has not specified rules for identifying analyses as ineligible 
for review.
    Starting with the 2021 review, HomVEE proposes to exclude certain 
analyses within manuscripts about RCTs and QEDs as ineligible, as 
described in Chapter III, Section A.3 of the HomVEE Draft Version 2 
Handbook. HomVEE's mission is to determine whether research shows that 
a home visiting model improves outcomes for children and families. 
Questions about the mechanisms behind how a model works, the settings 
where it might work best, and the populations who benefit the most from 
the intervention are outside of the scope of the HomVEE review. 
Although answers to these questions are important for understanding and 
improving home visiting models, the primary aim of the HomVEE review is 
to identify currently available models that are effective. For this 
reason, certain types of analyses designed to answer questions other 
than whether a model is effective are not eligible for review.
    In addition, analyses of how the home visiting model affected only 
sample members who received it are sometimes ineligible for review if 
other analyses in the manuscript better address HomVEE's mission.

3.1.2.1 Mediating and Moderating Analyses

    First, HomVEE proposes that most mediating and moderating analyses 
(except some structural equation models, see Section 3.3.2 below), 
would be ineligible for review. HomVEE focuses on research that answers 
the following question: Is the home visiting model effective? Mediating 
and moderating analyses answer important, but slightly different 
questions of how, and for whom, the model works.

3.1.2.2 Endogenous Analyses

    Second, HomVEE proposes that the review would exclude analyses that 
control for endogenous characteristics. These characteristics (1) are 
defined by behavior emerging after study participants know whether they 
will be in the intervention group or the comparison group and (2) could 
theoretically be affected by a home visiting model. Analyses that 
control for endogenous characteristics produce biased estimates of the 
effectiveness of an intervention. Analyses of subgroups defined by 
endogenous characteristics would also be ineligible for review.

3.1.2.3 Analyses of the Impact of the Treatment on the Treated

    The HomVEE Draft Version 2 Handbook also specifies a proposed 
clarification to how HomVEE would review studies that examine the 
effect of the treatment (the home visiting model) on the treated (study 
sample members who receive the treatment). Specifically, when a study's 
researchers examine the effect of both the intent to treat (ITT) and 
the treatment on the treated (TOT), HomVEE proposes to focus its review 
on the ITT, because those estimates more realistically depict the 
average magnitude of the effect that a program replicating the model 
would cause. If those researchers report only TOT estimates, HomVEE 
reviews those using What Works Clearinghouse (WWC) Version 4.1 guidance 
on reviewing for Complier Average Causal Effects.

3.1.3 Eligible Outcomes and Baseline Assessability

    Since its inception, HomVEE has reviewed findings of home visiting 
impact studies that fall into eight domains related to child, maternal, 
and family well-being. In the HomVEE Draft Version 2 Handbook, HomVEE 
proposes clarifying which specific findings within eligible analyses 
are eligible for review (see Chapter III, Section A.4). These 
clarifications formalize and expand HomVEE's existing internal guidance 
on eligible outcomes and baseline assessability.
    First, the HomVEE Draft Version 2 Handbook clarifies that only 
unique findings would be eligible for review (those that report results 
on a different outcome, sample or subgroup, or time period, or with a 
different analytic approach, than findings reported in other 
manuscripts about the same home visiting model). In these cases, 
consistent with current HomVEE practices, the review simply would 
reference the other manuscript--the first or most complete one in which 
HomVEE encountered the finding--where HomVEE users could find those 
results and the review conclusions. The Draft Version 2 Handbook also 
clarifies that HomVEE would not consider simple transformations of 
analyses with the same sample, outcome, and time period to be unique 
findings within a manuscript if they (1) transform findings data from 
frequency to a ratio (such as percentage or per thousand) or (2) 
transform findings data across different ratio types (such as from 
percentage to per thousand) because these simple transformations do not 
constitute a different analytic approach. In manuscripts with such 
transformations, HomVEE proposes to review the finding that is 
calculated as a percentage, because it is an intuitive measure to many 
readers and can be easily compared across studies.
    Second, the HomVEE Draft Version 2 Handbook specifies 
categorization practices and baseline equivalence requirements for 
outcomes that HomVEE reviews. See Exhibit III.2 and Chapter III, 
Section B.3 of the HomVEE Draft Version 2 Handbook for a summary and 
Appendix C of that document for a detailed listing. That appendix 
indicates which outcomes or outcome categories belong under each of the 
eight domains. Also, although HomVEE requires NEDs and certain RCTs 
(those with high attrition or compromised randomization) to establish 
that the intervention and comparison groups are equivalent at baseline, 
the review team recognizes that some measures cannot or should not be 
measured at baseline. Therefore, Appendix C of the HomVEE Draft Version 
2 Handbook clarifies which outcomes HomVEE would expect authors to 
assess at baseline.

3.2 Changes To Align HomVEE With Standards of Other Federally Sponsored 
Systematic Reviews

    HomVEE's initial standards aligned to WWC standards, Version 2.1, 
which were the latest standards implemented when the HomVEE review 
began. These standards define the criteria that research must meet to 
be assigned each of three ratings. HomVEE calls these ratings high, 
moderate, and low, although the WWC rates research as Meets Standards 
(HomVEE high), Meets Standards with Reservations (HomVEE moderate), and 
Does Not Meet Standards (HomVEE low). The WWC remains a prominent and 
influential

[[Page 47381]]

federally sponsored systematic evidence review. In early 2020, WWC 
released Version 4.1 standards. Furthermore, in the time since HomVEE 
began, ACF has begun overseeing another, related systematic review: The 
Title IV-E Prevention Services Clearinghouse. That review focuses on 
child welfare research, some of which overlaps with home visiting 
research, and its standards are similar to those of WWC Version 4.1. 
ACF is interested in aligning standards for HomVEE and the Prevention 
Services Clearinghouse where appropriate.
    In its HomVEE Draft Version 2 Handbook, HomVEE proposes to adopt 
many aspects of the latest WWC standards and some aspects of the 
Prevention Services Clearinghouse standards so that the review stays 
synchronized with accepted best practices in federally sponsored 
systematic reviews. The sections below describe proposed changes to the 
HomVEE review that would affect study ratings as the HomVEE criteria 
stand now. (HomVEE proposes to fully adopt WWC Version 4.1 criteria for 
regression discontinuity design studies, which are not described below 
because HomVEE has not, to date, reviewed any studies with this 
design.)

3.2.1 Requirement for Validity and Reliability of Outcome Measures

    To date, HomVEE has had no stated validity and reliability 
requirements that outcomes must meet, although the review reports 
whether outcomes are primary (which HomVEE defines as an outcome 
measured through direct observation, direct assessment, or 
administrative data; or self-reported data collected using a 
standardized [normed] instrument) or secondary (for HomVEE, most self-
reported data, excluding self-reports based on a standardized 
instrument). With the Draft Version 2 Handbook (see Chapter III, 
Section B.4), HomVEE proposes to introduce face validity and 
reliability standards. HomVEE reviewers will apply these new standards 
to all findings that are within one of HomVEE's eight outcome domains 
and to all measures HomVEE uses to assess baseline equivalence. 
Findings about outcomes that do not meet both the face validity and the 
reliability standard would rate low. With this change, HomVEE proposes 
to stop sorting outcomes as primary or secondary.
    To meet the face validity standard, an outcome measure must be (1) 
clearly defined and (2) measure the construct it was designed to 
measure. This information could come from the manuscript reviewers 
examine, or from supplemental information that HomVEE requests from the 
author. HomVEE reviewers propose to consult with project leaders 
whenever it is not clear whether a measure meets the validity 
requirement, and project leaders would in turn consult with subject 
matter experts and with ACF about the validity of new and of modified 
standardized measures.
    Some outcome measures are not appropriate to validate with 
psychometric tests. HomVEE proposes to assume that the following 
measures are reliable: (1) Administrative records obtained from child 
welfare or other social service agencies, hospitals or clinics, and 
schools; (2) demographic characteristics; and (3) medical or physical 
tests.
    Otherwise, to demonstrate reliability, outcome measures must meet 
at least one of the following standards:
     Internal consistency (such as Cronbach's alpha) of 0.50 or 
higher.
     Test-retest reliability of 0.40 or higher.
     Inter-rater reliability (as indicated by percentage 
agreement, correlation, or kappa) of 0.50 or higher.
    Under the proposed approach, HomVEE reviewers would prioritize 
reliability statistics on the sample of participants in the manuscript 
under review, but would also consider statistics from test manuals or 
studies of the psychometric properties of the measures. The review team 
may ask authors to provide additional information about the reliability 
of their measures.

3.2.2 In Some Cases, Some Sample Loss Does Not Count as Attrition

    Attrition happens when outcome data are missing for some members of 
the intervention and comparison groups in a study. Previously, HomVEE 
counted all sample loss as attrition unless the authors had imputed 
findings (see Section 3.2.3, below). In alignment with Version 4.1 of 
the WWC Standards, the HomVEE Draft Version 2 Handbook (see Chapter 
III, Section B.1) proposes that some types of sample loss will not 
count as attrition in HomVEE. First, losing sample members after random 
assignment because of acts of nature, such as hurricanes, fires, or the 
COVID-19 pandemic, is not considered attrition if the loss affects the 
intervention and comparison conditions in the same way. However, if the 
sample loss due to an act of nature was concentrated in one of the 
conditions, then the sample loss would be considered attrition. Second, 
when researchers exclude a subsample of the randomly assigned sample 
from their analysis, HomVEE would not consider that excluded subsample 
to constitute attrition if (1) the subsample was randomly selected or 
(2) the subsampling was based on characteristics that were clearly 
determined before the start of the intervention and applied 
consistently across the intervention and comparison conditions.

3.2.3 Standards for Addressing Missing Data

    The original HomVEE standards did not specify how reviewers would 
respond when study authors used various analytic strategies to account 
for missing data. The HomVEE Draft Version 2 Handbook (see Chapter III, 
C.2, as well as Appendix F) proposes to align HomVEE's practices to the 
way WWC Version 4.1 standards handle studies with missing data. 
Specifically, HomVEE proposes to first calculate attrition based on the 
analytic sample in the manuscript, treating any imputed values as lost 
sample. If baseline data in the analytic sample are missing or imputed, 
baseline equivalence would have to be established using the largest 
baseline difference accounting for missing or imputed baseline data. 
Second, manuscripts about studies with missing data would only be 
eligible for review by HomVEE if the authors had used the following 
specific approaches to address the missing data:

 Complete case analysis
 Maximum likelihood (including expectation maximization and 
full information maximum likelihood)
 Multiple imputation (must be conducted separately by treatment 
status)
 Nonresponse weights (must be conducted separately by treatment 
status; acceptable only for missing outcome data, not for missing 
baseline data)

    In alignment with WWC version 4.1 standards, if the baseline data 
include imputed data, HomVEE would also apply other criteria when 
assessing baseline equivalence (see 3.2.5).

3.2.4 No Baseline Equivalence Requirement for Low-Attrition RCTs

    Original HomVEE standards require authors of RCTs and non-
experimental designs to establish baseline equivalence on race and 
ethnicity, socioeconomic status, and measures of outcomes that are 
feasible to assess when the study begins.
    In low-attrition RCTs, the original standards allow authors to 
instead implement statistical controls for these characteristics. In 
alignment with WWC Version 4.1 Standards, the HomVEE

[[Page 47382]]

Draft Version 2 Handbook proposes that HomVEE would no longer require 
that RCTs with low attrition establish equivalence or adjust for 
baseline differences. This is because proper randomization is expected 
to produce groups that are similar, and baseline differences that might 
be observed on one or more measures are not generally evidence of 
differences that will introduce bias into research findings.

3.2.5 Baseline Equivalence Depends on Difference in Effect Sizes, and 
Other Considerations

    The original HomVEE standards based the assessment of equivalence 
on measuring statistically significant differences between intervention 
and comparison groups at baseline. In line with WWC Version 4.1 
standards, HomVEE's Draft Version 2 Handbook proposes that HomVEE will 
assess baseline equivalence based on the magnitude of the difference in 
standard deviation units (effect size).
    To limit bias that can arise from differences in the treatment and 
comparison group units used to measure the effect of a home visiting 
model on outcomes, the groups must appear similar on the relevant 
baseline characteristics that are thought to be related to the 
outcomes. This balance is best shown using the observed magnitude of 
differences in the sample.
    Specifically, the new HomVEE criterion for baseline equivalence 
proposes to rely on effect size, computed as the absolute value of the 
difference between treatment and comparison groups in standard 
deviation units. HomVEE would require the following to be true for 
research to demonstrate baseline equivalence for a specified 
characteristic:
     A baseline effect size less than or equal to 0.05 meets 
the baseline equivalence requirement and requires no statistical 
adjustment.
     For a baseline effect size that is greater than 0.05 and 
less than or equal to 0.25, an acceptable statistical adjustment for 
the baseline characteristic is required to meet the baseline 
equivalence requirement.
     If the baseline effect size is greater than 0.25, HomVEE 
considers the intervention and comparison groups to be nonequivalent, 
that is, the intervention and comparison groups do not meet the 
baseline equivalence requirement for the specified characteristic.
    Under the proposed new standards, HomVEE would also consider the 
following when assessing baseline equivalence:
     HomVEE would allow baseline data that include imputed data 
to be used to demonstrate baseline equivalence of the analytic sample 
in some cases. If the baseline data include imputed data, HomVEE would 
first estimate how large the baseline difference (in standard deviation 
units) between intervention and comparison groups might be under 
different assumptions about how the missing data are related to 
measured and unmeasured factors. Then HomVEE would use the largest of 
those estimates in absolute value as the effect size for assessing 
baseline equivalence.
     The measures used to establish baseline equivalence must 
be at the same level as the unit of analysis. For example, in an 
analysis at the individual or family level, measures of socioeconomic 
status at the ZIP code level may not be used to establish baseline 
equivalence between the individuals or families in the intervention and 
comparison groups.
     If the impact analyses use weights, then the baseline 
means must be calculated using the same weights.
     If the study conducted random assignment within blocks or 
strata, and the analyses include dummy variables that differentiate 
these blocks or strata, then these same dummy variables can be used to 
adjust the baseline means.
    This criterion is described in the HomVEE Draft Version 2 Handbook, 
Chapter III, Section B.2.

3.2.6 Allowable Statistical Adjustment Techniques

    HomVEE has always required that authors implement statistical 
adjustments for baseline differences if their studies use (1) an RCT 
design or (2) one type of quasi-experimental design, NED. To date, 
HomVEE has not specified allowable techniques for that adjustment. With 
the HomVEE Draft Version 2 Handbook (see Chapter III, Section B.2.b), 
HomVEE proposes to follow WWC Version 4.1 guidelines about which 
statistical adjustment procedures are acceptable. Those are:
     Acceptable analytic methods to adjust for baseline 
differences:
[cir] Regression adjustments
[cir] Analysis of covariance (ANCOVA) or multivariate analysis of 
covariance (MANCOVA)
[cir] Repeated measures analysis of variance (ANOVA) or multivariate 
analysis of variance (MANOVA)
[cir] Estimating impacts only for groups defined at baseline (for 
example, ever had a baby versus never had a baby)
[cir] Growth curve modeling (this approach to modeling repeated 
measures research is also subject to other requirements; see 3.3.1 
below)
 Acceptable methods if baseline and follow-up measures of 
outcome are the same and have a strong relationship to each other
[cir] Gain or change scores (pre-post differences)
[cir] Difference-in-difference adjustments
[cir] Fixed effects for individuals

3.2.7 Cluster RCTs

    HomVEE rarely encounters RCTs with a cluster design in which a 
group of sample members, such as a neighborhood, is assigned to be 
offered a home visiting model or some other condition. However, in such 
designs, limiting sample loss at both the cluster (for example, a 
neighborhood) and subcluster (for example, a family that received home 
visiting) levels is important to maintaining the integrity of the 
randomization design. Under HomVEE's original standards, reviewers 
would apply a cluster correction to findings if authors themselves had 
not done so, but no special requirements were in place for rating 
studies that used a cluster design.
    Instead, in research reviewed under the HomVEE Draft Version 2 
Handbook, HomVEE proposes to align to WWC Version 4.1 guidelines for 
calculating attrition and non-response of subcluster members (such as 
individuals or families) in cluster RCTs. Exhibit III.14 of the HomVEE 
Draft Version 2 Handbook specifies how studies would be rated based on 
their combination of attrition at the cluster and individual levels, 
and authors' decisions about implementing statistical controls. In 
brief, a cluster RCT would be eligible to rate high only if it has low 
sample loss at both the cluster level and the individual level. To rate 
moderate, research about cluster RCTs with high attrition and research 
about cluster NEDs would need to demonstrate baseline equivalence of 
the analytic sample. Additional detail about this new standard appears 
in Chapter III, Section C.1 of the HomVEE Draft Version 2 Handbook.

3.2.8 Adopt New WWC Version 4.1 Standards for Regression Discontinuity 
Designs

    Regression discontinuity research has been eligible for review by 
HomVEE since the project's inception, using earlier WWC pilot criteria 
for this research. This design is rare in home visiting research, and, 
to date, HomVEE has not reviewed any research that uses this design.
    HomVEE proposes to align its Version 2 standards to WWC's latest 
(Version 4.1) RDD standards. The updates to the RDD standards consist 
of:

[[Page 47383]]

     A new set of procedures for reviewing ``fuzzy'' RDDs (for 
example, those in which some intervention group members do not receive 
intervention services and the analysis adjusts for this 
nonparticipation),
     Expanded procedures for reviewing multi-site and multiple 
assignment variable regression discontinuity designs, and
     A preference for local bandwidth impact estimation over 
global impact regression with flexible functional forms.
    Appendix D of the HomVEE Draft Version 2 Handbook thoroughly 
describes the new WWC Version 4.1 RDD standards that HomVEE propose to 
implement, and their corresponding reporting procedures.

3.3 Other Changes

    The HomVEE Draft Version 2 Handbook proposes two other changes to 
standards for reviewing impact studies. The two changes would define a 
new approach to reviewing designs that are becoming increasingly common 
in home visiting studies--repeated measures analyses and structural 
equation models. The third change pertains to SCD research.

3.3.1 Repeated Measures Analyses

    In repeated measures analyses, authors measure the research sample 
at several time points to chart its growth over the course of the 
intervention and sometimes beyond. To date, HomVEE has not specified 
any standards for reviewing repeated measures analyses in group-design 
studies (such as RCTs and NEDs), nor have other federally sponsored 
systematic evidence reviews thoroughly addressed this.
    In the proposed new standard, HomVEE would only review and report 
findings from repeated measures analyses with multiple follow-ups in 
RCTs and NEDs when the findings are available for individual time 
points, relative to baseline. When rating each time point, HomVEE would 
apply its Version 2 RCT or NED standards. Generally, when gathering 
information to rate each time point, HomVEE would defer to what the 
author reported or what the review team could calculate based on 
details the author provided. As a last resort, when adjusted analyses 
are necessary in order to rate the study, and the HomVEE team cannot 
make the necessary calculations, the HomVEE team would ask authors to 
reanalyze their data to calculate adjusted time point findings. HomVEE 
would exclude from its review of a repeated measures analysis any time 
points for which an impact cannot be included in HomVEE reports because 
neither author-provided nor HomVEE-calculated estimates are available. 
Chapter III, Section C.3 of the HomVEE Draft Version 2 Handbook 
describes the proposed new approach to rating repeated measures 
analyses in detail.

3.3.2 Structural Equation Models

    Structural equation models (SEMs) examine the relationship between 
a dependent variable and multiple independent variables, often 
incorporating multiple outcomes from different follow-up periods. To 
date, HomVEE standards did not define how the review would incorporate 
SEM research. Chapter III, Section C.4 of the HomVEE Draft Version 2 
Handbook specifies how HomVEE proposes to approach the review of 
research with this design going forward. In brief, only SEMs that are 
accompanied by a path diagram (including one authors may submit in 
response to a query from HomVEE) and that are identified (that is, the 
degrees of freedom are greater than the parameters to be estimated) 
would be eligible for review. Within SEMs that are eligible for review, 
HomVEE would review only findings for which the answer to the following 
two questions is yes: (1) Is there a direct pathway from the 
intervention to the outcome? and (2) Are there no pathways leading to 
that outcome from another outcome? This approach is consistent with 
HomVEE's proposed new approach to mediated and moderated analyses. See 
also Section 3.1.2.1 above.

3.3.3 Review of Single-Case Design (SCD) Research

    SCDs are quasi-experimental research designs in which an individual 
case serves as its own control, and the outcome is measured repeatedly 
within and across different conditions (as defined in What Works 
Clearinghouse [WWC] Version 4.1 standards). SCD research has been 
eligible for HomVEE review since its inception, using earlier WWC pilot 
criteria for this research. With Version 4.1, WWC has removed the 
``pilot'' designation from its standards and has updated its procedures 
for reviewing SCD research in several ways; HomVEE proposes aligning to 
WWC's version 4.1. Although WWC previously standards instructed 
reviewers to only use visual analysis of changes in the outcome over 
time and across conditions to characterize the findings from an SCD 
study, the new standards from WWC also have reviewers calculate and use 
a design-comparable effect size to characterize the findings. Reviewers 
would still use visual analysis to assess whether a SCD study is well 
designed. To calculate a design-comparable effect size, the HomVEE 
contractor review team would use data presented in the study if 
possible, or (only if necessary) contact the study authors to request 
raw study data so the team could calculate that value. Appendix E of 
the HomVEE Draft Version 2 Handbook thoroughly describes the WWC 
Version 4.1 SCD standards that HomVEE propose to implement, and their 
newly updated reporting procedures.

4.0 Timeline for HomVEE To Apply New Procedures and Standards

    HomVEE proposes to apply the new procedures and standards beginning 
with the 2021 review. HomVEE will not retroactively apply the new 
standards to previously reviewed research about evidence-based models 
unless it is SCD research about a model HomVEE prioritizes and selects 
for review.
    To promote consistency in reporting across the review, 
clarifications about the outcomes that are eligible for review in each 
domain retroactively will apply to all models regardless of (1) their 
evidence-based status according to HHS criteria and (2) whether they 
are prioritized and selected for review. However, manuscripts that have 
findings excluded or moved to other domains will not be re-reviewed 
with HomVEE Version 2 standards (unless they are manuscripts about a 
SCD study). In addition, the HomVEE team will retroactively apply 
clarified definitions of study, manuscript, and subgroup, with the aim 
of relabeling HomVEE products so they use consistent language.
    Also, HomVEE typically reviews eligible models every other year at 
the earliest. In 2021 (the first year that new procedures and standards 
are in effect), HomVEE will suspend this rule for one year only, so 
that models reviewed in 2020 are not excluded from consideration for 
the 2021 review.

4.1 HomVEE Will Not Retroactively Apply New Procedures and Standards to 
Inactive Models

    For models that no longer provide implementation support, the 
HomVEE team generally does not plan to retroactively apply the new 
procedures and standards, except to apply the clarifications about the 
outcomes that are eligible for review in each domain and about the 
definitions of study, manuscript, and subgroup. The team proposes to 
update reports about those models on the HomVEE website (https://homvee.acf.hhs.gov) to indicate that they were reviewed under the

[[Page 47384]]

initial HomVEE procedures and standards.

5.0 Request for Information (RFI)

    Through this Federal Register Notice, ACF is soliciting information 
from a broad array of stakeholders on the proposed revisions to 
HomVEE's procedures. Federal, state, and local decision makers rely on 
HomVEE to know which home visiting models are effective. New 
definitions, rules, and procedures about model versions may affect 
which models are deemed effective by HomVEE. New procedures may affect 
which models are eligible for review and deemed effective by HomVEE. 
New standards may affect which studies constitute well-designed 
research that serves as an evidence base for models that meet HHS 
criteria for an ``evidence-based early childhood home visiting service 
delivery model.''
    Responses to this Federal Register notice will inform ACF's ongoing 
discussion about HomVEE's procedures and standards, with the aim of 
publishing a final HomVEE Version 2 Handbook by the end of 2020. This 
RFI is for information and planning purposes only and should not be 
construed as a solicitation or as an obligation on the part of ACF or 
HHS.

(Authority: Social Security Act Title V Sec.  511 [42 U.S.C. 711], 
as extended by the Bipartisan Budget Act of 2018 (Pub. L. 115-123) 
through fiscal year 2022)

John M. Sweet Jr,
ACF/OPRE Certifying Officer.

References

    U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance, 
What Works Clearinghouse. What Works Clearinghouse Version 2.1 
Evidence Standards. 2011. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v2_1_standards_handbook.pdf. 
Accessed June 19, 2020.
    U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance, 
What Works Clearinghouse. What Works Clearinghouse Procedures 
Handbook: Version 4.1., 2020a. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf. 
Accessed June 19, 2020.
    U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance, 
What Works Clearinghouse. What Works Clearinghouse Standards 
Handbook: Version 4.1., 2020b. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf. 
Accessed June 19, 2020.
[FR Doc. 2020-17001 Filed 8-4-20; 8:45 am]
BILLING CODE 4184-74-P