Revised Single-Case Design Procedures and Standards: Home Visiting Evidence of Effectiveness (HomVEE) Review, 15221-15225 [2021-05840]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Administration for Children and Families

[Federal Register Volume 86, Number 53 (Monday, March 22, 2021)]
[Notices]
[Pages 15221-15225]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2021-05840]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Revised Single-Case Design Procedures and Standards: Home 
Visiting Evidence of Effectiveness (HomVEE) Review

AGENCY: Administration for Children and Families, U.S. Department of 
Health and Human Services.

ACTION: Request for public comment.

-----------------------------------------------------------------------

SUMMARY: The Administration for Children and Families (ACF), within the 
U.S. Department of Health and Human Services (HHS), oversees the Home 
Visiting Evidence of Effectiveness (HomVEE) review, which is proposing 
to revise the procedures and standards that guide its work with single-
case design (SCD) research. The current Federal Register notice (FRN) 
seeks comments on proposed changes related to revised procedures and 
standards for SCD research. Readers are referred to the full version of 
the HomVEE Version 2 Handbook on the HomVEE website for more details, 
particularly Appendix D (https://homvee.acf.hhs.gov/publications/methods-standards). HomVEE will release an updated Handbook (Version 
2.1) after consideration of public comments received in response to 
this FRN.

DATES: Send comments on or before April 19, 2021.

ADDRESSES: Submit questions, comments, and supplementary documents to 
[email protected] with ``HomVEE SCD procedures and standards FRN 
comment'' in the subject line.

SUPPLEMENTARY INFORMATION: Invitation to Comment: HHS invites comments 
regarding this notice. To ensure that your comments are clearly stated, 
please identify the section of this notice that your comments address.

1.0 Background

    To help policymakers, program administrators, model developers, 
researchers, and the public identify rigorous research and understand 
which early childhood home visiting models are effective, ACF's Office 
of Planning, Research, and Evaluation (OPRE) within HHS oversees the 
HomVEE review, in partnership with the Health Resources and Services 
Administration (HRSA). HomVEE's mission is to conduct a thorough and 
transparent review of early childhood home visiting models that serve 
pregnant women and children from birth to kindergarten entry. The 
review identifies well-designed, well-executed research, then extracts 
and summarizes the findings from that research.
    One critical use of HomVEE's results is to determine which home 
visiting models meet the HHS criteria for an ``evidence-based early 
childhood home visiting service delivery model'' (see Section 1.1 
below), a key requirement of eligibility for implementation of the 
model with Maternal, Infant, and Early Childhood Home Visiting (MIECHV) 
Program funding.
    The MIECHV Program is administered by HRSA in partnership with ACF. 
Created in 2010, the MIECHV Program provides funding to states, 
territories, and tribal entities to implement home visiting models. 
MIECHV awardees have the flexibility to tailor the program to serve the 
specific needs of their communities. Through a needs assessment, 
awardees identify at-risk communities and select home visiting service 
delivery models that best meet state and/or local needs. In accordance 
with MIECHV's authorizing statute, state and territory awardees must 
spend the majority of their MIECHV Program

[[Page 15222]]

grants to implement evidence-based home visiting models with up to 25 
percent of funding available to implement promising approaches that 
will undergo rigorous evaluation.
    In December 2020, HomVEE released an updated Version 2 Handbook of 
Procedures and Evidence Standards (Sama-Miller et al. 2020). The 
updates incorporated input from methodological experts, other federal 
evidence reviews, and public comments on two FRNs released on August 5, 
2020 (85 FR 47376 and 85 FR 47384). The HomVEE Version 2 Handbook 
adopted the What Works Clearinghouse (WWC) Version 4.1 Procedures and 
Standards Handbooks (2020a and 2020b), including revised SCD procedures 
and standards.
    This FRN proposes further changes to the HomVEE Version 2 Handbook 
to clarify how HomVEE would apply the WWC 4.1 procedures and standards 
that HomVEE already adopted. The proposed changes focus exclusively on 
SCD research, including:
    (1) Requirements for considering SCD research toward the HHS 
criteria for evidence-based models (discussed in Section 2 of the FRN).
    (2) Flexibilities for applying certain SCD standards (discussed in 
Section 3 of the FRN).
    A work group consisting of federal staff and HomVEE contractor 
staff, including methodological and home visiting experts, met to 
discuss and develop the proposed changes outlined in this FRN. Through 
this FRN, ACF seeks to gather stakeholder input on draft changes and to 
provide a transparent account of how the review operates. The remainder 
of Section 1 summarizes recent and current HomVEE procedures and 
standards with respect to SCD research.
    After a period of public comment (and consultation with selected 
experts outside HomVEE), HomVEE will release an updated Handbook 
(Version 2.1). Section 5 of this FRN specifies where in the updated 
Version 2.1 Handbook HomVEE would insert the proposed changes.

1.1 HHS Criteria for an ``Evidence-Based Early Childhood Home Visiting 
Service Delivery Model''

    As described in Section 2 of this FRN, HomVEE is proposing changes 
to requirements for how SCD research would be considered toward the HHS 
criteria. However, HomVEE is not proposing changes to the HHS criteria 
themselves. To meet HHS criteria for an ``evidence-based early 
childhood home visiting service delivery model,'' models must meet at 
least one of the following criteria:
    (1) At least one high- or moderate-quality impact study of the 
model finds favorable, statistically significant impacts in two or more 
of the eight outcome domains.
    (2) At least two high- or moderate-quality impact studies of the 
model using non-overlapping analytic study samples find one or more 
favorable, statistically significant impacts in the same domain.
    In both cases, the impacts must either (1) be found in the full 
sample for the study, or (2) if found for subgroups but not for the 
full sample, be replicated in the same domain in two or more studies 
using non-overlapping analytic study samples.
    Additionally, following the MIECHV authorizing statute, if the 
model meets the above criteria based on findings from randomized 
controlled trials only, then two additional requirements apply. First, 
one or more favorable (statistically significant) impacts must be 
sustained for at least one year after program enrollment. Second, one 
or more favorable (statistically significant) impacts must be reported 
in a peer-reviewed journal.
    Since HomVEE's inception, research about SCD studies has had to 
satisfy certain requirements before manuscripts about those studies 
could be considered toward the HHS criteria. These requirements, known 
as the ``5-3-20 rule,'' are as follows:
    (1) At least five studies examining the intervention either met the 
WWC's SCD standards without reservations or met those standards with 
reservations (corresponding to a ``high'' or ``moderate'' rating in 
HomVEE, respectively).
    (2) The SCDs were conducted by at least three research teams, with 
no overlapping authorship at three institutions.
    (3) The combined number of cases was at least 20.
    Beyond the 5-3-20 rule, the question of statistical significance is 
relevant to findings from SCD research because the HHS criteria focus 
on favorable, statistically significant findings. Authors of SCD 
manuscripts rarely describe their findings in terms of statistical 
significance, which HomVEE needs to apply the HHS criteria. However, 
the WWC Version 4.1 Procedures Handbook, adopted in the HomVEE Version 
2 Handbook, introduces the design-comparable effect size (D-CES), from 
which statistical significance can be determined.

2.0 Proposed Changes to How SCD Research Would Be Considered Toward HHS 
Criteria for an ``Evidence-Based Early Childhood Home Visiting Service 
Delivery Model''

    The HomVEE Version 2 Handbook adopted the WWC Single-Case Design 
Procedures and Standards (Version 4.1). Therefore, the latest round of 
updates to each of the WWC and HomVEE procedures and standards included 
three key complementary changes. The changes, described in more detail 
in Subsections 2.1 through 2.3, are as follows:
    (1) The WWC no longer uses the ``5-3-20'' rule, which established 
thresholds for when SCD research could contribute to evidence ratings. 
Consequently, HomVEE's previous requirements for SCD research about an 
early childhood home visiting model to be considered toward the HHS 
criteria (see Section 1.1) no longer align with current best practices 
for systematic reviews.
    (2) Under HomVEE's new procedures as defined in the Version 2 
Handbook and adopted from the WWC, it is now possible to calculate 
effect sizes and determine statistical significance for SCD research 
using the D-CES. Consequently, HomVEE proposes to align the application 
of the HHS criteria for SCD research to the procedures already in 
effect for other eligible research designs.
    (3) Reviewers will query authors for numerical data for calculating 
the D-CES, a step that had not been relevant under prior versions of 
procedures for HomVEE (or the WWC). Because calculation of the D-CES 
requires numerical data from each time point, HomVEE proposes to query 
authors, as needed, for the information required to calculate the D-
CES.

2.1 The 5-3-20 Rule No Longer Applies

    Additional requirements for considering results from SCD research 
toward the HHS criteria (the 5-3-20 rule) have been in effect since 
HomVEE's inception; those requirements were consistent with the WWC's 
then-current approach to SCD research. The 5-3-20 rule provided a 
threshold for determining when SCD research could contribute toward a 
decision about whether a model was evidence-based. Specifically, for 
SCD research to be considered toward the HHS criteria, the model's SCD 
research has been required to consist of at least five studies with 
high or moderate internal validity, to be conducted by three 
independent research teams, and

[[Page 15223]]

to include at least 20 cases. To date, no models reviewed by HomVEE 
have met the requirements of the 5-3-20 rule.
    The WWC Version 4.1 Procedures Handbook removed the 5-3-20 rule. 
Correspondingly, HomVEE now proposes to remove the 5-3-20 requirement 
for SCD research to be considered toward the HHS criteria.

2.2 Effect Sizes and Statistical Significance Can Now Be Computed

    The HHS criteria for evidence-based models require evidence of 
favorable, statistically significant findings. However, HomVEE's 
procedures did not previously specify how HomVEE would apply the HHS 
criteria to SCD research, for which researchers generally express 
findings in terms of visual patterns rather than statistical 
significance.
    Advances in meta-analysis have made it possible to calculate an 
effect size (the D-CES) and determine associated statistical 
significance for findings from most SCD research. Therefore, under its 
Version 4.1 procedures and standards, the WWC calculates an effect size 
from SCD studies, if feasible and appropriate, which is then treated as 
any other effect size when determining intervention ratings. Consistent 
with the WWC, the HomVEE Version 2 Handbook includes an explanation of 
how the review will calculate the D-CES for SCD findings.
    The D-CES is comparable to a standardized mean-difference effect 
size and can be interpreted similarly to effect sizes from group design 
impact studies, such as randomized controlled trials, regression 
discontinuity designs, and non-experimental group designs. Because 
HomVEE can calculate the D-CES and then use the D-CES to determine the 
statistical significance of a finding for most SCDs, it is now possible 
to align the application of the HHS criteria for SCD research to group 
design impact studies. HomVEE proposes to align the application of HHS 
criteria accordingly.
    A D-CES can be computed for SCDs using the following designs:
    (1) Multiple baseline designs focused on the same outcome across 
three or more cases.
    (2) Multiple probe designs focused on the same outcome across three 
or more cases.
    (3) Reversal/withdrawal designs focused on the same outcome across 
three or more cases.
    A D-CES cannot be computed for SCDs using the following designs:
    (1) Multiple baseline designs focused on the same case across three 
or more settings.
    (2) Multiple probe designs focused on the same case across three or 
more settings.
    (3) Several reversal/withdrawal designs focused on the same case.
    HomVEE proposes to review all eligible SCD manuscripts. HomVEE also 
proposes to determine statistical significance based on the D-CES for 
all manuscripts for which a D-CES can be calculated. For SCD 
manuscripts for which a D-CES cannot be calculated, for either design 
or data reasons, HomVEE proposes to report on the rating of the 
manuscript and the findings reported in it. However, research for which 
a D-CES cannot be calculated will not be included in the summary of 
evidence that contributes to the assessment of whether a model meets 
HHS criteria. Without a D-CES, HomVEE cannot determine statistical 
significance of the manuscript's findings.

2.3 Proposed Changes to Author Query Procedures

    SCD manuscripts frequently include graphical representation of data 
at each time point. To calculate the D-CES requires numerical data from 
each time point. HomVEE proposes to request numerical data via an 
author query, as needed. If the author does not provide the numerical 
data, then HomVEE proposes to use a software package, such as 
WebPlotDigitizer (Rohatgi 2020), to extract numerical data from 
graphical presentations. This approach is consistent with WWC 
procedures.

2.4 Summary of Proposed Changes to Requirements for SCD Research To Be 
Considered Toward the HHS Criteria

    To account for the elimination of the 5-3-20 rule and to promote 
consistency in how well-designed, well-executed impact studies are 
considered toward the HHS criteria, HomVEE proposes the following 
updates:
    (1) Eliminate the 5-3-20 additional requirement that SCD research 
must meet to be considered toward the HHS criteria and instead consider 
SCD research toward the HHS criteria in a manner consistent with other 
designs.
    (2) To support the use of SCD findings toward the HHS criteria, 
implement three procedural steps that align with the Version 4.1 
practices of the WWC, as follows:
    a. Request numerical data from authors, if necessary.
    b. Calculate the D-CES, if possible, for those designs that are 
rated moderate or high.
    c. Calculate statistical significance for the D-CES.
    These proposed changes ensure that any impact study with high or 
moderate quality, as determined by the application of HomVEE Version 2 
standards, can contribute to the determination of whether a model meets 
the HHS criteria for an ``evidence-based early childhood home visiting 
service deliver model.''

3.0 Flexibilities for Applying Certain SCD Standards

    HomVEE proposes to clarify the flexibilities related to the 
application of certain SCD standards consistent with current WWC 
practice. Such flexibilities are intended to ensure that HomVEE 
standards are not unnecessarily stringent as more SCD manuscripts are 
reviewed. HomVEE proposes to update the HomVEE Version 2 Handbook text 
to clarify how existing flexibilities may be applied in the HomVEE 
review process.

3.1. Flexibility Related to the Timing of Probe Point Collection When a 
Case Receives the Intervention

    The first flexibility concerns the collection of probe data points 
in a multiple probe SCD. The current standard is strict requiring that 
baseline probes for cases without the intervention be collected in the 
same session when another case starts the intervention. The purpose of 
the baseline probe collection is to assess whether cases not receiving 
the intervention have changes in outcomes prior to receiving the 
intervention.
    HomVEE, and the WWC, recognize the requirement that baseline probes 
for cases with and without the intervention take place in the exact 
same session may be overly restrictive. The goal of the requirement 
related to the timing of baseline data collection for cases not yet 
receiving the intervention is to provide support that any change in 
outcomes in the cases receiving the intervention is likely due to the 
intervention and not some external factor (internal validity).
    In alignment with WWC procedures, HomVEE currently grants 
exceptions to this standard for individual manuscripts or interventions 
in consultation with subject matter experts. HomVEE proposes making the 
requirement more flexible by clarifying that baseline probe points may 
be collected when the intervention is introduced or in subsequent 
baseline sessions.

3.2 Flexibilities Related to the Minimum Number of Effects Demonstrated 
or Data Points Required

    The other flexibilities relate to the demonstration of effects--
either the minimum number of effects demonstrated or the data points 
required to demonstrate an effect.

[[Page 15224]]

HomVEE currently allows exceptions to each of these standards. These 
flexibilities are needed to accommodate possible nuances in outcomes 
that may be examined in SCDs -- for example, outcomes that are 
challenging for researchers to collect without burdening families (such 
as outcomes based on skills that may be frustrating to be tested on 
repeatedly if they have not been taught) or outcomes that are dangerous 
to collect repeatedly without intervening (such as some child 
maltreatment outcomes).
    To facilitate a transparent review, HomVEE proposes to clarify the 
process for granting and documenting the application of these 
flexibilities. Specifically, HomVEE proposes to specify that, if 
warranted, the HomVEE team can grant exceptions in collaboration with 
subject matter experts, and the exception (and its rationale) will be 
documented clearly in the review and related dissemination efforts.

4.0 Timeline for HomVEE To Apply New Procedures and Standards

    HomVEE proposes to apply the new procedures and standards for SCD 
research beginning with the 2021 review.

5.0 Summary of the Proposed Changes and Placement in the Version 2 
Handbook

    Following the 2020 substantial update to HomVEE procedures and 
standards, HomVEE proposes additional changes focused specifically on 
SCD research. The table below summarizes the proposed changes to be 
made to the HomVEE Version 2 Handbook and the relevant handbook 
section(s) that HomVEE proposes to update.

----------------------------------------------------------------------------------------------------------------
                                                            Description of proposed
                                        Topic               change and relevant FRN      Relevant HomVEE Version
                                                                    section                2 Handbook section
----------------------------------------------------------------------------------------------------------------
1...........................  Requirements for           SCD studies no longer have     Section B.2.c and
                               considering single-case    additional requirements to     Exhibit II.11.
                               design (SCD) research      meet. The statistical
                               toward HHS criteria for    significance of a finding
                               an ``evidence-based        (based on a design-
                               early childhood home       comparable effect size, or D-
                               visiting service           CESCES) will be considered
                               delivery model''.          for SCD research when
                                                          applying the HHS criteria.
                                                          Manuscripts for which the D-
                                                          CESCES cannot be calculated
                                                          will be reviewed and
                                                          reported but will not
                                                          contribute to summaries of
                                                          evidence. (See Sections 2.1,
                                                          2.2, and 2.4 for more
                                                          details.).
2...........................  Author queries for         HomVEE will query authors for  Section B.1.b.i Appendix
                               numerical data for SCD     the numerical data             D, Section D.1..
                               manuscripts.               underlying graphical results
                                                          in manuscripts, as needed.
                                                          The numerical data are
                                                          necessary for calculating
                                                          the D-CESCES, which is used
                                                          to determine statistical
                                                          significance..
                                                         If the author does not
                                                          provide numerical data, then
                                                          HomVEE proposes to use a
                                                          software package to extract
                                                          numerical data from
                                                          graphical representations.
                                                          (See Section 2.3 for more
                                                          details.).
3...........................  Additional baseline data   HomVEE will require multiple   Appendix D, Section B.5,
                               point requirements for     probe designs to have          Footnote 80 and related
                               multiple probe SCDs.       baseline data in the same or   statement.
                                                          subsequent baseline session
                                                          for cases not receiving the
                                                          intervention when a case
                                                          starts the intervention.
                                                          (See Section 3.1 for more
                                                          details.).
4...........................  Flexibility in the         HomVEE may grant an exception  Appendix D, Section B.6,
                               requirement for            to this requirement in         Footnote 81.
                               demonstration of three     consultation with subject
                               attempts of intervention   matter experts. If granted,
                               effects at three           it will be documented in the
                               different points in time.  review and related
                                                          dissemination efforts,
                                                          including the rationale for
                                                          the exception. (See Section
                                                          3.2 for more details.).
5...........................  Number of data points      HomVEE may grant an exception  Appendix D, Section B.6,
                               required.                  to this requirement in         Footnotes 82 and 83.
                                                          consultation with subject
                                                          matter experts. If granted,
                                                          it will be documented in the
                                                          review and related
                                                          dissemination efforts,
                                                          including the rationale for
                                                          the exception. (See Section
                                                          3.2 for more details.).
----------------------------------------------------------------------------------------------------------------

6.0 Request for Information (RFI)

    Through this FRN, ACF is soliciting information from a broad array 
of stakeholders on the proposed revisions to HomVEE's procedures and 
standards related to SCD research. Federal, state, and local decision 
makers rely on HomVEE to know which home visiting models are effective. 
Applying the HHS criteria to SCDs as they are applied to other quasi-
experimental designs or randomized controlled trials means that well-
designed, well-executed SCD research will be treated on par with other 
forms of well-designed, well-executed impact studies.
    Responses to this FRN will inform ACF's ongoing discussion about 
HomVEE's procedures and standards with the aim of publishing a final 
HomVEE Version 2.1 Handbook of Procedures and Standards in 2021. This 
RFI is for information and planning purposes only and should not be 
construed as a solicitation or as an obligation on the part of ACF or 
HHS.

    Authority: Social Security Act Title V Sec.  511 [42 U.S.C. 
711], as extended by the Bipartisan Budget Act of 2018 (Pub. L. 115-
123) through fiscal year 2022.

Naomi Goldstein,
Deputy Assistant Secretary for Planning, Research, and Evaluation, 
Administration for Children and Families, U.S. Department of Health and 
Human Services.

References

Rohatgi, A. WebPlotDigitizer, version 4.3. 2020. Available at 
https://automeris.io/WebPlotDigitizer. Accessed February 26, 2021.
Sama-Miller, E., J. Lugo-Gil, J. Harding, L. Akers, and R. Coughlin. 
Home Visiting Evidence of Effectiveness (HomVEE) Systematic Review: 
Handbook of Procedures and Evidence Standards: Version 2, December 
2020. Available at https://homvee.acf.hhs.gov/publications/methods-standards. Accessed January 7, 2021.
U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance, 
What Works Clearinghouse. What Works Clearinghouse Procedures 
Handbook: Version 4.1., 2020a. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf. 
Accessed June 19, 2020.

[[Page 15225]]

U.S. Department of Education, Institute of Education Sciences, 
National Center for Education Evaluation and Regional Assistance, 
What Works Clearinghouse. What Works Clearinghouse Standards 
Handbook: Version 4.1., 2020b. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf. 
Accessed June 19, 2020.

[FR Doc. 2021-05840 Filed 3-19-21; 8:45 am]
BILLING CODE 4184-74-P