Maternal, Infant, and Early Childhood Home Visiting Program, 43172-43177 [2010-18013]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Health Resources and Services Administration
• Administration for Children and Families
• Children and Families Administration

[Federal Register Volume 75, Number 141 (Friday, July 23, 2010)]
[Notices]
[Pages 43172-43177]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 2010-18013]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Health Resources and Services Administration

Administration for Children and Families


Maternal, Infant, and Early Childhood Home Visiting Program

AGENCY: Health Resources and Services Administration and Administration 
for Children and Families, HHS.

ACTION: Request for public comment on criteria for evidence of 
effectiveness of home visiting program models for pregnant women, 
expectant fathers, and caregivers of children birth through 
kindergarten entry.

-----------------------------------------------------------------------

SUMMARY: The Health Resources and Services Administration and

[[Page 43173]]

Administration for Children and Families, HHS, solicit comments by 
August 17, 2010 on proposed criteria for evidence of effectiveness of 
home visiting program models for pregnant women, expectant fathers, and 
primary caregivers of children birth through kindergarten entry. Final 
criteria for evidence of effectiveness will be included in the program 
announcement inviting eligible entities to apply for funding under the 
Affordable Care Act Maternal, Infant, and Early Childhood Home Visiting 
Program.

SUPPLEMENTARY INFORMATION: Invitation to Comment: HHS invites comments 
regarding this notice, both on the proposed criteria and proposed 
methodology for HHS's systematic review of the evidence. To ensure that 
your comments have maximum effect, please identify clearly the specific 
criterion or other section of this notice that your comment addresses.

1.0 Purpose of Program

    The Affordable Care Act (ACA) Maternal, Infant, and Early Childhood 
Home Visiting program is designed to strengthen and improve home 
visiting programs, improve service coordination for at risk 
communities, and identify and provide comprehensive evidence-based home 
visiting services to families who reside in at risk communities. The 
legislation specifies that most program funds must be used for 
``evidence-based'' home visiting program models. This Notice (1) 
proposes criteria to be considered in assessing whether home visiting 
models have evidence of effectiveness, and (2) describes the 
methodology for a systematic review of evidence, applying the criteria 
proposed in this Notice, which HHS is currently conducting. The Notice 
solicits comments on both items.

2.0 Background

2.1 Legislative Context

    On March 23, 2010, the President signed into law the Patient 
Protection and Affordable Care Act of 2010 (Pub. L. 111-148) (also 
known as the Affordable Care Act or ACA); historic legislation designed 
to make quality, affordable health care available to all Americans, 
reduce costs, improve health care quality, enhance disease prevention, 
and strengthen the health care workforce. Through a provision adding 
Section 511 to Title V of the Social Security Act to create the 
Maternal, Infant, and Early Childhood Home Visiting program, the Act 
responds to the diverse needs of children and families in at risk 
communities and provides an historic and unique opportunity for 
collaboration at the Federal, State, and local level to assure 
coordination and delivery of critical health, development, early 
learning, and child abuse and neglect prevention services to most 
effectively serve these children and families. By supporting evidence-
based home visiting program models, the ACA Maternal, Infant, and Early 
Childhood Home Visiting program plays a crucial role in national 
efforts to build quality, comprehensive statewide early childhood 
systems for pregnant women, parents, and caregivers, and children from 
birth to 8 years of age.
    The ACA Maternal, Infant, and Early Childhood Home Visiting Program 
is designed: (1) To strengthen and improve the programs and activities 
carried out under Title V; (2) to improve coordination of services for 
at risk communities; and (3) to identify and provide comprehensive 
services to improve outcomes for families who reside in at risk 
communities. At risk communities will be identified through a statewide 
assessment of needs and of existing resources to meet those needs. HHS 
intends that the home visiting program will result in a coordinated 
system of early childhood home visiting in every State that has the 
capacity to provide infrastructure and supports to assure high-quality, 
evidence-based practice.
    The program enables eligible entities to utilize what is known 
about effective home visiting services to provide evidence-based 
programs to promote: improvements in prenatal, maternal and newborn 
health; child health and development including prevention of child 
injuries and maltreatment and improvements in cognitive, language, 
social-emotional, and physical development; parenting skills; school 
readiness; reductions in crime or domestic violence; improvements in 
family economic self-sufficiency; and improvements in the coordination 
and referrals for other community resources and supports.

2.2 Use of Funds for ``Evidence-Based'' Programs

    Section 511(d)(3)(A) of Title V, as amended by the Affordable Care 
Act, reserves the majority of grant funds for home visiting program 
models with evidence of effectiveness based on rigorous evaluation 
research. The legislation specifies that models must meet the following 
requirements in order to be considered ``evidence-based'':

    (I) The model conforms to a clear consistent home visitation 
model that has been existence for at least 3 years and is research-
based, grounded in relevant empirically-based knowledge, linked to 
program determined outcomes, associated with a national organization 
or institution of higher education that has comprehensive home 
visitation program standards that ensure high quality services 
delivery and continuous program improvement, and has demonstrated 
significant, (and in the case of the service delivery model 
described in item (aa), sustained) positive outcomes, as described 
in the benchmark areas specified in paragraph (1)(A) and the 
participant outcomes described in paragraph (2)(B), when evaluated 
using a well-designed and rigorous--
    (aa) randomized controlled research designs, and the evaluation 
results have been published in a peer-reviewed journal; or
    (bb) quasi-experimental research designs.

    The legislation charges the Secretary of Health and Human Services 
with establishing criteria for evidence of effectiveness of the home 
visiting program models and ensuring the process for establishing the 
criteria is transparent and provides the opportunity for public 
comment.
    This Notice (1) proposes criteria to be considered in assessing 
whether home visiting models have evidence of effectiveness and (2) 
describes the methodology for a systematic review of evidence, applying 
the criteria proposed in this Notice, which HHS is currently 
conducting. The Notice solicits comments on both items. After comments 
are received, HHS will finalize the criteria and methodology and 
complete the systematic review of the available evidence of 
effectiveness of selected home visiting program models.
    It is expected that eligible entities will also have an opportunity 
to present documentation in their applications for the ACA Maternal, 
Infant, and Early Childhood Home Visiting program to demonstrate that 
additional home visiting models meet the final criteria. Such 
documentation will be reviewed by HHS using the same procedures applied 
in HHS' systematic review and described below.
    The criteria proposed in this notice apply only to the home 
visiting program for States and territories authorized by Section 
511(c) of Title V. Criteria for the ACA Tribal Maternal, Infant, and 
Early Childhood Home Visiting Program authorized by Section 
511(h)(2)(A) of Title V will be issued separately. Based on a careful 
review of available research evidence on home visiting interventions 
with Tribal populations, the Secretary will develop alternative 
evidence-based criteria for identifying home visiting models likely to 
improve outcomes for families in Tribal communities.

[[Page 43174]]

3.0 Proposed Criteria for Evidence of Effectiveness

    A home visiting model must have been (1) evaluated using rigorous 
methodology and (2) shown to have a positive impact on outcomes in 
order to meet criteria for evidence of effectiveness. The following two 
types of criteria (3.1 and 3.2) must be met in order for a home 
visiting model to be considered evidence-based for the purposes of the 
Maternal, Infant, and Early Childhood Home Visiting Program:

3.1 Criteria for Well-Designed, Rigorous Impact Research

    In order to ensure the highest probability of producing unbiased 
estimates of program impacts, there are a number of variables that 
should be considered. These variables include study design (i.e. 
randomized controlled trial [RCT] or quasi-experimental design [QED]), 
level of attrition, baseline equivalence, reassignment of participants 
from one condition to another in the trial, the reliability and 
validity of outcome measures studied, and confounding factors.
    Two types of impact study designs have the potential to be both 
well-designed and rigorous: Randomized controlled trials and quasi-
experimental designs. HHS proposes to define randomized controlled 
trials as a study design in which sample members are assigned to the 
program and comparison groups by chance. Randomized control designs are 
often considered the ``gold standard'' of research design because 
personal characteristics (before the program begins) do not affect 
whether someone is assigned to the program or control group. HHS 
proposes to define a quasi-experimental design as a study design in 
which sample members are selected for the program and comparison groups 
in a nonrandom way. For example, families may self-select into groups 
(deciding whether they want services or not) or an administrator may 
assign families to groups based on family risk factors. Quasi-
experimental designs are considered weaker than randomized controlled 
trials because characteristics that may be related to outcomes, such as 
motivation or need, may also influence whether someone is in the 
program or comparison group.
    HHS proposes that an impact study will be considered high, moderate 
or low quality depending on the study's capacity to provide unbiased 
estimates of program impact. Studies that are rated ``high'' and 
``moderate'' (see Table 1 below), therefore, would meet requirements to 
be considered ``well-designed, rigorous impact research.'' In brief, 
the high rating would be reserved for random assignment studies with 
low attrition of sample members and no reassignment of sample members 
after the original random assignment. The moderate rating would apply 
to studies that use a quasi-experimental design and to random 
assignment studies that, due to flaws in the study design or execution 
(for example, high sample attrition), do not meet all the criteria for 
the high rating. To receive the moderate rating, studies would have to 
demonstrate that at the study's onset, the intervention and comparison 
groups were well matched on specified measures (i.e. baseline 
equivalence), such as a pretest measure of targeted outcomes or race 
and maternal education. Studies that do not meet all of the criteria 
for either high or moderate quality would be considered low quality.
    As summarized in Table 1, the rating scheme would consider five 
dimensions: (1) Study design, (2) attrition, (3) baseline equivalence, 
(4) reassignment, and (5) confounding factors.

                          Table 1--Criteria for Well-Designed, Rigorous Impact Research
----------------------------------------------------------------------------------------------------------------
                                                                         Rating
           Rating Criteria            --------------------------------------------------------------------------
                                                 High                   Moderate                   Low
----------------------------------------------------------------------------------------------------------------
Study design.........................  Random assignment......  Quasi-experimental       Studies that do not
                                                                 design with a            meet the requirements
                                                                 comparison group;        for a high or moderate
                                                                 random assignment        rating.
                                                                 design with high
                                                                 attrition or any
                                                                 reassignment.
Attrition............................  Meets ``What Works       No requirement.........
                                        Clearinghouse'' (Dept.
                                        of Education)
                                        standards for
                                        acceptable rates of
                                        overall and
                                        differential attrition.
Baseline equivalence.................  No requirement other     Must establish baseline
                                        than random              equivalence of study
                                        assignment;              arms on selected
                                        Statistically            measures (see Table 1,
                                        significant              Note 2 below).
                                        differences must be
                                        controlled.
Reassignment (see Table 1, Note 1      Analysis must be based   No requirement.........
 below).                                on original assignment
                                        to study arms.
Confounding factors..................  Must have at least two   Must have at least two
                                        participants in each     participants in each
                                        study arm and no         study arm and no
                                        systematic differences   systematic differences
                                        in data collection       in data collection
                                        methods.                 methods.
----------------------------------------------------------------------------------------------------------------

    Table 1, Note 1: In random assignment studies, deviation from the 
original random assignment (for example, moving families from the 
treatment to the control group) can also bias the impact estimates. 
Therefore, in order for a RCT to meet our criteria for the high rating, 
the analysis must be performed on the sample as originally assigned. 
Subjects may not be reassigned for reasons such as contamination, 
noncompliance, or level of exposure. RCTs that somehow alter the 
original random assignment but otherwise meet the criteria for the high 
rating are considered for a moderate study rating, provided they meet 
the other criteria for that rating. Our criteria are similar to those 
developed by the WWC, which allows a study to be downgraded as a result 
of reassignment.
    Table 1, Note 2: When possible, baseline equivalence should be 
established on outcomes of interest. For some studies it is not 
feasible to collect baseline measures on the outcome of interest, for 
example, children's

[[Page 43175]]

outcomes when baseline is collected prenatally. For all studies, 
baseline equivalence must be established on two demographic factors: 
(1) The parent or child's race and ethnicity and (2) socioeconomic 
status.

3.2 HHS Proposed Criteria for Evidence of Effectiveness of a Home 
Visiting Service Delivery Model

    In order to have confidence in the impact estimates created from a 
high or moderate quality study design, a number of variables should be 
considered. These variables include statistical significance, whether 
impacts are sustained, and whether the impacts were found for the full 
sample or only for non-replicated subgroups.
    3.2.1 The ACA Maternal, Infant and Early Childhood Home Visitation 
Program legislation includes a number of participant outcome and 
benchmark areas. In determining program effectiveness HHS proposes to 
examine programs for impacts in the following eight program domains:
    (1) Maternal health
    (2) Child health
    (3) Child development and school readiness, including improvements 
in cognitive, language, social-emotional or physical development
    (4) Prevention of child injuries and maltreatment
    (5) Parenting skills
    (6) Reductions in crime or domestic violence
    (7) Improvements in family economic self-sufficiency
    (8) Improvements in the coordination and referrals for other 
community resources and supports.
    3.2.2 Taking into account the legislative language and the two 
types of criteria discussed in 3.1 and 3.2 above, HHS proposes to 
consider a program model eligible for evidence-based funding for the 
purposes of the ACA Maternal, Infant, and Early Childhood Home Visiting 
Program if it meets the following minimum criteria:
     At least one high- or moderate-quality impact study (see 
3.1) of the program model finds favorable, statistically significant 
impacts in two or more of the eight outcome domains (see 3.2.1); or
     At least two high- or moderate-quality impact studies 
using different samples (see 3.1) of the program model find one or more 
favorable, statistically significant impacts in the same domain (see 
3.2.1).
    In both cases, the impacts considered must be found either for the 
full sample or, if found for subgroups but not for the full sample, 
impacts must be replicated in the same domain in two or more studies 
using different samples.
    Additionally, if the program model meets the above criteria based 
on findings from randomized control trial(s) only, then one or more 
impacts in an outcome domain must be sustained for at least one year 
after program enrollment, and one or more impacts in an outcome domain 
must be reported in a peer-reviewed journal (consistent with section 
511(d)(3)(A)(i)(I)).
    Isolated positive findings, and impacts found only for a subgroup, 
but not the full sample in a study, raise concerns about false 
positives that may be artifacts of multiple statistical tests rather 
than reflecting true impacts. The requirements for replication of 
positive findings across samples or for findings in two or more outcome 
domains are meant to guard against this problem. HHS recognizes the 
importance of subgroup findings for determining impacts on subgroups of 
the population of interest, including specific racial or ethnic groups, 
and plans to report information on subgroup findings, whether 
replicated or not.

4.0 Proposed Methods for HHS's Systematic Review of Evidence of 
Effectiveness

    HHS is conducting a comprehensive and detailed program model-by-
model review of the available evidence of effectiveness of home 
visiting programs that support the following legislatively specified 
benchmarks and outcomes: Maternal health; child health; child 
development and school readiness including improvements in cognitive, 
language, social-emotional, and physical development; prevention of 
child injuries and maltreatment; parenting skills; reductions in crime 
or domestic violence; improvements in family economic self-sufficiency; 
and improvements in the coordination and referrals for other community 
resources and supports.
    The review is being carried out through a contract to Mathematica 
Policy Research, Inc. and led by the Administration for Children and 
Families in collaboration with the Health Resources and Services 
Administration, the Office of the Assistant Secretary for Planning and 
Evaluation, and the Centers for Disease Control and Prevention. The 
review will apply the HHS criteria proposed above to determine which of 
the program models reviewed meet the criteria for evidence of 
effectiveness. The review will be completed after comments on this 
notice are received and considered.

4.1 Review Process

    To conduct a through and transparent review of the home visiting 
program model research literature, the systematic review project is 
following five main steps, the first three of which have been 
provisionally completed. Comments on steps 4 and 5 are especially 
encouraged.
    1. Conduct a broad literature search;
    2. Screen studies for relevance;
    3. Prioritize program models for review;
    4. Rate the quality of impact studies with eligible designs;
    5. Assess the evidence of effectiveness for each program model.
    In addition, the project plans to review and make available 
implementation information for each program. Steps taken to address 
potential conflicts of interest are also described below.
4.1.1 Step 1: Conduct a Broad Literature Search
    The literature search included four main activities:
    1. Database Searches. The project team searched on relevant key 
words in a range of research databases. Key words included terms 
related to the service delivery approach, target population, and 
outcome domains emphasized in the Patient Protection and Affordable 
Care Act. The initial search was limited to studies published since 
1989; a more focused search on prioritized program models included 
studies published since 1979 (see Prioritizing Programs below).
    2. Web Site Searches. The project team used a custom Google search 
engine to search more than 50 relevant government, university, 
research, and nonprofit Web sites for unpublished reports and papers.
    3. Call for Studies. In November 2009, Mathematica issued a call 
for studies and sent it to approximately 40 relevant listservs for 
dissemination.
    4. Review of Existing Literature Reviews and Meta-Analyses. The 
project team checked search results against the bibliographies of 
recent literature reviews and meta-analyses of home visiting programs 
and added relevant missing citations to the search results.
    The literature search yielded approximately 8,200 unduplicated 
citations, including 150 articles submitted through the call for 
studies.
4.1.2 Step 2: Screen Studies for Relevance
    The project team then screened all citations identified through the 
literature search for relevance. Studies were screened out for the 
following reasons:

[[Page 43176]]

     The model under study did not use home visitation as a 
primary service delivery strategy. Programs that are primarily center-
based with infrequent or supplemental home visiting were excluded. In 
order to be considered a home visiting model, a program must offer home 
visiting services to most or all participants and these services must 
be integral to programmatic goals. Visits should occur solely or 
primarily where participating families reside but occasionally may 
occur elsewhere if the families are homeless or uncomfortable 
conducting visits in the home. The services could be voluntary or 
mandated (for example, court ordered).
     The study did not use an eligible design as described in 
3.1 above (randomized controlled trial, quasi-experimental design). The 
project team also included any studies on the implementation of 
specific home visiting models. These studies were used in the 
implementation reports described in section 5.0 of this Notice.
     The program did not include pregnant women and families 
with children from birth to kindergarten entry.
     The study did not examine any outcomes in the domains of: 
Maternal health and/or child health; child development and school 
readiness; reductions in child maltreatment; reductions in juvenile 
delinquency, family violence, and crime; positive parenting practices; 
and family economic factors. The legislatively specified domain of 
improvement in coordination and referrals for community resources and 
supports was not used in screening because of challenges in specifying 
discrete measures.
     The study did not examine a clear home visiting program 
model. For example, the study might focus on a specific home visiting 
strategy, such as comparing the use of professional and 
paraprofessional home visiting staff within home visiting program model 
broadly rather than a specific program model. Without a clearly 
identified program model, the evidence review could not use the impact 
study to assess the effectiveness of a specific program.
     The study was not published in English. This limitation 
reflects practical considerations, given the limited time available for 
the review.
     The study was published before 1989 for the initial search 
or 1979 for the focused search on prioritized program models. These 
limitations balance practical considerations, given limited time 
available, and were designed to ensure that seminal research was 
included.
4.1.3 Step 3: Prioritize Program Models for Review
    After screening, the initial search yielded studies on more than 
250 home visiting program models. Timing and resources do not allow for 
a detailed review of all of these home visiting program models prior to 
the implementation of the ACA Maternal, Infant, and Early Childhood 
Home Visiting Program. For each model the team examined the number and 
design of impact studies, sample sizes of the impact studies, the 
availability of implementation information, whether the program was 
currently in widespread use in the U.S., and whether the program had 
been implemented only in a developing-world context. The project staff 
eliminated programs that had no information available about 
implementation, were implemented only in a developing-world context, or 
were no longer in operation and provided no support for implementation. 
This decision was made so that resources could be focused on reviewing 
program models that States and territories would be readily able to 
implement and that would be likely to meet other statutory 
requirements.
4.1.4 Step 4: Rating the Quality of Impact Studies
    For the purposes of the systematic review, HHS plans to assign each 
impact study a rating of high, moderate, or low, per the criteria 
described in 3.1 above.
4.1.5 Step 5: Assessing Evidence of Effectiveness
    After rating the quality of all available impact studies for a 
program, HHS plans to assess the evidence across all studies of the 
program models that received a high or moderate rating and measured 
outcomes in at least one of the legislatively specified participant 
outcome domains utilizing the HHS proposed criteria for evidence for 
effectiveness discussed in 3.2 above.

5.0 Implementation Reviews

    To assist in implementation of the ACA Maternal, Infant and Early 
Childhood Home Visiting Program, the project plans to collect and 
publish information about implementation of the prioritized program 
models. The project plans to provide two kinds of implementation 
reports for each program model. One implementation report will focus on 
the support available to assist interested entities to implement the 
model (such as program model technical assistance staff or trainings) 
or infrastructure required to implement the model (such as the purchase 
of a specific data management system or curricula). The second kind of 
implementation report will focus on implementation experiences during 
the impact trials or in implementing the model in the field. These 
reports will provide information on the study samples in the impact 
trials, describe the locations where the specific model has been 
implemented, the average number of visits the participants receive, any 
available research on adaptations of the program models and lessons 
learned about implementing the models that have been reported in the 
available research.

6.0 Addressing Conflicts of Interest

    All members of the review team have signed a conflict of interest 
statement in which they declared any financial or personal connections 
to developers, studies, or products being reviewed and confirmed their 
understanding of the process by which they must inform the project 
director if such conflicts arise. The review team's project director 
assembled signed conflict of interest forms for all project staff and 
subcontractors and monitors for possible conflicts over time. If a team 
member is found to have a potential conflict of interest concerning a 
particular home visiting program model being reviewed, that team member 
is excluded from the review process for the studies of that program 
model. In addition, reviews for two programs evaluated by Mathematica 
Policy Research are being conducted by contracted reviewers who are not 
Mathematica[reg] employees.

7.0 Future Allocations Based on Application Strength

    To encourage exemplary programs and direct Federal funds where they 
can have the greatest impact, HHS plans to allocate the ACA Maternal, 
Infant and Early Childhood Home Visiting Program funding available in 
future years that exceeds funding available in FY 2010 competitively 
based upon States' capacity and commitment to improve child outcomes 
specified in the statute through improvements in service coordination 
and the implementation of home visiting programs with fidelity to high-
quality, evidence-based models. HHS plans to evaluate applications 
based on multiple criteria and invites comments on what criteria are 
appropriate. Among the criteria, HHS proposes to give significant 
weight to the strength of the available evidence of effectiveness of 
the model or models employed by the State. In this context, the use of 
program models satisfying the

[[Page 43177]]

criteria outlined in section 3.2.2 would be a minimal requirement, but 
HHS would consider additional criteria that further distinguish models 
with greater and lesser support in evidence. HHS is committed to 
ensuring that these criteria are transparent, methodologically sound, 
and increase the likelihood that federal funds will contribute to 
improved outcomes for at-risk children and families.
    There are a number of different ways that such a system could be 
structured. We invite comments on the proposal to distinguish among 
evidence-based models based on a rubric that weighs factors relating to 
research quality and findings. For example, one relatively simple 
approach would rate models using an index constructed by weighting 
several factors equally. Models might be given points for meeting each 
of the following criteria: Favorable impacts sustained at least one 
year after program completion, favorable impacts replicated in distinct 
samples, favorable impacts in studies conducted by independent 
evaluators, quality and relevance of outcome measures; and balance 
between favorable and unfavorable and null findings. Additional factors 
which might be considered could include further indicia of the quality 
of the research design and implementation (as reflected in 
randomization, sample size, attrition, and baseline equivalence). We 
invite comments on HHS' proposal to use evidence for program models as 
a factor in determining allocations of additional funds, how various 
factors should be weighed in assessing the evidence of effectiveness, 
how to define these categories, and any other role distinctions related 
to the strength of the evidence should play in funding allocation. As 
noted above, strength of evidence is proposed to be only one factor in 
the evaluation of the strength of States' applications, and we invite 
comments on other appropriate factors as well.

8.0 Future Considerations

    We invite comment on the following:
     HHS anticipates the criteria for evidence-based models 
will likely need to be altered over time as the state of the field 
changes. If HHS believes the criteria need to be changed in future 
years, it is anticipated the public will have an opportunity to comment 
on the proposed revisions.
     HHS intends to review the evidence base for home visiting 
models on an ongoing basis to ensure that new evidence is incorporated.

9.0 Submission of Comments

    Comments may be submitted until August 17, 2010 by e-mail to: 
mpr.com">HVEE@mathematica-mpr.com.

    Dated: July 19, 2010.
Mary K. Wakefield,
Administrator, Health Resources and Services Administration.
Carmen R. Nazario,
Assistant Secretary, Administration for Children and Families.
[FR Doc. 2010-18013 Filed 7-22-10; 8:45 am]
BILLING CODE 4165-15-P