Notice: Request for Comments; National Registry of Evidence-Based Programs and Practices (NREPP), 50381-50390 [05-17034]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Substance Abuse and Mental Health Services Administration (samhsa)
• Substance Abuse and Mental Health Services Administration

[Federal Register Volume 70, Number 165 (Friday, August 26, 2005)]
[Notices]
[Pages 50381-50390]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 05-17034]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Substance Abuse and Mental Health Services Administration (SAMHSA)


Notice: Request for Comments; National Registry of Evidence-Based 
Programs and Practices (NREPP)


    Authority: Sec. 501, Pub. L. 106-310

SUMMARY: The Substance Abuse and Mental Health Services Administration 
(SAMHSA) is committed to preventing the onset and reducing the 
progression of mental illness, substance abuse and substance related 
problems among all individuals, including youth. As part of this 
effort, SAMHSA is expanding and refining the agency's National Registry 
of Evidence-based Programs and Practices (NREPP) so that the system 
serves as a leading national resource for contemporary and reliable 
information on the scientific basis and practicality of interventions 
to prevent and/or treat mental illness and substance use and abuse.
    NREPP represents a major agency activity within SAMHSA's Science to 
Service initiative. The initiative seeks to accelerate the translation 
of research into practice by promoting the implementation of effective, 
evidence-based interventions for preventing and/or treating mental 
disorders and substance use and abuse. Of equal measure, the initiative 
emphasizes the essential role of the services community in providing 
input and feedback to influence and better frame the research questions 
and activities pursued by researchers in these areas.
    Through SAMHSA's Science to Service initiative, the agency 
ultimately seeks to develop a range of tools that will facilitate 
evidence-based decision-making in substance abuse prevention, mental 
health promotion, and the treatment of mental and substance use 
disorders. In addition to NREPP, SAMHSA is developing an informational 
guide of web-based

[[Page 50382]]

resources on evidence-based interventions that will be available in 
2006. SAMHSA also is exploring the feasibility of supporting a 
searchable web database of evidence-based information (e.g., systematic 
reviews, meta-analyses, clinical guidelines) for mental health and 
substance abuse prevention and treatment providers. Such a system could 
reduce the lag time between the initial development and broader 
application of research knowledge by serving as a real-time resource to 
providers for ``keeping current'' in ways that will enhance their 
delivery of high quality, effective services. In combination, these 
three tools--NREPP, guide to web-based resources, and database of 
evidence-based information--would provide valuable information that can 
be used in a variety of ways by a range of interested stakeholders.
    With regard to NREPP, during the past two years, SAMHSA convened a 
series of scientific/stakeholder panels to inform the agency's 
expansion of the system to include interventions in all substance abuse 
and mental health treatment and prevention domains. These panels 
thoroughly assessed the existing NREPP review process and review 
criteria and provided comments and suggestions for refining and 
enhancing NREPP. As part of this expansion effort, SAMHSA also engaged 
a contractor to assess the NREPP process and review criteria, including 
how the system and criteria compare to other, similar evidence review 
and rating systems in the behavioral and social sciences. The 
cumulative results of these activities have guided efforts to refine 
the NREPP review process and review criteria, as well as inform the 
agency's plans for how such a system may be used to promote greater 
adoption of evidence-based interventions within typical community-based 
settings.
    This Federal Register Notice (FRN) provides an opportunity for 
interested parties to become familiar with and comment on SAMHSA's 
plans for expansion and use of NREPP.

DATES: Submit comments on or before October 25, 2005.

ADDRESSES: Address all comments concerning this notice to: SAMHSA c/o 
NREPP Notice, 1 Choke Cherry Road, Rockville, MD 20857. See 
SUPPLEMENTARY INFORMATION for information about electronic filing.

FOR FURTHER INFORMATION CONTACT: Kevin D. Hennessy, PhD, Science to 
Service Coordinator/SAMHSA, 1 Choke Cherry Road, Room 8-1017, 
Rockville, Maryland 20857. Dr. Hennessy may be reached at (240) 276-
2234.

SUPPLEMENTARY INFORMATION:

Electronic Access and Filing Addresses

    You may submit comments by sending electronic mail (e-mail) to 
nrepp.comments@samhsa.hhs.gov.

    Dated: August 18, 2005.
Charles G. Curie,
Administrator.

Overview

    Increasingly, individuals and organizations responsible for 
purchasing, providing and receiving services to prevent substance abuse 
and/or treat mental and substance use disorders are considering the 
extent to which these services are ``evidence-based''--that there 
exists some degree of documented scientific support for the outcomes 
obtained by these services. As the Federal agency responsible for 
promoting the delivery of substance abuse and mental health services, 
SAMHSA is particularly interested in supporting and advancing 
activities that encourage greater adoption of effective, evidence-based 
interventions to prevent and/or treat mental and substance use 
disorders. With this in mind, SAMHSA proposes to refine and expand its 
National Registry of Evidence-based Programs and Practices (NREPP). 
SAMHSA believes that the growth and evolution of NREPP can serve as an 
important mechanism for promoting greater adoption of evidence-based 
substance abuse and mental health services,--one that can do so in 
conjunction with an ever-growing array of scientific knowledge, 
clinical expertise and judgment, and patient/recipient values and 
perspectives. By clearly identifying and assessing the scientific basis 
and disseminability of a range of behavioral interventions, NREPP is 
likely to prove an important resource to both individuals and systems 
seeking information on the effectiveness of various services to prevent 
and/or treat mental and substance use disorders.

Background and Need

    As SAMHSA promotes the identification and greater use of effective, 
evidence-based interventions for individual-, population-, policy-, and 
system-level changes, the agency seeks to build upon the strong 
foundation provided by the precursor to the National Registry of 
Evidence-based Programs and Practices--namely, the National Registry of 
Effective Prevention Programs. The previous system provides an 
important building block in the agency's efforts to develop a SAMHSA-
wide registry.
    The National Registry of Effective Prevention Programs developed in 
SAMHSA's Center for Substance Abuse Prevention (CSAP) beginning in 1997 
as a way to help professionals in the field become better consumers of 
prevention programs. Between 1997 and 2004, NREPP reviewed and rated 
more than 1,100 prevention programs, with more than 150 obtaining 
designation as a Model, Effective, or Promising Program.
    Information on all current NREPP programs is available through the 
Model Programs Web site at https://www.modelprograms.samhsa.gov. 
Additional details about the review process, review criteria, and 
rating system for the National Registry of Effective Prevention 
Programs are available in the SAMHSA publication ``Science-Based 
Prevention programs and Principles 2002,'' which can be downloaded from 
SAMHSA's Model Programs Web site https://www.modelprograms.samhsa.gov) 
by clicking on ``Publications'' on the tool bar on the left side of the 
page; or by requesting the publication through SAMHSA's National 
Clearinghouse for Alcohol and Drug Information (NCADI) at 1-800-729-
6686 (or by visiting the NCADI Web site at https://www.health.org).
    As SAMHSA expands the NREPP system, one area of potential 
improvement is in the efficient screening and triage of applications. 
Given the historical applications trends among substance abuse 
prevention programs, combined with the increased demands on the system 
through expansion to other SAMHSA domains, it is essential that the 
agency develop a transparent and scientifically defensible process for 
screening and triaging applications.
    Moreover, as SAMHSA engaged NREPP scientific/stakeholder panels 
over the past 2 years, concerns about the existing review process and 
review criteria emerged. In particular, a range of scientific experts 
voiced concerns regarding specific review criteria and other elements 
of the review process.
    In addition, systematic efforts to examine and compare the current 
NREPP review criteria with other evidence-grading systems in the social 
and behavioral sciences has revealed both areas of relative strength 
and relative weakness. At a minimum, this comparison has affirmed for 
SAMHSA the importance and value of reexamining and refining the NREPP 
review process and review criteria in ways that reflect to the public 
SAMHSA's commitment to identifying

[[Page 50383]]

and promoting interventions that have shown to be effective through 
prevailing scientific standards. One important element of this process 
is providing support for the re-review of existing NREPP programs 
against these prevailing scientific standards (see below), while 
another component is identifying both SAMHSA and other mechanisms and 
resources for supporting efforts to evaluate and document the evidence-
base of innovative interventions in ways that will maximize their 
opportunity for entry into NREPP.
    Further, SAMHSA's experience with NREPP to date suggests that the 
system is limited in its ability to identify and rate interventions 
designed to promote population-, policy-, and system-level outcomes, 
such as those promoted by community prevention coalitions. SAMHSA's 
plans for NREPP include an expansion of the system in this area. As 
part of this expansion, SAMHSA proposes a second set of review criteria 
for these interventions, with the recognition that some interventions 
may be designed to affect a community over time, and that the 
prevailing scientific standards for assessing the effectiveness of 
these interventions may indeed be different than those for 
interventions seeking to change individual-level outcomes. Finally, 
input into the NREPP process to date suggests the need for SAMHSA to 
provide greater policy guidance on how best to use the system to 
appropriately select specific interventions, as well as contextual 
guidance on how NREPP might be used in conjunction with other important 
information--such as clinical expertise, patient values, and 
administrative and policy perspectives and data--in making decisions 
regarding the financing and delivery of substance abuse and mental 
health services.

Proposal

    After extensive consultation with both scientific experts and a 
range of stakeholders. SAMHSA is seeking your comments on a proposal to 
advance a voluntary rating and classification system for mental health 
and substance abuse prevention and treatment interventions--a system 
designed to categorize and disseminate information about programs and 
practices that meet established evidentiary criteria. This proposal 
presents in detail the new NREPP system, including refinements to the 
review process and review criteria for programs and practices in 
substance abuse and mental health, as well as an expansion of the 
system to include successful community coalition efforts to achieve 
population-, policy-, or system-level outcomes. The proposal also 
describes SAMHSA's plans for a new Web site that will highlight the 
scientific evidence-base of interventions, as well as provide a range 
of practical information needed by those who are considering 
implementation of programs or practices on the Registry.
    SAMHSA further anticipates that additional revisions and 
refinements to the NREPP system may be needed on a periodic basis, and 
proposes the formation of an external advisory panel to regularly 
assist the agency in assessing proposed suggestions for improvements to 
the system (see Question  10 below).

Initial Input From the Field

    Upon determining that SAMHSA would expand NREPP to include 
interventions in all agency domains, three expert panel meetings of 
both scientist and nonscientist stakeholders were convened to provide 
feedback on the current review system and review criteria, as well as 
solicit suggestions about redesigning the system to promote the goals 
noted above. Each meeting was conducted over a 2-day period, and 
included invited participants representing a range of relevant 
organizations, expertise, and perspectives. All meetings took place in 
Washington, DC, in 2003, with mental health experts meeting in April, 
substance abuse prevention and mental health promotion experts meeting 
in September, and substance abuse treatment experts meeting in 
December. Transcripts of these meetings are available on-line at the 
NREPP Web page accessible through the ``Quick Picks'' section on 
SAMHSA's Home page (https://www.samhsa.gov).
    SAMHSA also convened a meeting in May 2005 to solicit 
recommendations for integrating evidence-based findings from community 
coalitions into NREPP. The 2-day meeting brought together prominent 
researchers and practitioners who reaffirmed the importance of 
including prevention coalitions within NREPP, and offered suggestions 
as to the types of outcomes and evidence criteria appropriate to the 
assessment of community coalitions. A summary of this meeting is 
available on-line at the NREPP web page accessible through the ``Quick 
Picks'' section on SAMHSA's Home page (https://www.samhsa.gov).

Review Process for Determining Individual-Level Outcome Ratings for 
Interventions

    A primary goal of the Registry is to provide the public with 
reliable information about the evidence quality and strength of 
scientific support for specific interventions. The strength of 
scientific support includes: the quality of evaluation design (e.g., 
experimental or quasi-experimental designs); fidelity to predetermined 
intervention components; confidence in the link between intervention 
components and specific outcome(s) achieved; freedom from internal and 
external sources of bias and error; and the statistical significance 
and practical magnitude (e.g., effect size) of outcomes achieved.
    An additional goal is to provide key information about the 
transferability of these programs and practices to real-world 
prevention and treatment settings. NREPP utility descriptors provide 
information about the appropriate settings and populations for 
implementation of specific interventions, the availability of training 
and dissemination materials, and their practicality and costs.
    This section describes the NREPP review process, including the 
evidence rating criteria and utility descriptors that will form the 
basis for Web-based information about programs and practices.
    Based on important feedback from scientists and practitioners in 
the prevention and treatment fields, the NREPP review process has been 
enhanced in several important respects:

--Programs and practices will be rated on the strength of evidence for 
specific outcomes achieved, rather than on global assessments of the 
effectiveness of intervention(s). In addition, indicators of strength 
of association or magnitude of outcome effects, such as effect size 
statistics, will be utilized in NREPP to complement traditional, 
statistical significance (null-hypothesis) testing.
--There will be multiple, outcome-specific ratings of evidence quality 
strength. All programs and/or practices listed on the Registry will be 
considered ``Effective'' for demonstrating specific outcomes having 
varying levels of evidence quality and confidence resulting from 
independent (or applicant) replication(s).
--Evidence rating criteria have been refined and now emphasize 
intervention impacts, evaluation design and fidelity, quality of 
comparison conditions, and replications.

    The section below is an overview of the NREPP process for obtaining 
expert reviewers' ratings of the evidence quality for outcome-specific 
program and practice interventions. The process includes an internal 
screening and triage process conducted by qualified

[[Page 50384]]

NREPP contractor staff serving as review coordinators, as well as an 
independent, external scientific review process conducted by qualified 
and trained external scientists--working independently of one another--
to assess the evidence quality of candidate interventions.

I. Submitted Materials

    Applicants will submit a variety of documents that allow a panel of 
expert reviewers to rate objectively the evidence for an intervention's 
impact on substance abuse or mental health outcomes. Materials 
submitted may include:
     Descriptive program summaries, including the specific 
outcomes targeted;
     Research reports and published and unpublished articles to 
provide scientific evidence (experimental or quasi-experimental 
studies) about the effectiveness of the intervention for improving 
specific outcomes;
     Documents to verify that participants were assured privacy 
and freedom from negative consequences regardless of participating 
(used to assess participant response bias, not human subjects 
protections per se); and
     Documents to provide evidence that outcomes and analytic 
approaches were developed through a theory-driven or hypothesis-based 
approach.
    These materials will provide SAMHSA and potential reviewers with 
objective evidence to support conclusions about the validity and impact 
of the program or practice. Reviewers must be assured that program 
investigators did not capitalize on chance findings or excessive 
postintervention data analyses to find effects that were not components 
of the intervention design or theory.

II. Internal Review and Triage

    Upon receipt, each set of materials will be assigned to an NREPP 
review coordinator (contractor staff), who will inventory and document 
the contents of the submission. The review coordinator will contact the 
applicant by phone and/or e-mail confirming receipt and notifying the 
applicant if additional application materials are required.
    When all materials for a program have been received by the review 
coordinator, an internal review is conducted to eliminate those 
programs or practices that do not meet NREPP minimum evidence-based 
standards. These minimum standards are (1) An intervention that is 
consistent with SAMHSA's matrix of program priority areas; (2) one or 
more evaluations of the intervention using an experimental or quasi-
experimental design; and (3) statistically significant intervention 
outcome(s) related to either the prevention or treatment of mental or 
substance use disorders. Non-reviewed programs will receive written 
notification of this decision, including problematic or missing 
components of their application, and may be considered for re-review at 
a later date.
    SAMHSA will maintain oversight over the entire NREPP application 
and selection process. Moreover, SAMHSA's Administrator and Center 
Directors may establish specific program and practice areas for 
priority review.

III. Ratings by Reviewers

    For all NREPP applications determined appropriate for review, three 
(3) independent, external reviewers will evaluate and rate the 
intervention. Reviewers are doctoral-level researchers and 
practitioners who have been selected based on their training and 
expertise in the fields of mental health promotion or treatment and/or 
substance abuse prevention or treatment. Moreover, reviewers will be 
thoroughly trained in the NREPP review process, and will be monitored 
and provided feedback periodically on their performance. Of note, 
interventions targeting individuals or populations with co-occurring 
mental health and substance use disorders (or other cross-domain 
initiatives) will be assigned to reviewers across these categorical 
domains.
    To maintain the objectivity and autonomy of the peer review 
process, SAMHSA will not disclose to the public or applicants the 
identities of individual reviewers assigned to specific reviews. On a 
periodic basis, SAMHSA may post listings of reviewer panels within 
specific SAMHSA domains as an informational resource to the public.
    Reviewers will be selected based on their qualifications and 
expertise related to specific interventions and/or SAMHSA priority 
areas. In addition to reviewers identified by SAMHSA, NREPP will 
consider third-party and self-nominations to become part of the 
reviewer pool. All reviewers will provide written assurance, to be 
maintained on file with the NREPP contractor, that they do not have a 
current or previous conflict of interest (e.g., financial or business 
interest) that might impact a fair and impartial review of specific 
programs or practices applying for NREPP review.
    Reviewers provide independent assessments of the evidence quality 
and provide numerical summary scores across the 16 outcome-specific 
evidence quality criteria. Each criterion is scored on a 0 to 4 scale. 
The 16 evidence quality criteria are presented below.

Individual-Level Outcome Evidence Rating Criteria

1. Theory-Driven Measure Selection
    Outcome measures for a study should be selected before data are 
collected and should be based on a priori theories of hypotheses.

0 = The applicant selected the measure after data collection for the 
apparent purpose of obtaining more favorable results than would be 
expected from using the measures originally planned, OR there is no 
documentation of selection prior to data analysis.
4 = Documentation shows that the applicant selected the measure prior 
to study implementation, OR the measure was selected after study 
inception, but before data analysis, and is supported by a peer review 
panel or literature related to study theories or hypotheses.
2. Reliability
    Outcome measures should have acceptable reliability to be 
interpretable. ``Acceptable'' here means reliability at a level that is 
conventionally accepted by experts in the field.

0 = No evidence of measure realibility.
1 = Reliability coefficients indicate that some but not all relevant 
types of reliability (e.g., test-retest, inter-rater, inter-item) are 
acceptable.
3 = All relevant types of realibility have been documented to be at 
acceptable levels in studies by the applicant.
4 = All relevant types of reliability have been documented to be 
acceptable levels in studies by independent investigators.

3. Validity
    Outcome measures should have acceptable validity to be 
interpretable. ``Acceptable'' here means validity at a level that is 
conventionally accepted by experts in the field.

0 = No evidence of measure validity, or some evidence that the measure 
is not valid.
1 = Measure has face validity.
3 = Studies by applicant show that measure has one or more acceptable 
forms of criterion-related validity that are correlated with 
appropriate, validated measures or objective criteria; OR, as an 
objective measure of response, there are procedural checks to confirm 
data validity, but they have not been adequately documented.
4 = Studies by independent investigators show that measure has

[[Page 50385]]

one or more acceptable forms of criterion-related validity that are 
correlated with appropriate, validated measures or objective criteria; 
OR, as an objective measure of response, there are adequately 
documented procedural checks that confirm data validity.
4. Intervention Fidelity
    The ``experimental'' intervention implemented in a study should 
have fidelity to the intervention proposed by the applicant. 
Instruments that have tested acceptable psychometric properties (e.g., 
interrator reliability, validity as shown by positive association with 
outcomes) provides the highest level of evidence.

0 = There is evidence the intervention implemented was substantially 
different from the one proposed.
1 = There is only narrative evidence that the applicant or provider 
believes the intervention was implemented with acceptable fidelity.
2 = There is evidence of acceptable fidelity in the form of judgment(s) 
by experts, based on limited on-site evaluation and data collection.
3 = There is evidence of acceptable fidelity, based on the systematic 
collection of data on factors such as dosage, time spent in training, 
adherence to guidelines or a manual, or a fidelity measure with 
unspecified or unknown psychometric properties.
4 = There is evidence of acceptable fidelity from a tested fidelity 
instrument shown to have reliability and validity.
5. Comparison Fidelity
    A study's comparison condition should be implemented with fidelity 
to the comparison condition proposed by the applicant. Instruments for 
measuring fidelity that have tested acceptable psychometric properties 
(e.g., interrater reliability, validity as shown by predicted 
association with outcomes) provide the highest level of evidence.

0 = There is evidence that the comparison condition implemented was 
substantially different from one proposed.
1 = There is only narrative evidence that the applicant or provider 
believes the comparison condition was implemented with fidelity.
2 = Researchers report observational or administrative data that the 
comparison condition was implemented with fidelity.
3 = Documentation confirms that comparison group participants did not 
receive interventions that were very similar or identical to 
intervention participants, AND there is documentation of degree of 
participation in any comparison conditions such as lectures or 
treatment.
4 = There is evidence from a tested instrument suggesting that the 
comparison condition was implemented with fidelity.
6. Nature of Comparison Condition
    The quality of evidence for an intervention depends in part on the 
nature of the comparison condition(s), including assessments of their 
active components and overall effectiveness. Interventions have the 
potential to cause more harm than good; therefore, an active comparison 
intervention should be shown to be better than no treatment.

0 = There was no comparison condition.
1 = The comparison condition is an active intervention that has not 
been proven to better than no treatment.
2 = The comparison condition is no service or wait-list, or an active 
intervention shown to be as effective as or better than no treatment.
3 = The comparison condition is an attention control.
4 = The comparison condition was shown to be as safe or safer and more 
effective than an attention control.
7. Assurances to Participants
    Study participants should always be assured that their responses 
will be kept confidential and not affect their care or services. When 
these procedures are in place, participants are more likely to disclose 
valid data.

0 = There was no effort to encourage and reassure subjects about 
privacy and that consent or participation would have no effect on 
services.
1 = Data collector was the service provider, AND there were documented 
assurances to participants about privacy and that consent or 
participation would have no effect on care or services.
2 = Data collector was not the service provider. There were 
indications, but no documentation, that participants were assured about 
their privacy and that consent or participation would have no effect on 
care or services.
4 = Data collector was not the service provider, AND there were 
documented assurances to participants about privacy and that consent or 
participation would have no effect on care or services; OR, data were 
not collected directly from participants.
8. Participant Expectations
    Participants can be biased by how an intervention is introduced to 
them and by an awareness of their study condition. Information used to 
recruit and inform study participants should be carefully crafted to 
equalize expectations. Masking treatment conditions during 
implementation of the study provides the strongest control for 
participant expectancies.

0 = Investigators did not make adequate attempts to mask study 
conditions or equalize expectations among participants in the 
experimental and comparison conditions, or differential participant 
expectations were measured and found to be too great to control for 
statistically.
2 = Investigators attempted to mask study conditions or equalize 
expectations among participants in the experimental and comparison 
conditions. Some participants appeared likely to have known their study 
condition assignment (experimental or comparison).
3 = Investigators attempted to mask study conditions or equalize 
expectations among participants in the experimental and comparison 
conditions. Some participants appeared likely to have known their study 
condition assignment (experimental or comparison), but these 
differential participant expectations were measured and appropriately 
controlled for statistically.
4 = Investigators adequately masked study conditions. Participants did 
not appear likely to have known their study condition assignment.
9. Standardized Data Collection
    All outcome data should be collected in a standardized manner. Data 
collectors trained and monitored for adherence to standardized 
protocols provide the highest quality evidence of standardized data 
collection.

0 = Applicant did not use standardized data collection protocols.
2 = Data was collected using standardized protocol and trained data 
collectors.
3 = Data was collected using standardized protocol and trained data 
collectors, with evidence of good initial adherence by data collectors 
to the standardized protocol.
4 = Data was collected using standardized protocol and trained data 
collectors, with evidence of good initial adherence to data collectors 
to the standardized protocol and evidence of data collector retraining

[[Page 50386]]

when necessary to control for rater ``drift.''
10. Data Collector Bias
    Data collector bias is most strongly controlled when data 
collectors are not aware of the conditions to which study participants 
have been assigned. When data collectors are aware of specific study 
conditions, their expectations should be controlled for through 
training and/or statistical methods.

0 = Data collectors were not masked to participants' conditions, and 
nothing was done to control for possible bias, OR collector bias was 
measured and found to be too great to control for statistically.
2 = Data collectors were not masked to participants' conditions, but 
data collectors received training to reduce possible bias.
3 = Data collectors were not masked to participants' conditions; 
possible bias was appropriately controlled for statistically.
4 = Data collectors were masked to participants' conditions.
11. Selection Bias
    Concealed random assignment of participants provides the strongest 
evidence of control for selection bias. When participants are not 
randomly assigned, covariates and confounding variables should be 
controlled as indicated by theory and research.

0 = There was no comparison condition, OR participants or providers 
selected conditions.
3 = Participants were not assigned randomly, but researchers controlled 
for theoretically relevant confounding variables, OR participants were 
assigned with non-concealed randomization.
4 = Selection bias was controlled with concealed random assignment.
12. Attrition
    Study results can be biased by participant attrition. Statistical 
methods as supported by theory and research can be employed to control 
for attrition that would bias results, but studies with no attrition 
needing adjustment provide the strongest evidence that results are not 
biased.

0 = Attrition was taken into account inadequately, OR there was too 
much attrition to control for bias.
1 = No significant differences were found between participants lost to 
attrition and remaining participants.
2 = Attrition was taken into account by simpler methods that crudely 
estimate data for missing observations.
3 = Attrition was taken into account by more sophisticated methods that 
model missing data, observations, or participants.
4 = There was no attrition, OR there was no attrition needing 
adjustment.
13. Missing Data
    Study results can be biased by missing data. Statistical methods as 
supported by theory and research can be employed to control for missing 
data that would bias results, but studies with no missing data needing 
adjustment provide the strongest evidence.

0 = Missing data were an issue and were taken into account 
inadequately, OR levels of missing data were too high to control for 
bias.
1= Missing data were an issue and were taken into account, but high 
quantity makes the control for bias suspect.
2= Missing data were an issue and were taken into account by simpler 
methods (mean replacement, last point carried forward) that 
simplistically estimate missing data; control for missing data is 
plausible.
3= Missing data were an issue and were taken into account by more 
sophisticated methods that model missing data; control for missing data 
very plausible.
4= Missing data were not an issue.
14. Analysis Meets Data Assumptions
    The appropriateness of statistical analyses is a function of the 
properties of the data being analyzed and the degree to which meet 
statistical assumptions.

0= Analyses were clearly inappropriate to the data collected; severe 
violation(s) of assumptions make analysis uninterpretable.
1= Some data were analyzed appropriately, but for other analyses 
important violation(s) of assumptions cast doubt on interpretation.
2= There were minor violations of assumptions for most or all analyses, 
making interpretation of results arguable.
3= There were minor violations of assumptions for only a few analyses; 
results were generally interpretable.
4= There were no violations of assumptions for any analysis.
15. Theory-Driven Selection of Analytic Methods
    Analytic methods should be selected for a study based on a priori 
theories or hypotheses underlying the intervention. Changes to analytic 
methods after initial data analysis (e.g., to ``dredge'' for 
significant results) decrease the confidence that can be placed in the 
findings.

0= Analysis selected appears inconsistent with the intervention theory 
or hypotheses; insufficient rational provided by investigator.
1= Analysis selected appears inconsistent with the intervention theory 
or hypotheses, but applicant provides a potentially viable rationale.
3= Analysis is widely accepted by the field as the most consistent with 
study theory or hypotheses; no documentation showing methods were 
selected prior to data analysis.
4= Analysis is widely accepted by the field as the most consistent with 
study theory or hypotheses; documentation shows that methods were 
selected prior to data analysis.
16. Anomalous Findings
    Findings that contradict the theories and hypotheses underlying an 
intervention suggest the possibility of confounding causal variables 
and limit the validity of study findings.

0 = There were anomalous findings suggesting alternate explanations for 
outcomes reported.
4 = There were no anomalous findings, OR researchers explained 
anomalous findings in a way that preserves the validity of results 
reported.

Based upon the independent reviewer assessments, review coordinators 
will compute average evidence quality ratings for specific outcome 
measures (based on the 16 evidence quality criteria), and then ask 
reviewers to determine the overall intervention outcome evidence 
ratings according to two components: quality of evidence and 
intervention replications. Average evidence quality ratings scores 
below 2.0 will be considered ``insufficient current evidence'' for the 
effectiveness of a given outcome, and will not be included in the 
Registry. Evidence quality rating scores of 2.0 to 2.5 will be 
considered ``emerging evidence'' for effectiveness, and scores of 2.5 
and higher (4.0 is the maximum) will be considered ``strong evidence.''
    Specific rating category labels for effective outcomes remain to be 
finalized, but might include categories such as: (1) Strong evidence 
with independent replication(s); (2) Strong evidence with developer 
replication(s); (3) Strong evidence without replication; (4) Emerging 
evidence with independent replication(s); (5) Emerging evidence with 
developer replication(s); and (6) Emerging evidence without 
replication.

IV. Applicant Notification

    Applicants will be notified in writing of their final NREPP rating 
category(s) by the SAMHSA Administrator or his/

[[Page 50387]]

her designee within 2 weeks following the completion of the review. 
This notification will include the summary comments of reviewers as 
well as the consensus ratings on each review criteria. Where relevant, 
the notification letter will provide applicants with the effective date 
of the program status designation.
    Applicants will have the opportunity to appeal any review decision 
by submitting a written request to the NREPP contractor within 30 days 
of notification. Appeals will be resolved through the assignment of two 
(2) additional reviewers to conduct focused reviews of the evidence 
quality for specific, disputed outcome ratings. Reviews conducted as 
part of a formal appeal process will be independent and reviewers will 
be unaware of previous ratings and decisions. The numeric evidence 
ratings will be averaged across the five (i.e., 3 original; 2 appeal) 
reviews for a final determination of intervention outcome rating(s).

V. Utility Descriptors

    The NREPP utility descriptors will provide information to program 
purchasers and planners, service providers, consumers and the general 
public about the transferability of intervention technologies to 
different (including non-research-based) settings. These descriptors 
complement NREPP's scientific evidence-based program and practice 
ratings with information pertaining to the following dimensions:
    1. Implementation. What kinds of materials are available to support 
implementation and what audiences are they appropriate for? What kinds 
of training and training resources are available to support 
implementation?
    2. Quality Monitoring. What tools, procedures, and documentation 
are available to support quality monitoring and quality improvement as 
the program or practice is implemented?
    3. Unintended or Adverse Events. What procedures, systems and data 
have been identified to indicate whether the intervention has ever 
resulted in unintended or adverse events?
    4. Population Coverage. Were the study samples recruited 
representative of the persons identified to receive the intervention in 
the theory/framework?
    5. Cultural Relevance and Cultural Competence. Were the outcomes 
demonstrated to be effective and applicable to specific demographic and 
culturally defined groups? If so, are training and other implementation 
materials available to promote culturally competent implementation of 
the intervention?
    6. Staffing. What is aggregate level of staffing (e.g., FTEs) 
required to implement the intervention effectively? What are the 
individual skills, expertise, and training required of staff to deliver 
the intervention?
    7. Cost. What are the estimated start-up and annual costs per 
person served and unit of service for the intervention at full 
operation?
    Interventions with outcomes achieving any one of the effective 
statuses will be asked to provide descriptive information about the 
intervention's readiness for implementation, appropriateness for 
different populations, freedom from unintended or adverse effects, and 
staffing and cost requirements. These utility descriptors will be 
featured on the NREPP Web site to help assure a proper match between 
specific prevention and treatment interventions and the settings and 
populations to which they are most appropriate.
    In light of SAMHSA's commitment to consumer and family involvement, 
the agency is seeking ways to ensure that these groups provide input 
into the assessment of interventions that achieve NREPP status. SAMHSA 
seeks suggestions regarding the most useful and efficient way to 
conduct this process (see Question 7 below).

Review Process for Determining Population-, Policy-, and System-Level 
Outcome Ratings for Interventions

    The NREPP Evidence Rating Criteria for Population-, Policy-, and 
System-Level Outcomes are proposed as the basis for reviewer ratings of 
outcomes generated by community prevention coalitions and other 
environmental interventions to promote resiliency and recovery at the 
community level. SAMHSA's rationale for use of these separate criteria 
comes through a recognition that some interventions may be designed to 
affect a community over time, and that the prevailing scientific 
standards for assessing the effectiveness of these interventions may 
indeed be different than those for interventions seeking to change 
individual-level outcomes.
    An outcome of a prevention or treatment intervention qualifies for 
review under these 12 criteria only when it can be included in one of 
the following three categories:
    1. Population-Level Outcome--measures the effect of an intervention 
of an existing, predefined population. Examples of such existing, 
predefined populations include ``all youth residing in a 
neighborhood,'' ``all female employees of a manufacturing plant,'' or 
``all Native Americans receiving social services from a tribal 
government.'' ``All patients receiving a specific treatment,'' in 
contrast, cannot be defined as an existing, predefined population 
because that population would have come into existence as a direct 
response to the intervention.
    2. Policy-Level Outcome--measures the effect of an intervention on 
enactment, maintenance, or enforcement of policies that are assumed to 
have a positive aggregate impact on resiliency or recovery. Examples of 
such outcomes include ``the rate of passage of legislation restricting 
access to alcoholic beverages'' or ``the percentage of arrests for 
illicit drug manufacturing that result in convictions.''
    3. System-Level Outcome--measures the effect of an intervention on 
prevention and treatment capacity, efficiency, or effectiveness in an 
existing system or community. Examples of such outcomes include 
``increased capacity of a State government to quantify alcohol- or 
drug-related problems'' or ``increased effectiveness of a community 
treatment system to respond to the comprehensive needs of individuals 
with Axis I mental health diagnoses.''
    To support the transparency of the review process, SAMHSA wants 
stakeholders to understand clearly the NREPP procedures and decision-
making processes. All community coalition interventions included in 
NREPP will have demonstrated evidence of effectiveness at the 
population, policy, or system level. The ratings will indicate the 
strength of the supporting evidence, and may be as follows:
    (1) Strong evidence with replication;
    (2) Strong evidence without replication;
    (3) Emerging evidence with replication; and
    (4) Emerging evidence without replication.
    All NREPP evidence ratings are defined at the level of specific 
outcomes. The 12 evidence rating criteria used for population-, policy- 
and system-level outcomes, summarized as an average Evidence Quality 
Score (EQS) for each outcome, allow independent expert reviewers to 
score along dimensions of outcome measurement, intervention fidelity, 
comparison conditions, participant and data collector biases, design 
and analysis, and anomalous findings. Each of the 12 criteria is 
assessed by independent reviewers on a 0 to 2 scale, in which a ``1'' 
indicates that methodological rigor may have been acceptable and a 
``2'' indicates that adequate methodological rigor was achieved for 
this type of outcome.

[[Page 50388]]

Preliminary discussions of classifications have suggested that ``Strong 
evidence'' be defined as an average EQS of 1.75 or above (out of a 
possible 2.0), and that ``Emerging evidence'' be defined as an average 
EQS between 1.50 and 1.74 (out of a possible 2.0).
    The 12 criteria applied to each population-, policy-, or system-
level outcome measures are:
    1. Logic-Driven Selection of Measures
    2. Reliability
    3. Validity
    4. Intervention Fidelity
    5. Nature of Comparison Condition
    6. Standardized Data Collection
    7. Data Collector Bias
    8. Population Studied
    9. Missing Data
    10. Analysis Meets Data Assumptions
    11. Theory-Driven Selection of Analytic Methods
    12. Anomalous Findings

Outcome Measurement Criteria

1. Logic-Driven Selection of Measures
    Outcome measures should be based on a theory or logic model that 
associates them with the intervention.

0 = The applicant appears to have selected outcome measures for the 
purpose of identifying favorable results rather than from a logic-based 
rationale.
1 = There is no explicit description of a guiding logic model or theory 
for measures, although a rationale for the inclusion of most measures 
can be inferred.
2 = Measures are supported by a theory or logic model that associates 
the intervention with the outcome.
2. Reliability
    Outcome measures should have acceptable reliability to be 
interpretable. ``Acceptable'' here means reliability at a level that is 
conventionally accepted by experts in the field.

0 = No evidence of reliability of measures is presented.
1 = Relevant reliability measures are in the marginal range.
2 = Relevant reliability measures are in clearly acceptable ranges.
3. Validity
    Outcome measures should have acceptable validity to be 
interpretable.

0 = No evidence of validity of measures is presented or evidence that 
is presented suggests measures are not valid.
1 = Measures has face validity.
2 = Relevant validity has been documented to be at acceptable levels in 
independent studies.
4. Intervention Fidelity
    The intervention should be well defined and its implementation 
should be described in sufficient detail to assess whether 
implementation affected outcomes.

0 = The intervention and/or its implementation are not described in 
sufficient detail to verify that the intervention was implemented as 
intended.
1 = The intervention and its implementation are described in adequate 
detail, including justification for significant variation during 
implementation.
2 = The intervention and its implementation are described in adequate 
detail, reflecting variation during implementation with little or no 
plausible impact on outcomes.
5. Nature of Comparison Condition
    The quality of evidence for an intervention depends in part on the 
nature of the comparison condition(s).

0 = Research design either lacks a comparison condition, or employs a 
before/after comparison.
1 = Comparison condition was no service or wait-list (including 
baseline comparison for a multipoint time series), or an active 
intervention that has not been shown to be safer or more effective than 
no service.
2 = Comparison condition was an active intervention shown to be as safe 
as, or safer and more effective than, no service.
6. Standardized Data Collection
    All outcome data should be collected in a standardized manner. Data 
collectors trained and monitored for adherence to standardized 
protocols provide the highest quality evidence of standardized data 
collection.

0 = Data collection or archival sources used by the evaluation to 
assess outcome did not use standardized data collection protocol(s).
1 = All outcome data were collected using standardized protocol(s).
2 = All outcome data were collected using standardized protocol(s) and 
trained data collectors.
7. Data Collector Bias
    Data collector bias is most strongly controlled when data 
collectors are not aware of the interventions to which populations have 
been exposed. When data collectors are aware of specific interventions, 
their expectations should be controlled for through training and/or 
statistical analysis methods on resultant data.

0 = Data collectors were not masked to the population's condition, and 
nothing was done to control for possible bias, OR collector bias was 
identified and not controlled for statistically.
1 = Data collectors were not masked to the population's condition; 
possible bias was appropriately controlled for statistically or through 
training.
2 = Data collectors were masked to the population's condition, or only 
archival data was employed.
8. Population Studied
0 = A single group pre/posttest design was applied without a comparison 
group, OR the alleged comparison group is significantly different from 
the population receiving the intervention.
1 = Population(s) were studied using time trend analysis, multiple 
baseline design, or a regression-discontinuity design that uses within-
group differences as a substitute for comparison groups.
2 = Population matching or similar techniques were used to compare 
outcomes of population that received the intervention with the outcomes 
of a valid comparison group.
9. Missing Data
    Study results can be biased by missing data. Statistical methods as 
supported by theory and research can be employed to control for missing 
data that would bias results, but studies with no missing data needing 
adjustment provide the strongest evidence.

0 = Missing data were an issue and were taken into account 
inadequately, OR levels of missing data were too high to control for 
bias.
1 = Missing data were an issue and were taken into account, but high 
quality makes the control for bias suspect.
2 = Missing data were not an issue or were taken into account by 
methods that estimate missing data.
10. Analysis Meets Data Assumptions
    The appropriateness of statistical analysis is a function of the 
properties of the data being analyzed and the degree to which data meet 
statistical assumptions.

0 = Analyses were clearly inappropriate to the data collected; severe 
violation(s) of assumptions make analysis uninterpretable.
1 = There were minor violations of assumptions, making interpretation 
of results arguable.
2 = There were no or only very minor violations of assumptions; result 
were generally interpretable.

[[Page 50389]]

11. Theory-Driven Selection of Analytic Methods
    In addition to the properties of the data, analytic methods should 
be based on a logic model or theory underlying the intervention. 
Changes to analytic methods after initial data analysis (e.g., to 
dredge for significant results) decrease the confidence that can be 
placed in the findings.

0 = Analysis selected appears inconsistent with the intervention theory 
or hypotheses; insufficient rationale was provided by the investigator.
1 = Analysis selected appears inconsistent with the intervention logic 
model or hypotheses, but the investigator provides a potentially viable 
rationale.
2 = Analysis is widely accepted by the field as consistent with the 
intervention logic model or hypotheses.
12. Anomalous Findings
    Findings that contradict the theories and hypotheses underlying an 
intervention suggest the possibility of confounding causal variables 
and limit the validity of study findings.

0 = There were anomalous findings suggesting alternate explanations for 
outcomes reported that were not acknowledged by the applicant.
1 = There were a few anomalous findings, but additional analysis or 
previous literature cited by the applicant provide a reasonable 
explanation.
2 = There were no anomalous findings, OR researchers explained 
anomalous findings in a way that preserves the validity of results 
reported.

Re-Review of Existing NREPP Programs

    As noted above, SAMHSA believes it is important to ensure that both 
current and future NREPP interventions meet consistent scientific 
standards so that the public and other interested stakeholders can be 
confident in the effectiveness of these interventions. With this in 
mind, SAMHSA is committed to expeditiously re-reviewing all existing 
NREPP programs under the new process. As part of this effort, SAMHSA 
plans to provide--directly or indirectly--sufficient resources to each 
existing NREPP program to cover costs associated with a re-review. In 
addition, programs already received by NREPP and pending review will be 
reviewed under the new process. If additional support is needed by 
these pending programs regarding the new review processes, these 
resources will also be provided.
    In order to accomplish these re-reviews efficiently and 
expeditiously, NREPP review coordinators will work with each program to 
obtain any additional documentation that might be needed for re-review. 
These review coordinators will then conduct a re-review of each program 
against the new review criteria. Programs with favorable re-reviews 
will be included in the new NREPP system when it is launched in 2006. 
Programs not receiving favorable re-reviews will have the opportunity 
to appeal the re-review decision, and will be eligible for re-review by 
independent, external reviewers. However, the schedule for re-reviews 
of appealed programs will be subject to SAMHSA Administrator and SAMHSA 
Center Director review priorities.

New Web Site

    The primary goal of the revised NREPP Web site--https://
www.nationalregistry.samhsa.gov--will be to provide the public with 
contemporary and reliable information about the scientific basis and 
practicality of interventions to prevent and treat mental and substance 
use disorders. All interventions achieving NREPP status will be listed 
on the Web site. Average ratings and evaluate scores from scientific 
peer reviewers, as well as information on the utility and 
transferability of these interventions, will be posted on the site.
    In addition, a searchable outcomes database of evidence-based 
interventions will be a key feature. The Web site will also contain a 
variety of learning and self-assessment tools for prospective and 
current NREPP interventions to continuously improve their scientific 
evidence base. Features of the new Web site will include:
     Evidence rating criteria and utility descriptors
     Detailed review guidelines
     Self-assessment tool to assist interventions in 
determining if they are ready to submit an application to NREPP
     Detailed information on how to apply
     Links to technical assistance resources available to 
potential applicants
     Relevant resources, including publications, presentations, 
links, and other supplemental materials
     Frequently Asked Questions (FAQs)
     Information on how to contact a representative from the 
NREPP team
     Glossary of terms

Support for Innovative Interventions

    SAMHSA recognizes that the long term utility and value of NREPP 
rests, in part, on the ability of SAMHSA and others to support efforts 
to evaluate and document the evidence-base of innovative interventions 
in ways that will maximize their opportunity for entry into NREPP. 
SAMHSA is considering potential options for both the direct and 
indirect provision of such support, and will seek to clarify its 
intentions in this area sometime in Fiscal Year 2006.

Questions To Consider in Making Your Comments

    Responders should feel free to comment on any, or all, questions, 
as well as provide relevant suggestions not included in the specific 
questions. In order to facilitate the compilation and analysis of 
comments, responders are asked to be explicit about the questions to 
which they are responding.
    1. SAMHSA is seeking to establish an objective, transparent, 
efficient, and scientifically defensible process for identifying 
effective, evidence-based interventions to prevent and/or treat mental 
and substance use disorders. Is the proposed NREPP system--including 
the suggested provisions for screening and triage of applications, as 
well as potential appeals by applicants--likely to accomplish these 
goals?
    2. SAMHSA's NREPP priorities are reflected in the agency's matrix 
of program priority areas. How might SAMHSA engage interested 
stakeholders on a periodic basis in helping the agency determine 
intervention priority areas for review by NREPP?
    3. There has been considerable discussion in the scientific 
literature on how to use statistical significance and various measures 
of effect size in assessing the effectiveness of interventions based 
upon both single and multiple studies (Schmidt & Hunter, 1995; 
Rosenthal, 1996; Mason, Schott, Chapman, & Tu, 2000: Rutledge & Loh, 
2004). How should SAMHSA use statistical significance and measures of 
effect size in NREPP? Note that SAMHSA would appreciate receiving 
citations for published materials elaborating upon responders 
suggestions in this area.
    4. SAMHSA's proposal for NREPP would recognize as effective several 
categories of interventions, ranging from those with high-quality 
evidence and more replication to those with lower quality evidence and 
fewer replications. This would allow for the recognition of emerging as 
well as fully evidence-based interventions. Some view this as a 
desirable feature that reflects the continuous nature of evidence; 
provides important options for interventions

[[Page 50390]]

recipients, providers, and funders when no or few fully evidence-based 
interventions are available; and helps promote continued innovation in 
the development of evidence-based interventions. Others have argued 
that several distinct categories will confuse NREPP users. Please 
comment on SAMHSA's proposal in this area.
    5. SAMHSA recognizes the importance of considering the extent to 
which interventions have been tested with diverse populations and in 
diverse settings. Therefore, the agency anticipates incorporating this 
information into the web site descriptions of interventions listed on 
NREPP. This may allow NREPP users to learn if interventions are 
applicable to their specific needs and situations, and may also help to 
identify areas where additional studies are needed to address the 
effectiveness of interventions with diverse populations and in diverse 
locations.
    SAMHSA is aware that more evidence is needed on these topics. 
Please comment on SAMHSA's approach in this area.
    6. To promote consistent, reliable, and transparent standards to 
the public, SAMHSA proposes that all existing programs on NREPP meet 
the prevailing scientific criteria described in this proposal, and that 
this be accomplished through required re-reviews of all programs 
currently on NREPP. SAMHSA has considered an alternative approach that 
would ``grandfather'' all existing NREPP programs under the new system, 
but would provide clear communication that these existing programs have 
not been assessed against the new NREPP scientific standards. Please 
comment on which approach you believe to be in the best interests of 
SAMHSA stakeholders.
    7. What types of guidance, resources, and/or specific technical 
assistance activities are needed to promote greater adoption of NREPP 
interventions, and what direct and indirect methods should SAMHSA 
consider in advancing this goal?
    8. SAMHSA is committed to consumer, family, and other nonscientist 
involvement in the NREPP process. The panels convened by SAMHSA and 
described earlier in this notice suggested that these stakeholders be 
included specifically to address issues of intervention utility and 
practicality. Please comment on how consumer, family, and other 
nonscientist stakeholders could be involved in NREPP.
    9. SAMHSA has identified NREPP as one source of evidence-based 
interventions for selection by potential agency grantees in meeting the 
requirements related to some of SAMHSA's discretionary grants. What 
guidance, if any, should SAMHSA provide related to NREPP as a source of 
evidence-based interventions for use under the agency's substance abuse 
and mental health block grants?
    10. SAMHSA believes that NREPP should serve as an important, but 
not exclusive source, of evidence-based interventions to prevent and/or 
treat mental and substance use disorders. What steps should SAMHSA take 
to promote consideration of other sources (e.g., clinical expertise, 
consumer or recipient values) in stakeholders' decisions regarding the 
selection, delivery and financing of mental health and substance abuse 
prevention and treatment services?
    11. SAMHSA anticipates that once NREPP is in operation, various 
stakeholders will make suggestions for improving the system. To 
consider this input in a respectful, deliberate, and orderly manner, 
SAMHSA anticipates annually reviewing these suggestions. These reviews 
would be conducted by a group of scientist and nonscientist 
stakeholders knowledgeable about evidence in behavioral health and the 
social sciences. Please comment on SAMHSA's proposal in this area.

References

    Mason, CA, Scott, KG, Chapman, DA, Tu, S. A review of some 
individual and community level effect size indices for the study of 
risk factors for child and adolescent development. Ed. Psychol. 
Measurement. 2000; 60(3): 385-410.
    Rosenthal, JA. Qualitative descriptors of strength of 
association and effect size. J. of Soc. Serv. Research. 1996; 21(4): 
37-59.
    Rutledge, T, Loh, C. Effect size and statistical testing in the 
determination of clinical significance in behavioral medicine 
research. Annals of Beh. Med. 2004; 27(2): 138-145.
    Schmidt, F, Hunter, JE. The impact of data-analysis methods on 
cumulative research knowledge: statistical significance testing, 
confidence intervals, and meta-analysis. Eval. Health Prof. 1995 
Dec; 18(4): 408-27.
[FR Doc. 05-17034 Filed 8-25-05; 8:45 am]
BILLING CODE 4160-01-M