Changes to the National Registry of Evidence-Based Programs and Practices (NREPP), 13132-13155 [06-2313]

Agencies

• DEPARTMENT OF HEALTH AND HUMAN SERVICES
• Substance Abuse and Mental Health Service Administration
• Substance Abuse and Mental Health Services Administration

[Federal Register Volume 71, Number 49 (Tuesday, March 14, 2006)]
[Notices]
[Pages 13132-13155]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 06-2313]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Substance Abuse and Mental Health Service Administration


Changes to the National Registry of Evidence-Based Programs and 
Practices (NREPP)

AGENCY: Substance Abuse and Mental Health Services Administration, HHS.

ACTION:  Notice.

-----------------------------------------------------------------------

SUMMARY: The Substance Abuse and Mental Health Services Administration 
(SAMHSA) is committed to preventing the onset and reducing the 
progression of mental illness, substance abuse, and substance-related 
problems among all individuals, including youth. As part of this 
effort, SAMHSA has expanded and refined the agency's National Registry 
of Evidence-based Programs and Practices (NREPP) based on a systematic 
analysis and consideration of public comments received in response to a 
previous Federal Register notice (70 FR 50381, Aug. 26, 2005).
    This Federal Register notice summarizes SAMHSA's redesign of NREPP 
as a decision support tool for promoting a greater adoption of 
evidence-based interventions within typical community-based settings, 
and provides an opportunity for interested parties to become familiar 
with the new system.

FOR FURTHER INFORMATION CONTACT: Kevin D. Hennessy, Ph.D., Science to 
Service Coordinator/SAMHSA, 1 Choke Cherry Road, Room 8-1017, 
Rockville, MD 20857, (240) 276-2234.

Charles G. Curie,
Administrator, SAMHSA.

Advancing Evidence-Based Practice Through Improved Decision Support 
Tools: Reconceptualizing NREPP

Introduction

    The Substance Abuse and Mental Health Services Administration 
(SAMHSA) strives to provide communities with effective, high-quality, 
and cost-efficient prevention and treatment services for mental and 
substance use disorders. To meet this goal, SAMHSA recognizes the needs 
of a wide range of decisionmakers at the local, state, and national 
levels to have readily available and timely information about 
scientifically established interventions to prevent and/or treat these 
disorders.
    SAMHSA, through its Science to Service Initiative, actively seeks 
to promote Federal collaboration (e.g., with the National Institutes of 
Health [NIH]) in translating research into practice. The ideal outcome 
of this Initiative is that individuals at risk for or directly 
experiencing mental and substance abuse use disorders will be more 
likely to receive appropriate preventive or treatment services, and 
that these services will be the most effective and the highest quality 
that the field has to offer.
    This report provides a summary of activities conducted during the 
past year to critically evaluate SAMHSA's recent activities and future 
plans for the National Registry of Evidence-based Programs and 
Practices (NREPP). It outlines the major themes that emerged from a 
formal public comment process and links this feedback to new review 
procedures and Web-based decision support tools that will enhance 
access to evidence-based knowledge for multiple audiences.
    The report is presented in four sections:
     Section I briefly states the background of NREPP and 
SAMHSA's recent request for public comments.
     Section II discusses the analysis of comments that was 
conducted and presents the key recommendations for NREPP based on this 
analysis.
     Section III describes the new approach that SAMHSA is 
advancing for NREPP.
     Section IV presents the specific dimensions of the NREPP 
system in its new framework as a decision support tool.

[[Page 13133]]

     Section V describes future activities at SAMHSA to support 
NREPP.

I. Background: The National Registry of Evidence-Based Programs and 
Practices

    The National Registry of Evidence-based Programs and Practices was 
designed to represent a key component of the Science to Service 
Initiative. It was intended to serve as a voluntary rating and 
classification system to identify programs and practices with a strong 
scientific evidence base. An important reason for developing NREPP was 
to reduce the significant time lag between the generation of scientific 
knowledge and its application within communities.\1\ Quality treatment 
and prevention services depend on service providers' ability to access 
evidence-based scientific knowledge, standardized protocols, practice 
guidelines, and other practical resources.
---------------------------------------------------------------------------

    \1\ As cited by the Institute of Medicine (2001), studies have 
suggested it takes an average of 17 years for research evidence to 
diffuse to clinical practice. Source: Balas, E.A., & Boren, S.A. 
(2000). Managing clinical knowledge for health care improvement. In: 
J. Bemmel & A.T. McCray (Eds.), Yearbook of medical informatics 
2000: Patient-centered systems. Stuttgart, Germany: Schattauer.
---------------------------------------------------------------------------

    The precursor of NREPP, the National Registry of Effective 
Prevention Programs, was developed by SAMHSA's Center for Substance 
Abuse Prevention (CSAP) as a way to help professionals in the field 
become better consumers of substance abuse prevention programs. Through 
CSAP's Model Program Initiative, over 1,100 programs were reviewed, and 
more than 150 were designated as Model, Effective, or Promising 
Programs.
    Over the past 2 years, SAMHSA convened a number of scientific 
panels to explore the expansion of the NREPP review system to include 
interventions in all domains of mental health and substance abuse 
prevention and treatment. In addition, SAMHSA committed itself to three 
guiding principles--transparency, timeliness, and accuracy of 
information--in the development of an evidence-based registry of 
programs and practices.
    During this process it was determined that, to provide the most 
transparent and accurate information to the public, evidence should be 
assessed at the level of outcomes targeted by an intervention, not at 
the more global level of interventions or programs. Based on this 
decision, SAMHSA's current NREPP contractor conducted a series of pilot 
studies to explore the validity and feasibility of applying an outcome-
specific, 16-criteria evidence rating system to an expanded array of 
programs and practices. Through extensive dialogues with the prevention 
community, SAMHSA also explored ways to provide evidence-based reviews 
of population- and community-level interventions within NREPP.
    In an effort to augment the information gained through these 
activities, SAMHSA solicited formal public comments through a notice 
posted in the Federal Register on August 26, 2005. The notice asked for 
responses to the agency's plans for NREPP, including (1) revisions to 
the scientific review process and review criteria; (2) the conveying of 
practical implementation information about NREPP programs and practices 
to those who might purchase, provide, or receive these interventions; 
and (3) the types of additional agency activities that may be needed to 
promote wider adoption of interventions on NREPP, as well as support 
innovative interventions seeking NREPP status. A brief summary of the 
public comments and key public recommendations is presented in Section 
II. The complete analysis of the public responses is included in the 
Appendix to this report.

II. Public Responses to the Federal Register Notice

    Senior staff at SAMHSA engaged in a comprehensive review of 
comments received in response to the Federal Register notice. 
Particular attention was directed to comments from prominent state and 
Federal stakeholders, including providers and policymakers, who stand 
to be the most affected by whatever system is ultimately implemented. 
Efforts were taken to balance SAMHSA's responsiveness to public 
feedback with the need to adhere to rigorous standards of scientific 
accuracy and to develop a system that will be fair and equitable to 
multiple stakeholder groups.

Recommendations for NREPP

    In the more than 100 comments received as part of the public 
comment process, a number of recurring themes and recommendations were 
identified. While all specific and general recommendations for 
modification of the NREPP review process were carefully considered by 
SAMHSA, the following are those that were considered most essential to 
the development of an accurate, efficient, and equitable system that 
can meet the needs of multiple stakeholders:
     Limit the system to interventions that have demonstrated 
behavioral change outcomes. it is inherently appealing to the funders, 
providers, and consumers of prevention and treatment services to know 
that an intervention has a measurable effect on the actual behavior of 
participants. As researchers at the University of Washington 
recommended, ``the system should be reserved for policies, programs, 
and system-level changes that have produced changes in actual drug use 
or mental health outcomes.''
     Rereview all existing programs. There was near consensus 
among the respondents to the notice that existing programs with Model, 
Effective, and Promising designations from the old reviews should be 
rereviewed under the new system. The Committee for Children pointed out 
that ``a `grandfather' system may give the impression to users, right 
or wrong, that these interventions aren't as good as those that have 
undergone the new review process.'' One individual suggested that 
programs and practices needed to be rated ``according to a consistent 
set of criteria'' so that ``the adoption of an intervention by a 
provider can be made with confidence.''
     Train and utilize panels of reviewers with specific 
expertise related to the intervention(s) under review. Respondents to 
the notice noted that it would be important for the NREPP review 
process to utilize external reviewers with relevant scientific and 
practical expertise related to the intervention being assessed. In 
addition, the pool of available reviewers should broadly include 
community-level and individual-level prevention as well as treatment 
perspectives. In order to promote transparency of the review process, 
the reviewer training protocols should be available for review by the 
public (e.g., posted on the NREPP Web site).
     Provide more comprehensive and balanced descriptions of 
evidence-based practices, by emphasizing the important dimension of 
readiness for dissemination. The American Psychological Association 
(APA) Committee on Evidence-Based Practice recommended greater emphasis 
on the utility descriptors (i.e., those items describing materials and 
resources to support implementation), stating, ``these are key outcomes 
for implementation and they are not adequately addressed in the 
description of NREPP provided to date. This underscores earlier 
concerns noted about the transition from efficacy to effectiveness.'' 
The APA committee noted that generalizability of programs listed on 
NREPP will remain an issue until this ``gap between efficacy and 
effectiveness'' is explicitly addressed under a revised review system.
     Avoid limiting flexibility and innovation; implement a 
system that is

[[Page 13134]]

fair and inclusive of programs and practices with limited funding, and 
establish policies that seek to prevent the misuse of information 
contained on NREPP. The National Association for Children of Alcoholics 
voiced this concern: ``It has been intrinsically unfair that only 
grants [referring to NIH-funded efforts] have been able to establish 
`evidence' while many programs appear very effective--often more 
effective in some circumstances than NREPP approved programs, but have 
not had the Federal support or other major grant support to evaluate 
them. The SAMHSA grant programs continue to reinforce the designation 
of NREPP programs in order to qualify for funding, and the states tend 
to strengthen this `stipulation' to local programs,who then drop good 
(non-NREPP) work they have been doing or purchase and manipulate NREPP 
programs that make the grant possible. This is not always in the best 
interest of the client population to be served.''
     Recognize multiple ``streams of evidence'' (e.g., 
researcher, practitioner, and consumer) and the need to provide 
information to a variety of stakeholders in a decision support context. 
A number of comments suggested that NREPP should be more inclusive of 
the practitioner and consumer perspective on what defines evidence. For 
example, one commenter noted: ``The narrowed interpretation of 
evidence-based practice by SAMHSA focuses almost solely on the research 
evidence to the exclusion of clinical expertise and patient values.'' 
Several comments noted that NREPP should be consistent with the 
Institute of Medicine's definition of evidence-based practice, which 
reflects multiple ``streams of evidence'' that include research, 
clinical, and patient perspectives.
     Provide a summary rating system that reflects the 
continuous nature of evidence quality. There was substantial 
disagreement among those responding to the notice concerning whether 
NREPP should include multiple categories of evidence quality. While a 
number of individuals and organizations argued for the use of 
categorical evidence ratings, there were many who suggested that NREPP 
should provide an average, numeric scale rating on specific evidence 
dimensions to better reflect the ``continuous nature of evidence.'' 
This approach would allow the user of the system to determine what 
level of evidence strength is required for their particular application 
of an intervention.
     Recognize the importance of cultural diversity and provide 
complete descriptive information on the populations for which 
interventions have been developed and applied. Most comments reflected 
the knowledge that cultural factors can play an important role in 
determining the effectiveness of interventions. The Oregon Office of 
Mental Health and Addiction Services noted, ``SAMHSA should focus 
considerable effort on identifying and listing practices useful and 
applicable for diverse populations and rural areas. Providers and 
stakeholders from these groups have repeatedly expressed the concern 
they will be left behind if no practices have been identified which fit 
the need of their area. We need to take particular care to ensure that 
their fear is not realized.''
     In addition to estimating the effect size of intervention 
outcomes, NREPP should include additional descriptive information about 
the practical impacts of programs and practices. In general, comments 
suggested that that effect size should not be used as an exclusionary 
criterion in NREPP. It was widely noted that effect size estimates for 
certain types of interventions (e.g., community-level or population-
based) will tend to be of smaller magnitude, and that ``professionals 
in the field have not reached consensus on how to use effect size.'' 
Researchers at the University of Washington suggested the inclusion of 
information about the reach of an intervention, when available, as 
complementary information to effect sizes. Several comments also 
suggested that effect size is often confused with the clinical 
significance of an intervention and its impact on participants.
     Acknowledge the need to develop additional mechanisms of 
Federal support for technical assistance and the development of a 
scientific evidence base within local prevention and treatment 
communities. Nearly one third of the comments directly addressed the 
need for SAMHSA to identify and/or provide additional technical 
assistance resources to communities to help them adapt and implement 
evidence-based practices. The Oregon Office of Mental Health and 
Addiction Services wrote, ``The adoption of new practices by any entity 
is necessarily a complex and long-term process. Many providers will 
need technical support if adoption and implementation is to be 
accomplished effectively. Current resources are not adequate to meet 
this challenge.''
    In order to align NREPP with the important recommendations 
solicited through the public comment process, SAMHSA also recognized 
the importance of the following goals:
     Provide a user-friendly, searchable array of descriptive 
summary information as well as reviewer ratings of evidence quality.
     Provide an efficient and cost-effective system for the 
assessment and review of prospective programs and practices.
    Section III, Streamlined Review Procedures, provides a complete 
description of the modified and streamlined review process that SAMHSA 
will adopt in conducting evidence-based evaluations of mental health 
and substance abuse interventions.

III. Streamlined Review Procedures

    The number and range of NREPP reviews are likely to expand 
significantly under the new review system, requiring that SAMHSA 
develop an efficient and cost-effective review process. The streamlined 
review procedures, protocols, and training materials will be made 
available on the NREPP Web site for access by all interested 
individuals and organizations.
    Reviews of interventions will be facilitated by doctoral-level 
Review Coordinators employed by the NREPP contractor. Each Review 
Coordinator will support two external reviewers who will assign 
numeric, criterion-based ratings on the dimensions of Strength of 
Evidence and Readiness for Dissemination. Review Coordinators will 
provide four important support and facilitative functions within the 
peer review process: (1) They will assess incoming applications for the 
thoroughness of documentation related to the intervention, including 
documentation of significant outcomes, and will convey summaries of 
this information to SAMHSA Center Directors for their use in 
prioritizing interventions for review; (2) they will serve as the 
primary liaison with the applicant to expedite the review of 
interventions; (3) they will collaborate with the NREPP applicant to 
draft the descriptive dimensions for the intervention summaries; and 
(4) they will provide summary materials and guidance to external 
reviewers to facilitate initial review and consensus discussions of 
intervention ratings.

Interventions Qualifying for Review

    While NREPP will retain its open submission policy, the new review 
system emphasizes the important role of SAMHSA's Center Directors and 
their staff (in consultation with key stakeholders) in setting 
intervention review priorities that will identify the particular 
content areas, types of

[[Page 13135]]

intervention approaches, populations, or even types of research designs 
that will qualify for review under NREPP. Under the streamlined review 
procedures, the sole requirement for potential inclusion in the NREPP 
review process is for an intervention to have demonstrated one or more 
significant behavioral change outcomes. Center-specific review 
priorities will be established and communicated to the field by posting 
them to the NREPP Web site at the beginning of each fiscal year.\2\
---------------------------------------------------------------------------

    \2\ Except for FY06 when priorities will be established and 
posted when the new system Web site is launched (i.e., within the 
third FY quarter).
---------------------------------------------------------------------------

Review of Existing NREPP Programs and Practices

    It will be the prerogative of SAMHSA Center Directors to establish 
priorities for the review and interventions already on, and pending 
entry on, NREPP. As indicated above, these decisions may be linked to 
particular approaches, populations, or strategic objectives as 
identified by SAMHSA as priority areas. Until reviews of existing NREPP 
programs and practices are completed and posted to the new NREPP Web 
site, the current listing on the SAMHSA Model Programs Web site will 
remain intact.

Notifications to Program/Practice Developers

    Upon the completion of NREPP reviews program/practice developers 
(or principal investigators of a research-based intervention) will be 
notified in writing within 2 weeks of the review results. A complete 
summary, highlighting information from each of the descriptive and 
rating dimensions, will be provided for review. Program/practice 
developers who disagree with the descriptive information or ratings 
contained in any of the dimensions will have an opportunity to discuss 
their concerns with the NREPP contractor during the 2-week period 
following receipt of the review outcome notification. These concerns 
must be expressed in writing to the contractor within this 2-week 
period. If no comments are received, the review is deemed completed, 
and the results may be posted to the NREPP Web site. If points of 
disagreement cannot be resolved by the end of this 2-week period, then 
written appeals for a rereview of the intervention may be considered on 
a case-by-case basis.

NREPP Technical Expert Panel

    SAMHSA will organize one or more expert panels to perform periodic 
(e.g., annual assessments of the evidence review system and recommend 
enhancements to to the review procedures and/or standards for evidence-
based science and practice. Panel membership will represent a balance 
of perspectives and expertise. The panels will be comprised of 
researchers with knowledge of evidence-based practices and initiatives, 
policymakers, program planners and funders, practitioners, and 
consumers.
    The modified NREPP system embodies a commitment by SAMHSA and its 
Science to Service Initiative to broaden the appeal and utility of the 
system to multiple audiences. While maintaining the focus on the 
documented outcomes achieved through a program or practice, NREPP also 
is being developed as a user-friendly decision support tool to present 
information along multiple dimensions of evidence. Under the new 
system, interventions will not receive single, overall ratings as was 
the case with the previous NREPP (e.g., Model, Effective, or 
Promising). Instead, an array of information from multiple evidence 
dimensions will be provided to allow different user audiences to both 
identify (through Web-searchable means) and prioritize the factors that 
are important to them in assessing the relative strengths of different 
evidence-based approaches to prevention or treatment services.
    Section IV presents in more detail the specific dimensions of 
descriptive information and ratings that NREPP will offer under this 
new framework.

IV. NREPP Decision Support Tool Dimensions

    The NREPP system will support evidence-based decisionmaking by 
providing a wide array of information across multiple dimensions. Many 
of these are brief descriptive dimensions that will allow users to 
identify and search for key intervention attributes of interest. 
Descriptive dimensions would frequently include a brief, searchable 
keyword or attribute (e.g., ``randomized control trial'' under the 
Evaluation Design dimension) in addition to narrative text describing 
that dimension. Two dimensions, Strength of Evidence and Readiness for 
Dissemination, will consist of quantitative, criterion-based ratings by 
reviewers. These quantitative ratings will be accompanied by reviewer 
narratives summarizing the strengths and weaknesses or the intervention 
along each dimension.

Considerations for Using NREPP as a Decision Support Tool

    It is essential for end-users to understand that the descriptive 
information and ratings provided by NREPP are only useful within a much 
broader context that incorporates a wide range of perspectives--
including clinical, consumer, administrative, fiscal, organizational, 
and policy--into decisions regarding the identification, selection, and 
successful implementation of evidence-based services. In fact, an 
emerging body of literature on implementation science \3\ suggests that 
a failure to carefully attend to this broader array of data and 
perspectives may well lead to disappointing or unsuccessful efforts to 
adopt evidence-based interventions. Because each NREPP user is likely 
to be seeking somewhat different information, and for varied purposes, 
it is unlikely that any single intervention included on NREPP will 
fulfill all of the specific requirements and unique circumstances of a 
given end-user. Appreciation of this basic premise of NREPP as a 
decision support tool to be utilized in a broader context will thus 
enable system users to make their own determinations regarding how best 
to assess and apply the information provided.
---------------------------------------------------------------------------

    \3\ Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman, R.M., & 
Wallace, F. (2005). Implementation research: A synthesis of the 
literature. Tampa, Florida: University of South Florida, Louis de la 
Parte Florida mental Health Institute, The National Implementation 
Network (FMHI Publication 231).
    Rogers (1995). Diffusion of innovaations (5th Ed.) New York: The 
Free Press.
---------------------------------------------------------------------------

    The NREPP decision support dimensions include:
     Descriptive Dimensions
     Strength of Evidence Dimension Ratings
     Readiness for Dissemination Dimension Ratings
    A complete description of these dimensions is provided in the 
sections below.

Descriptive Dimensions

     Intervention Name and Summary: Provides a brief summary of 
the intervention, including title, description of conceptual or 
theoretical foundations, and overall goals. Hyperlinks to graphic logic 
model(s), when available, could be accessed from this part of the 
summary.
     Contract Information: Lists key contact information. 
Typically will include intervention developer's title(s), affiliation, 
mailing address, telephone and fax numbers, e-mail address, and Web 
site address.
     Outcome(s): A searchable listing of the behavioral 
outcomes that the intervention has targeted.
     Effects and Impact: Provides a description and 
quantification of the effects observed for each outcome.

[[Page 13136]]

Includes information on the statistical significance of outcomes, the 
magnitude of changes reported including effect size and measures of 
clinical significance (if available), and the typical duration of 
behavioral changes produced by the intervention.
     Relevant Populations and Settings: Identifies the 
populations and sample demographics that characterize existing 
evaluations. The settings in which different populations have been 
evaluated will be characterized along a dimension that ranges from 
highly controlled and selective (i.e., efficacy studies), to less 
controlled and more representative (i.e., effectiveness studies), to 
adoption in the most diverse and realistic public health and clinical 
settings (i.e., dissemination studies).\4\
---------------------------------------------------------------------------

    \4\ For more description of these types of studies and their 
role in supporting evidence-based services, see the report: Bridging 
science and service: A report by the National Advisory mental Health 
Council's Clinical Treatment and Services Research Workgroup (http:/
/www.nimh.nih.gov/publicat/nimhbridge.pdf).
---------------------------------------------------------------------------

     Costs: Provides a breakdown of intervention cost(s) per 
recipient/participant or annual as appropriate (including capital 
costs, other direct costs [travel, etc.]). Start-up costs including 
staff training and development. A standardized template would be 
provided to applicants for estimating and summarizing the 
implementation and maintenance costs of an intervention.
     Adverse Effects: Reported with regard to type and number, 
amounts of change reported, type of data collection, analyses used, 
intervention and comparison group, and subgroups.
     Evaluation Design: Contains both a searchable index of 
specific experimental and quasi-experimental designs (e.g., pre-/
posttest nonequivalent groups designs, regression-discontinuity 
designs, interrupted time series designs, etc.) \5\ as well as a 
narrative description of the design (including intervention and 
comparison group descriptions) used to document intervention outcomes.
---------------------------------------------------------------------------

    \5\ Campbell, D.T., & Stanley, J.C. (1966). Experimental and 
quasi-experimental designs for research. Chicago: Rand McNally.
---------------------------------------------------------------------------

     Replication(s): Coded as ``None,'' or will state the 
number of replications to date (only those that have been evaluated for 
outcomes). Replications will be additionally characterized as having 
been conducted in efficacy, effectiveness, or dissemination contexts.
     Proprietary or Public Domain Intervention: Typically will 
be one or the other, but proprietary components or instruments used as 
part of an intervention will be identified.
     Cultural Appropriateness: Coded as ``Not Available'' (N/A) 
if either no data or no implementation/training materials for 
particular culturally identified groups are available. When culture-
specific data and/or implementation materials exist for one or more 
groups, the following two Yes/No questions will be provided for each 
group:
     Was the intervention developed with participation by 
members of the culturally identified group?
     Are intervention and training materials translated or 
adapted to members of the culturally identified group?
     Implementation History: Provides information relevant to 
the sustainability of interventions. Provides descriptive information 
on (1) the number of sites that have implemented the intervention; (2) 
how many of those have been evaluated for outcomes; (3) the longest 
continuous length of implementation (in years); (4) the average or 
modal length of implementation; and (5) the approximate number of 
individuals who have received or participated in the intervention.

Strength of Evidence Dimension Ratings

    Quantitative, reviewer-based ratings on this dimension will be 
provided within specific categories of research/evaluation design. In 
this manner, users can search and select within those categories of 
research designs that are most relevant to their particular standards 
of evidence-based knowledge. The categories of research design that are 
accepted within the NREPP system are described below.

Research Design

    Quality of evidence for an intervention depends on the strength of 
adequately implemented research design controls, including comparison 
conditions for quasi-experimental and randomized experimental designs 
(individual studies). Aggregation (e.g., meta-analysis and systematic 
research reviews) and/or replication across well-designed series of 
quasi-experimental and randomized control studies provide the strongest 
evidence. The evidence pyramid presented below represents a typical 
hierarchy for classifying the strength of causal inferences that can be 
obtained by implementing various research designs with rigor.\6\ 
Designs at the lowest level of evidence pyramid (i.e., observational, 
pilot, or case studies), while acceptable as evidence in some knowledge 
development contexts, would not be included in the NREPP system.
---------------------------------------------------------------------------

    \6\ Biglan, A., Mrazek, P., Carnine, D.W., & Flay, B. R. (2003). 
The integration of research and practice in the prevention of youth 
problem behaviors. American Psychologist, 58, 433-440.
    Chambless, D. L., & Hollon, S. (1998). Defining empirically 
supported therapies. Journal of Consulting and Clinical Psychology, 
66, 7-18.
    Gray, J. A. (1997), Evidence-based healthcare: How to make 
health policy and management decisions. New York: Churchill 
Livingstone.

---------------------------------------------------------------------------

[[Page 13137]]

[GRAPHIC] [TIFF OMITTED] TN14MR06.000

1. Reliability \7\
---------------------------------------------------------------------------

    \7\ Each criterion would be rated on an ordinal scale ranging 
from 0 to 4. The endpoints and midpoints of the scale would be 
anchored to a narrative description of that rating. The remaining 
integer points of the scale (i.e., 1 and 3) would not be explicitly 
anchored, but could be used by reviewers to assign intermediate 
ratings at their discretion.
---------------------------------------------------------------------------

    Outcome measures should have acceptable reliability to be 
interpretable. ``Acceptable'' here means reliability at a level that is 
conventionally accepted by experts in the field.\8\
---------------------------------------------------------------------------

    \8\ Marshall, M., Lockwood, A., Bradley, C., Adams, C., Joy, C., 
& Fenton, M. (2000). Unpublished rating scales: A major source of 
bias in randomised controlled trials of treatments for 
schizophrenia. British Journal of Psychiatry, 176, 249-252.
---------------------------------------------------------------------------

0 = Absence of evidence of reliability or evidence that some relevant 
types of reliability (e.g., test-retest, interrater, interitem) did not 
reach acceptable levels.
2 = All relevant types of reliability have been documented to be at 
acceptable levels in studies by the applicant.
4 = All relevant types of reliability have been documented to be at 
acceptable levels in studies by independent investigators.
2. Validity
    Outcome measures should have acceptable validity to be 
interpretable. ``Acceptable'' here means validity at a level that is 
conventionally accepted by experts in the field.

0 = Absence of evidence measure validity, or some evidence that the 
measure is not valid.
2 = Measure has face validity; absence of evidence that measure is not 
valid.
4 = Measure has one or more acceptable forms of criterion-related 
validity (correlation with appropriate, validated measures or objective 
criteria); OR, for objective measures of response, there are procedural 
checks to confirm data validity; absence of evidence that measure is 
not valid.
 3. Intervention Fidelity
    The ``experimental'' intervention implemented in a study should 
have fidelity to the intervention proposed by the applicant. 
Instruments that have tested acceptable psychometric properties (e.g., 
interrater reliability, validity as shown by positive association with 
outcomes) provide the highest level of evidence.

0 = Absence of evidence or only narrative evidence that the applicant 
or provider believes the intervention was implemented with acceptable 
fidelity.
2 = There is evidence of acceptable fidelity in the form of judgment(s) 
by experts, systematic collection of data (e.g. dosage, time spent in 
training, adherence to guidelines or a manual), or a fidelity measure 
with unspecified or unknown psychometric properties.
4 = There is evidence of acceptable fidelity from a tested fidelity 
instrument shown to have reliability and validity.
4. Missing Data and Attrition
    Study results can be biased by participant attrition and other 
forms of missing data. Statistical methods as supported by theory and 
research can be employed to control for missing data and attrition that 
would bias results, but studies with no attrition needing adjustment 
provide the strongest evidence that results are not biased.

0 = Missing data and attrition were taken into account inadequately, OR 
there was too much to control for bias.
2 = Missing data and attrition were taken into account by simple 
estimates of data and observations, or by demonstrations of similarity 
between remaining participants and those lost to attrition.
4 = Attrition was taken into account by more sophisticated methods that

[[Page 13138]]

model missing data, observations, or participants; OR there was no 
attrition needing adjustment.
5. Potential Confounding Variables
    Often variables other than the intervention may account for the 
reported outcomes. The degree to which confounds are accounted for 
affects the strength of casual inference.

0 = Confounding variables or factors were as likely to account for the 
outcome(s) reported as were hypothesized causes.
2 = One or more potential confounding variables or factors were not 
completely addressed, but the intervention appears more likely than 
these confounding factors to account for the outcome(s) reported.
4 = All known potential confounding variables appear to have been 
completely addressed in order to allow causal inference between 
intervention and outcome(s) reported.
6. Appropriateness of Analyses
    Appropriate analysis is necessary to make an inference that an 
intervention caused reported outcomes.

0 = Analyses were not appropriate for inferring relationships between 
intervention and outcome, OR the sample size was inadequate.
2 = Some analyses may not have been appropriate for inferring 
relationships between intervention and outcome, OR the sample size may 
have been inadequate.
4 = Analyses were appropriate for inferring relationships between 
intervention and outcome. Sample size and power were adequate.

Readiness for Dissemination Dimension Ratings

1. Availability of Implementation Materials (e.g., Treatment Manuals, 
Brochures, Information for Administrators, etc.)
0 = Applicant has insufficient implementation materials.
2 = Applicant has provided a limited range of implementation materials, 
or a comprehensive range of materials of varying or limited quality.
4 = Applicant has provided a comrephensive range of standard 
implementation materials of apparent high quality.
2. Availability of Training and Support Resources
0 = Applicant has limited or no training and support resources.
2 = Applicant provides training and support resources that are 
partially adequate to support initial and ongoing implementation.
4 = Applicant provides training and support resources that are fully 
adequate to support initial and ongoing implementation (tested training 
curricula, mechanisms for ongoing supervision and consultation).
3. Quality Improvement (QI) Materials (e.g., Fidelity Measures, Outcome 
and Performance Measures, Manuals on How To Provide QI Feedback and 
Improve Practices)
0 = Applicant has limited or no materials.
2 = Applicant has materials that are partially adequate to support 
initial and ongoing implementation.
4 = Applicant provides resources that are fully adequate to support 
initial and ongoing implementation (tested quality fidelity and outcome 
measures, comprehensive and user-friendly QI materials).

Scoring the Strength of Evidence and Readiness for Dissemination 
Dimensions

    The ratings for the decision support dimensions of Strength of 
Evidence and Readiness for Dissemination are calculated by averaging 
individual rating criteria that have been scored by reviewers according 
to a uniform five-point scale. For these two quantitative dimensions, 
the average score on each dimension (i.e., across criteria and 
reviewers) as well as average score for each rating criterion (across 
reviewers) will be provided on the Web site for each outcome targeted 
by the intervention.\9\
---------------------------------------------------------------------------

    \9\ Note that it is unlikely that the Readiness for 
Dissemination dimension will vary by targeted outcome(s), insofar as 
the materials and resources are usually program specific as opposed 
to outcome specific.
---------------------------------------------------------------------------

V. Future Activities: Implementing and Sustaining a Streamlined NREPP

    SAMHSA plans to initiate reviews using the new NREPP review process 
and procedures in summer 2006. The precise number and characteristics 
of new interventions that will be prioritized for the first series of 
reviews have yet to be determined. SAMHSA anticipates that many of the 
existing programs and practices currently listed on the SAMHSA Model 
Programs Web site will undergo an expedited set of reviews using the 
new system. Regardless, the current Model Programs Web site will remain 
intact until all relevant programs have been included in a new Web 
site, https://www.national registry.samhsa.gov
    The identification of collaborative mechanisms for supporting the 
continued development and refinement of NREPP will represent a SAMHSA 
priority in 2006. SAMHSA will explore means for providing adequate 
technical assistance resources to communities seeking to initiate and/
or augment evidence-based practices. In addition, appropriate technical 
advisors and other scientific resources will be utilized to assure the 
continued evolution of NREPP as a state-of-the-art decision support 
tool.

Appendix: Analysis of Public Comments in Response to Federal Register 
Notice

Background and Overview

    The Substance Abuse and Mental Health Services Administration 
(SAMHSA), through its Science to Service initiative, develops tools and 
resources for providers of prevention and treatment services to 
facilitate evidence-based decisionmaking and practice. An important 
informational resource is the National Registry of Evidence-based 
Programs and Practices (NREPP). NREPP is a voluntary rating and 
classification system designed to provide the public with reliable 
information on the scientific basis and practicality of interventions 
designed to prevent and/or treat mental and addictive disorders. NREPP 
originated in SAMHSA's Center for Substance Abuse Prevention (CSAP) in 
1997 as a way to help professionals in the field become better 
consumers of prevention programs. The program was expanded in 2004 to 
include substance abuse treatment interventions within SAMHSA's Center 
for Substance Abuse Treatment (CSAT) and mental health promotion and 
treatment interventions within the Center for Mental Health Services 
(CMHS).
    During the past 2 years, SAMHSA reviewed existing evidence rating 
systems and developed and pilot-tested a revised approach to the rating 
of specific outcomes achieved by programs and practices. This 
development effort led SAMHSA to propose 16 evidence rating criteria as 
well as a set of proposed utility descriptors to describe the potential 
of a given intervention to be ``transported'' to real-world settings 
and populations.
    Considering the prominence of NREPP within its Science-to-Service 
initiative and the potential impact of NREPP on the research and 
provider communities, SAMHSA announced a formal request for public 
comments in the Federal Register on August 26, 2005 (70 FR 165, 50381-
50390) with a 60-day

[[Page 13139]]

public comment period ending October 26, 2005. The notice outlined in 
some detail the proposed review system, including scientific criteria 
for evidence reviews, the screening and triage of NREPP applications, 
and the identification by SAMHSA of priority review areas. The notice 
invited general as well as specific comments and included 11 questions 
soliciting targeted feedback. By request of the SAMHSA Project Officer, 
MANILA Consulting Group coded and analyzed the responses received in 
response to the 11 questions posted in the Federal Register notice. The 
results of the analysts are presented below.

Method

    A total of 135 respondents submitted comments via e-mail, fax, and 
postal mail during the comment period. Of these 135 respondents, 109 
(81%) answered at least some of the 11 questions posted in the Federal 
Register notice.

Respondents

    The 135 respondents included 53 providers, 36 researchers, 4 
consumers, 21 respondents with multiple roles, and 21 with unknown 
roles visa-[agrave]-vis NREPP. Respondents were labeled as having one 
or more of the following domains of interest: substance abuse 
prevention (N=68), substance abuse treatment (N=48), mental health 
promotion (N=22); and mental health treatment (N=20). The domain of 
interest was unknown for 33 respondents. The respondents represented 16 
national organizations, 10 state organizations, and 14 local 
organizations; 90 were private citizens; and 5 were individuals with 
unknown affiliations. Fifty-one respondents (38%) were labeled 
``noteworthy'' at the request of the SAMHSA Project Officer. Noteworthy 
respondents included those representing national or state governments 
or national organizations, and nationally known experts in substance 
abuse or mental health research or policy.
    Twenty-six responses were judged by the four MANILA coders and the 
SAMHSA Project Officer to contain no information relevant to the 11 
questions in the notice. These responses, labeled ``unanalyzable'' for 
the purposes of this report, could be categorized as follows:
     Mentioned topics related to SAMHSA but made no point 
relevant to the questions posted in the Federal Register notice (N=10);
     Mentioned only topics unrelated to SAMHSA or incoherent 
text (N=7);
     Asked general questions about NREPP and the Federal 
Register notice (N=4);Wanted to submit a program for NREPP review 
(N=4); and
     Wanted to submit a program for NREPP review (N=4); and
     Responded to another Federal Register notice (N=1).

Procedure

    Before coding began, responses were read to identify recurrent 
themes to include in the codebook (presented in Subpart A of this 
Appendix). Using this codebook, each submission was then assigned codes 
identifying respondent characteristics (name, location, domain of 
interest, affiliation/type of organization, functional role, and level 
of response) and the content or topical themes contained in the 
response. One pair of coders coded the respondent data, while another 
pair coded the content. Content coding was conducted by two doctoral-
level psychologists with extensive training and experience in social 
science research and methodology.
    Each response could be assigned multiple codes for content. Coders 
compared their initial code assignments for all responses, discussed 
reasons for their code assignments when there were discrepancies, and 
then decided upon final code assignments. In many cases, coders 
initially assigned different codes but upon discussion agreed that both 
coders' assignments were applicable. Coding assignments were ultimately 
unanimous for all text in all responses.

Results

    The following discussion of key themes in the public comments is 
presented in order of the 11 questions from the Federal Register 
notice. Tables containing detailed frequencies of themes in the 
comments and other descriptive information are provided in Subpart B.

Comments Addressing Question 1

    Question 1. ``SAMHSA is seeking to establish an objective, 
transparent, efficient, and scientifically defensible process for 
identifying effective, evidence-based interventions to prevent and/
or treat mental and substance use disorders. Is the proposed NREPP 
system--including the suggested provisions for screening and triage 
of applications, as well as potential appeals by applicants--likely 
to accomplish these goals?''

    Respondents submitted a wide range of comments addressing Question 
1. Highlights of these comments are presented below, organized by topic 
as follows:
    1. Individual-Level Criteria
    2. Population-, Policy-, and System-Level Criteria
    3. Utility Descriptors
    4. Exclusion From NREPP Due to Lack of Funding
    5. Potential Impact on Minority Populations
    6. Potential Impact on Innovation
    7. Provider Factors
    8. Other Agencies' Standards and Resources
    9. Reliance on Intervention Developers To Submit Applications
    10. Generalizability
    11. Other Themes and Notable Comments

1. Individual-Level Criteria

Number of respondents: 24 (22%).

    Recommendations made by respondents included adding cost 
feasibility as a 13th criterion (one respondent) and scoring all 
criteria equally (two respondents). Comments regarding specific 
criteria are presented in Subpart C.

2. Population-, Policy-, and System-Level Criteria

Number of respondents: 29 (27%).

    Comments on specific criteria are presented in Subpart D. 
Highlights of comments on more general issues are presented below.
Differences in Evaluation Approaches for Individual-Level and 
Population-, Policy-, and System-Level Outcomes
    Two respondents noted the proposed NREPP approach does not 
acknowledge key differences between evaluating individual-level 
outcomes and population-, policy-, and system-level outcomes. One of 
these respondents argued that NREPP is based on theories of change that 
operate only at the individual level of analysis, with the assumption 
that discrete causes lead to discrete effects, and therefore ``many of 
the NREPP criteria appear to be insufficient or inappropriate for 
determining the validity of community-based interventions and their 
context-dependent effects.''
Unclear What Interventions Are of Interest to NREPP
    One organization, Community Anti-Drug Coalitions of America, 
recommended that SAMHSA present a clear, operational definition of the 
types of interventions it wants to include in NREPP.
Match Scale to Individual-Level Outcomes
    Twelve respondents, including the Society for Prevention Research 
and a group of researchers from a major university, recommended that 
the same scale be used for outcomes at the

[[Page 13140]]

individual level as for the population, policy, and system levels.
Add Attrition Criterion
    The same group of university researchers suggested adding attrition 
as a 13th criterion to the rating criteria for studies of population 
outcomes. They noted, ``Just as attention to attrition of individuals 
from conditions is essential in individual-level studies, attention to 
attrition of groups or communities from studies is essential in group-
level studies. This is necessary in order to assess attrition as a 
possible threat to the validity of the claim that the population-, 
policy-, or system-level intervention produced observed outcomes.''
Include Only Interventions That Change Behavior
    It was recommended that NREPP only include interventions proven to 
change behavior. A group of university researchers noted:

    As currently described, these outcomes refer to implementation 
of changes in policy or community service systems, not to changes in 
behavioral outcomes themselves. In fact, as currently described, the 
policy or system change would not be required to show any effects on 
behavior in order to be included in NREPP. This is a serious 
mistake. The NREPP system should be reserved for policies, programs, 
and system-level changes that have produced changes in actual drug 
use or mental health outcomes.

3. Utility Descriptors

Number of respondents: 15 (14%).

    Only one respondent, the Committee for Children, recommended 
specific changes to the utility descriptors. Their comments are 
presented in Subpart E of this Appendix.
    Seven other respondents recommended using utility descriptors in 
some way to score programs. The American Psychological Association 
(APA) Committee on Evidence-Based Practice recommended more emphasis on 
the utility descriptors ``as these are key outcomes for implementation 
and they are not adequately addressed in the description of NREPP 
provided to date. This underscores earlier concerns noted about the 
transition from effectiveness to efficacy.''

4. Exclusion From NREPP Due To Lack of Funding

Number of respondents: 28 (26%).

    The possibility that NREPP will exclude programs due to lack of 
funding was a concern voiced by several organizations, including the 
National Association for Children of Alcoholics, the APA Committee on 
Evidence-Based Practice, the National Association of State Alcohol and 
Drug Abuse Directors, Community Anti-Drug Coalitions of America, and 
the California Association of Alcohol and Drug Program Executives. The 
National Association for Children of Alcoholics provided the following 
comment:

    NREPP should establish differing criteria for projects that 
collected data with [National Institutes of Health] grant funds and 
projects that collected data with no or very small amounts of funds. 
It has been intrinsically unfair that only grants have been able to 
establish ``evidence'' while many programs appear very effective--
often more effective in some circumstances than NREPP approved 
programs--but have not had the Federal support or other major grant 
support to evaluate them. The SAMHSA grant programs continue to 
reinforce the designation of NREPP programs in order to qualify for 
funding, and the states tend to strengthen this `stipulation' to 
local programs, who then drop good (non-NREPP) work they have been 
doing or purchase and manipulate NREPP programs that make the grant 
possible. This is not always in the best interest of the client 
population to be served.

    Another key concern was that funding for replication research is 
rarely available. Several respondents suggested that SAMHSA consider 
funding evaluation research, and many argued that the lack of funding 
resources could negatively impact minority populations or inhibit 
treatment innovation. The latter two themes were frequent enough to be 
coded and analyzed separately. Results are summarized in the following 
sections.

5. Potential Impact on Minority Populations

Number of respondents: 13 (12%).

    Thirteen respondents noted that the proposed NREPP approach could 
negatively impact specific populations, including minority client 
populations. The Federation of Families for Children's Mental Health 
suggested that NREPP would effectively promote certain practices 
``simply because the resources for promotion, training, evaluation are 
readily accessible * * * thus widening the expanse and disparities that 
currently exist.''
    Another frequently noted concern was that evidence-based practices 
are currently too narrowly defined, and thus as more funding sources 
begin to require evidence-based practices as a prerequisite for 
funding, some ethnic or racial minority organizations may be excluded 
from funding. One respondent also pointed to potential validity 
concerns, noting that ``Very little clinical trial evidence is 
available for how to treat substance use disorders in specific 
populations who may constitute most or all of those seen in particular 
agencies: HIV positive patients, native Americans, adolescents, 
Hispanics, or African Americans. Although it is unreasonable to expect 
all EBTs to be tested with all populations, the external validity of 
existing studies remains a serious concern.'' For these reasons, many 
respondents surmised that the widespread application of interventions 
developed in research contexts that might tend to limit the inclusion 
of minority and/or underserved populations could ultimately result in 
decreased cultural competence among service providers.

6. Potential Impact on Innovation

Number of respondents: 21 (19%).

    Twenty-one respondents cited concerns that the proposed NREPP 
approach could hamper innovation. CAADPE noted that its main concerns 
were ``the focus on the premise that treatment will improve if confined 
to inteventions for which a certain type of research evidence is 
available'' and ``the issue of `branding,' which could lead to some of 
our most innovative and effective small scale providers eliminated from 
funding considerations.''
    One respondent suggested that lists of evidence-based treatments 
could ``ossify research and practice, and thus become self-fulfilling 
prophecies * * * stifling innovation and the validation of existing 
alternatives.'' Several respondents observed that the potential for 
stifling innovation is even greater given that SAMHSA's NREPP is not 
the only list of evidence-based practices used by funders.
    The APA Practice Organization recommended that NREPP focus on 
``developing and promoting a range of more accessible and less 
stigmatized services that are responsive to consumers' needs and 
preference, and offer more extensive care opportunities.''

7. Provider Factors

Number of respondents: 22 (20%).

    A number of respondents noted the proposed NREPP approach does not 
acknowledge provider effects on treatment outcomes. The APA Committee 
on Evidence-Based Practice wrote, ``Relationship factors in a 
therapeutic process may be more important than specific interventions 
and may in fact be the largest determinant in psychotherapy outcome 
(see Lambert & Barley, 2002). How will NREPP address this concern and 
make this apparent to users?''
    Another respondent cited the Institute of Medicine's definition of 
evidence-

[[Page 13141]]

based practice as ``the integration of the best research evidence with 
clinical expertise and client values,'' noting that ``The narrowed 
interpretation of evidence-based practice by SAMHSA focuses almost 
solely on the research evidence to the exclusion of clinical expertise 
and patient values.''
    Several respondents suggested that NREPP could place too much 
emphasis on highly prescriptive, annualized treatments. Counselors can 
become bored when they are not able ti ``tinker'' with or adapt 
treatments. In addition, making minor modifications may actually make 
treatments more effective with different population groups.

8. Other Agencies' Standards and Resources

Number of respondents: 27 (25%).

    Nineteen respondents suggested that, in developing NREPP, SAMHSA 
should consult other agencies' standards and resources related to 
evidence-based practices--for example, the standards published by the 
APA, American Society for Addiction Medicine, and the Society for 
Prevention Research. One respondent suggested consulting with National 
Institutes of Health scientists about approaches for aggregating 
evidence; another recommended including in NREPP model programs 
identified by other agencies. One respondent submitted a bibliography 
of references for assessing the rigor of qualitative research.
    One respondent suggested that SAMHSA did not provide other 
institutions the opportunity to provide input on the development of 
NREPP prior to the request for public comments.

9. Reliance on Intervention Developers To Submit Applications

Number of respondents: 4 (4%).

    Four respondents cited problems with NREPP's reliance on 
intervention developers to submit applications, and suggested that 
literature reviews instead be used to identify programs eligible for 
NREPP. One private citizen wrote, ``If no one applies on behalf of a 
treatment method, is that one ignored? Why not simply start with the 
literature and identify treatment methods with adequate evidence of 
efficacy?''
    Another respondent observed that requiring an application creates a 
bias toward programs with advocates ``either ideologically or because 
of a vested interest in sales, visibility, and profits. An alternative 
is to select interventions for NREPP consideration solely by monitoring 
the peer-reviewed published literature, and including them regardless 
of whether or not the scientist responds or furthers the registration 
process.''
    The Society for Prevention Research suggested that SAMHSA convene a 
panel to periodically review available interventions that might not be 
submitted to NREPP because they ``lack a champion.''

10. Generalizability

Number of respondents: 48 (44%).

    Many respondents discussed the issue of generalizability of 
evidence, especially the concern that interventions proven to work in 
clinical trials do not always work in real-world settings. Several 
respondents pointed out the potential conflict between implementing an 
intervention with fidelity and having a adapt it for the setting.
    The APA Evidence-Based Practice Committee suggested that the 
proposed NREPP approach does not adequately distinguish between 
``efficacy'' and ``effectiveness,'' and strongly recommended that 
SAMHSA look for ways to bridge the two.
    The Associations of Addiction Services recommended paying more 
attention to how and where treatments are replicated: ``The highest 
level of evidence should be successful replication of the approach in 
multiple community treatment settings. Experience with [the National 
Institute on Drug Abuse] Clinical Trials Network suggests that an 
approach that shows meaningful outcome improvements in the `noisy' 
setting of a publicly funded community treatment program is truly an 
approach worth promoting.''
    A few respondents suggested that NREPP score interventions 
according to their readiness and amenability to application in real-
world settings.

11. Other Themes and Notable Comments

Distinguishing Treatment and Prevention
Number of respondents: 7 (6%).

    A few respondents called or evaluating treatment and prevention 
approaches differently. One respondent noted that some criteria appear 
to be more appropriate for treatment modalities than for preventive 
interventions, and recommended that SAMHSA ``confer with research 
experts in those respective fields and separate out those criteria that 
are more relevant to only treatment or prevention.''
    Another respondent suggested that the criteria are more appropriate 
for prevention that treatment:

    The criteria and selection for the peer review panels should be 
separate for prevention and treatment programs. The criteria and 
models are different and the panels should not be an across the 
board effort, but rather representative of prevention and treatment 
experts specific to the program being evaluated. The plan is based 
as the notice states on 1,100 prevention programs with little 
experience with treatment programs/practices.
Synthesizing Evidence
    Three respondents suggested using meta-analysis to synthesize 
evidence for outcomes. One recommended SAMHSA consult with National 
Institutes of Health experts in this area.
Replications
    The Teaching-Family Association recommended considering 
replications when evaluating evidence. The Society for Prevention 
Research wrote that it is unclear how replications would be used in the 
proposed NREPP, and suggested averaging ratings across studies.
Add Criteria
    The National Student Assistance Association Scientific Advisory 
Board and one other respondent suggested adding a cultural competence 
criterion. The Society for Prevention Research recommended adding a 
criterion to assess the clarity of causal inference.
Range of Reviewer Perspectives
    The APA Practice Association noted the importance of having a 
``large and broad'' reviewer pool: ``A small group of reviewers 
representing a limited range of perspectives and constituencies would 
have an undue impact on the entire system. We are pleased that a 
nominations process is envisioned.''
Cost Effectiveness
    One respondent called for incorporating program cost effectiveness 
into NREPP. In choosing what program to implement, end users often have 
to decide between diverse possibilities, such as attempting to pass a 
tax increase on beer or implementing additional classroom prevention 
curricula, each with competing claims about effectiveness. A cost-
effectiveness framework may be the only way to compare these choices.

Comments Addressing Question 2

    Question 2. ``SAMHSA's NREPP priorities are reflected in the 
agency's matrix of program priority areas. How might SAMHSA engage 
interested stakeholders on a periodic basis in helping the agency 
determine intervention priority areas for review by NREPP?''

Number of respondents: 16 (15%).


[[Page 13142]]


    Respondents recommended a number of approaches to engage 
stakeholders:
     Conduct meetings, conferences, and seminars.
     Send and solicit information via e-mail or a Web site.
     Send informational notices via newletters.
     Survey stakeholders.
     Work with the Addiction Technology Transfer Centers 
(ATTCs) to administer surveys.
     Consult the National Prevention Network and the Society 
for Prevention Research, which ``have forged a close working 
relationship to foster the integration of science and practice and * * 
* would be very helpful in answering this question.''

Comments Addressing Question 3

    Question 3. ``There has been considerable discussion in the 
scientific literature on how to use statistical significance and 
various measures of effect size in assessing