Final Information Quality Bulletin for Peer Review, 2664-2677 [05-769]

Agencies

• Office of Management and Budget
• Management and Budget Office

[Federal Register Volume 70, Number 10 (Friday, January 14, 2005)]
[Notices]
[Pages 2664-2677]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 05-769]


=======================================================================
-----------------------------------------------------------------------

OFFICE OF MANAGEMENT AND BUDGET


Final Information Quality Bulletin for Peer Review

AGENCY: Office of Management and Budget, Executive Office of the 
President.

ACTION: Final bulletin.

-----------------------------------------------------------------------

SUMMARY: On December 16, 2004, the Office of Management and Budget 
(OMB), in consultation with the Office of Science and Technology Policy 
(OSTP), issued its Final Information Quality Bulletin for Peer Review 
to the heads of departments and agencies (available at https://
www.whitehouse.gov/omb/memoranda/fy2005/m05-03.html). This new guidance 
is designed to realize the benefits of meaningful peer review of the 
most important science disseminated by the Federal Government. It is 
part of an ongoing effort to improve the quality, objectivity, utility, 
and integrity of information disseminated by the Federal Government to 
the public. This final bulletin has benefited from an extensive 
stakeholder process. OMB originally requested comment on its ``Proposed 
Bulletin on Peer Review and Information Quality,'' published in the 
Federal Register on September 15, 2003. OMB received 187 public 
comments during the comment period (available at https://
www.whitehouse.gov/omb/inforeg/2003iq/iq_list.html). In addition, to 
improve the draft Bulletin, OMB encouraged federal agencies to sponsor 
a public workshop at the National Academy of Sciences (NAS). The NAS 
workshop (November 18, 2003, at the National Academies in Washington, 
DC) attracted several hundred participants, including leaders in the 
scientific community (available at https://www7.nationalacademies.org/
stl/STL_Peer_Review_Agenda.html). OMB also participated in outreach 
activities with major scientific organizations and societies that had 
expressed specific interest in the draft Bulletin. A formal interagency 
review of the draft Bulletin, resulting in detailed comments from 
numerous Federal departments and agencies, was undertaken in 
collaboration with the White House Office of Science and Technology 
Policy. In light of the substantial interest in the Bulletin, including 
a wide range of constructive criticisms of the initial draft, OMB 
decided to issue a revised draft for further comment. This revised 
draft was published in the Federal Register on April 28, 2004, and 
solicited a second round of public comment. The revised draft 
stimulated a much smaller number of comments (57) (available at: http:/
/www.whitehouse.gov/omb/inforeg/peer2004/list_peer2004.html). OMB's 
response to the additional criticisms, suggestions, and refinements 
offered for consideration is available at: https://www.whitehouse.gov/
omb/inforeg/peer2004/peer_response.pdf. The final Bulletin includes 
refinements that strike a balance among the diverse perspectives 
expressed during the comment period. Part I of the SUPPLEMENTARY 
INFORMATION below provides background. Part II provides the text of the 
final Bulletin.

DATES: The requirements of this Bulletin, with the exception of those 
in Section V (Peer Review Planning), apply to information disseminated 
on or after June 16, 2005. However, they do not apply to information 
for which an agency has already provided a draft report and an 
associated charge to peer reviewers. The requirements in Section V 
regarding ``highly influential scientific assessments'' are effective 
June 16, 2005. The requirements in Section V regarding ``influential 
scientific information'' are effective December 16, 2005.

[[Page 2665]]


FOR FURTHER INFORMATION CONTACT: Dr. Margo Schwab, Office of 
Information and Regulatory Affairs, Office of Management and Budget, 
725 17th Street, NW., New Executive Office Building, Room 10201, 
Washington, DC 20503. Telephone (202) 395-5647 or email: OMB_peer_
review@omb.eop.gov.

SUPPLEMENTARY INFORMATION: 

Introduction

    This Bulletin establishes that important scientific information 
shall be peer reviewed by qualified specialists before it is 
disseminated by the Federal government. We published a proposed 
Bulletin on September 15, 2003. Based on public comments, we published 
a revised proposal for additional comment on April 28, 2004. We are now 
finalizing the April version, with minor revisions responsive to the 
public's comments.
    The purpose of the Bulletin is to enhance the quality and 
credibility of the government's scientific information. We recognize 
that different types of peer review are appropriate for different types 
of information. Under this Bulletin, agencies are granted broad 
discretion to weigh the benefits and costs of using a particular peer 
review mechanism for a specific information product. The selection of 
an appropriate peer review mechanism for scientific information is left 
to the agency's discretion. Various types of information are exempted 
from the requirements of this Bulletin, including time-sensitive health 
and safety determinations, in order to ensure that peer review does not 
unduly delay the release of urgent findings.
    This Bulletin also applies stricter minimum requirements for the 
peer review of highly influential scientific assessments, which are a 
subset of influential scientific information. A scientific assessment 
is an evaluation of a body of scientific or technical knowledge that 
typically synthesizes multiple factual inputs, data, models, 
assumptions, and/or applies best professional judgment to bridge 
uncertainties in the available information. To ensure that the Bulletin 
is not too costly or rigid, these requirements for more intensive peer 
review apply only to the more important scientific assessments 
disseminated by the Federal government.
    Even for these highly influential scientific assessments, the 
Bulletin leaves significant discretion to the agency formulating the 
peer review plan. In general, an agency conducting a peer review of a 
highly influential scientific assessment must ensure that the peer 
review process is transparent by making available to the public the 
written charge to the peer reviewers, the peer reviewers' names, the 
peer reviewers' report(s), and the agency's response to the peer 
reviewers' report(s). The agency selecting peer reviewers must ensure 
that the reviewers possess the necessary expertise. In addition, the 
agency must address reviewers' potential conflicts of interest 
(including those stemming from ties to regulated businesses and other 
stakeholders) and independence from the agency. This Bulletin requires 
agencies to adopt or adapt the committee selection policies employed by 
the National Academy of Sciences (NAS) \1\ when selecting peer 
reviewers who are not government employees. Those that are government 
employees are subject to federal ethics requirements. The use of a 
transparent process, coupled with the selection of qualified and 
independent peer reviewers, should improve the quality of government 
science while promoting public confidence in the integrity of the 
government's scientific products.
---------------------------------------------------------------------------

    \1\ National Academy of Sciences, ``Policy and Procedures on 
Committee Composition and Balance and Conflicts of Interest for 
Committees Used in the Development of Reports,'' May 2003: Available 
at: https://www.nationalacademies.org/coi/.
---------------------------------------------------------------------------

Peer Review

    Peer review is one of the important procedures used to ensure that 
the quality of published information meets the standards of the 
scientific and technical community. It is a form of deliberation 
involving an exchange of judgments about the appropriateness of methods 
and the strength of the author's inferences.\2\ Peer review involves 
the review of a draft product for quality by specialists in the field 
who were not involved in producing the draft.
---------------------------------------------------------------------------

    \2\ Carnegie Commission on Science, Technology, and Government, 
Risk and the Environment: Improving Regulatory Decision Making, 
Carnegie Commission, New York, 1993: 75.
---------------------------------------------------------------------------

    The peer reviewer's report is an evaluation or critique that is 
used by the authors of the draft to improve the product. Peer review 
typically evaluates the clarity of hypotheses, the validity of the 
research design, the quality of data collection procedures, the 
robustness of the methods employed, the appropriateness of the methods 
for the hypotheses being tested, the extent to which the conclusions 
follow from the analysis, and the strengths and limitations of the 
overall product.
    Peer review has diverse purposes. Editors of scientific journals 
use reviewer comments to help determine whether a draft scientific 
article is of sufficient quality, importance, and interest to a field 
of study to justify publication. Research funding organizations often 
use peer review to evaluate research proposals. In addition, some 
Federal agencies make use of peer review to obtain evaluations of draft 
information that contains important scientific determinations.
    Peer review should not be confused with public comment and other 
stakeholder processes. The selection of participants in a peer review 
is based on expertise, with due consideration of independence and 
conflict of interest. Furthermore, notice-and-comment procedures for 
agency rulemaking do not provide an adequate substitute for peer 
review, as some experts--especially those most knowledgeable in a 
field--may not file public comments with Federal agencies.
    The critique provided by a peer review often suggests ways to 
clarify assumptions, findings, and conclusions. For instance, peer 
reviews can filter out biases and identify oversights, omissions, and 
inconsistencies.\3\ Peer review also may encourage authors to more 
fully acknowledge limitations and uncertainties. In some cases, 
reviewers might recommend major changes to the draft, such as 
refinement of hypotheses, reconsideration of research design, 
modifications of data collection or analysis methods, or alternative 
conclusions. However, peer review does not always lead to specific 
modifications in the draft product. In some cases, a draft is in 
excellent shape prior to being submitted for review. In others, the 
authors do not concur with changes suggested by one or more reviewers.
---------------------------------------------------------------------------

    \3\ William W. Lowrance, Modern Science and Human Values, Oxford 
University Press, New York, NY 1985: 85.
---------------------------------------------------------------------------

    Peer review may take a variety of forms, depending upon the nature 
and importance of the product. For example, the reviewers may represent 
one scientific discipline or a variety of disciplines; the number of 
reviewers may range from a few to more than a dozen; the names of each 
reviewer may be disclosed publicly or may remain anonymous (e.g., to 
encourage candor); the reviewers may be blinded to the authors of the 
report or the names of the authors may be disclosed to the reviewers; 
the reviewers may prepare individual reports or a panel of reviewers 
may be constituted to produce a collaborative report; panels may do 
their work electronically or they may meet together in person to 
discuss and prepare their evaluations; and reviewers may be compensated 
for their work or they may donate their time as a

[[Page 2666]]

contribution to science or public service.
    For large, complex reports, different reviewers may be assigned to 
different chapters or topics. Such reports may be reviewed in stages, 
sometimes with confidential reviews that precede a public process of 
panel review. As part of government-sponsored peer review, there may be 
opportunity for written and/or oral public comments on the draft 
product.
    The results of peer review are often only one of the criteria used 
to make decisions about journal publication, grant funding, and 
information dissemination. For instance, the editors of scientific 
journals (rather than the peer reviewers) make final decisions about a 
manuscript's appropriateness for publication based on a variety of 
considerations. In research-funding decisions, the reports of peer 
reviewers often play an important role, but the final decisions about 
funding are often made by accountable officials based on a variety of 
considerations. Similarly, when a government agency sponsors peer 
review of its own draft documents, the peer review reports are an 
important factor in information dissemination decisions but rarely are 
the sole consideration. Agencies are not expected to cede their 
discretion with regard to dissemination or use of information to peer 
reviewers; accountable agency officials must make the final decisions.

The Need for Stronger Peer Review Policies

    There are a multiplicity of science advisory procedures used at 
Federal agencies and across the wide variety of scientific products 
prepared by agencies.\4\ In response to congressional inquiry, the U.S. 
General Accounting Office (now the Government Accountability Office) 
documented the variability in both the definition and implementation of 
peer review across agencies.\5\ The Carnegie Commission on Science, 
Technology and Government \6\ has highlighted the importance of 
``internal'' scientific advice (within the agency) and ``external'' 
advice (through scientific advisory boards and other mechanisms).
---------------------------------------------------------------------------

    \4\ Sheila Jasanoff, The Fifth Branch: Science Advisors as 
Policy Makers, Harvard University Press, Boston, 1990.
    \5\ U.S. General Accounting Office, Federal Research: Peer 
Review Practices at Federal Agencies Vary, GAO/RCED-99-99, 
Washington, DC, 1999.
    \6\ Carnegie Commission on Science, Technology, and Government, 
Risk and the Environment: Improving Regulatory Decision Making, 
Carnegie Commission, New York, 1993: 90.
---------------------------------------------------------------------------

    A wide variety of authorities have argued that peer review 
practices at federal agencies need to be strengthened.\7\ Some 
arguments focus on specific types of scientific products (e.g., 
assessments of health, safety and environmental hazards).\8\ The 
Congressional/Presidential Commission on Risk Assessment and Risk 
Management suggests that ``peer review of economic and social science 
information should have as high a priority as peer review of health, 
ecological, and engineering information.'' \9\
---------------------------------------------------------------------------

    \7\ National Academy of Sciences, Peer Review in the Department 
of Energy--Office of Science and Technology, Interim Report, 
National Academy Press, Washington, DC, 1997; National Academy of 
Sciences, Peer Review in Environmental Technology Development: The 
Department of Energy--Office of Science and Technology, National 
Academy Press, Washington, DC, 1998; National Academy of Sciences, 
Strengthening Science at the U.S. Environmental Protection Agency: 
Research-Management and Peer-Review Practices, National Academy 
Press, Washington, DC, 2000; U.S. General Accounting Office, EPA's 
Science Advisory Board Panels: Improved Policies and Procedures 
Needed to Ensure Independence and Balance, GAO-01-536, Washington, 
DC, 2001; U.S. Environmental Protection Agency, Office of Inspector 
General, Pilot Study: Science in Support of Rulemaking 2003-P-00003, 
Washington, DC, 2002; Carnegie Commission on Science, Technology, 
and Government, In the National Interest: The Federal Government in 
the Reform of K-12 Math and Science Education, Carnegie Commission, 
New York, 1991; U.S. General Accounting Office, Endangered Species 
Program: Information on How Funds Are Allocated and What Activities 
are Emphasized, GAO-02-581, Washington, DC, 2002.
    \8\ National Research Council, Science and Judgment in Risk 
Assessment, National Academy Press, Washington, DC, 1994.
    \9\ Presidential/Congressional Commission on Risk Assessment and 
Risk Management, Risk Commission Report, Volume 2, Risk Assessment 
and Risk Management in Regulatory Decision-Making, 1997:103.
---------------------------------------------------------------------------

    Some agencies have formal peer review policies, while others do 
not. Even agencies that have such policies do not always follow them 
prior to the release of important scientific products.
    Prior to the development of this Bulletin, there were no 
government-wide standards concerning when peer review is required and, 
if required, what type of peer review processes are appropriate. No 
formal interagency mechanism existed to foster cross-agency sharing of 
experiences with peer review practices and policies. Despite the 
importance of peer review for the credibility of agency scientific 
products, the public lacked a consistent way to determine when an 
important scientific information product is being developed by an 
agency, the type of peer review planned for that product, or whether 
there would be an opportunity to provide comments and data to the 
reviewers.
    This Bulletin establishes minimum standards for when peer review is 
required for scientific information and the types of peer review that 
should be considered by agencies in different circumstances. It also 
establishes a transparent process for public disclosure of peer review 
planning, including a Web-accessible description of the peer review 
plan that the agency has developed for each of its forthcoming 
influential scientific disseminations.

Legal Authority for the Bulletin

    This Bulletin is issued under the Information Quality Act and OMB's 
general authorities to oversee the quality of agency information, 
analyses, and regulatory actions. In the Information Quality Act, 
Congress directed OMB to issue guidelines to ``provide policy and 
procedural guidance to Federal agencies for ensuring and maximizing the 
quality, objectivity, utility and integrity of information'' 
disseminated by Federal agencies. Public Law No. 106-554, Sec.  515(a). 
The Information Quality Act was developed as a supplement to the 
Paperwork Reduction Act, 44 U.S.C. 3501 et seq., which requires OMB, 
among other things, to ``develop and oversee the implementation of 
policies, principles, standards, and guidelines to * * * apply to 
Federal agency dissemination of public information.'' In addition, 
Executive Order 12866, 58 FR 51,735 (Oct. 4, 1993), establishes that 
OIRA is ``the repository of expertise concerning regulatory issues,'' 
and it directs OMB to provide guidance to the agencies on regulatory 
planning. E.O. 12866, Sec.  2(b). The Order also requires that ``[e]ach 
agency shall base its decisions on the best reasonably obtainable 
scientific, technical, economic, or other information.'' E.O. 12866, 
Sec.  1(b)(7). Finally, OMB has authority in certain circumstances to 
manage the agencies under the purview of the President's Constitutional 
authority to supervise the unitary Executive Branch. All of these 
authorities support this Bulletin.

The Requirements of This Bulletin

    This Bulletin addresses peer review of scientific information 
disseminations that contain findings or conclusions that represent the 
official position of one or more agencies of the Federal government.

Section I: Definitions

    Section I provides definitions that are central to this Bulletin. 
Several terms are identical to or based on those used in OMB's 
government-wide information quality guidelines, 67 FR 8452 (Feb. 22, 
2002), and the Paperwork Reduction Act, 44 U.S.C. 3501 et seq.

[[Page 2667]]

    The term ``Administrator'' means the Administrator of the Office of 
Information and Regulatory Affairs in the Office of Management and 
Budget (OIRA).
    The term ``agency'' has the same meaning as in the Paperwork 
Reduction Act, 44 U.S.C. 3502(1).
    The term ``Information Quality Act'' means Section 515 of Public 
Law 106-554 (Pub. L. No. 106-554, Sec.  515, 114 Stat. 2763, 2763A-153-
154 (2000)).
    The term ``dissemination'' means agency initiated or sponsored 
distribution of information to the public. Dissemination does not 
include distribution limited to government employees or agency 
contractors or grantees; intra-or inter-agency use or sharing of 
government information; or responses to requests for agency records 
under the Freedom of Information Act, the Privacy Act, the Federal 
Advisory Committee Act, the Government Performance and Results Act, or 
similar laws. This definition also excludes distribution limited to 
correspondence with individuals or persons, press releases, archival 
records, public filings, subpoenas and adjudicative processes. In the 
context of this Bulletin, the definition of ``dissemination'' modifies 
the definition in OMB's government-wide information quality guidelines 
to address the need for peer review prior to official dissemination of 
the information product. Accordingly, under this Bulletin, 
``dissemination'' also excludes information distributed for peer review 
in compliance with this Bulletin or shared confidentially with 
scientific colleagues, provided that the distributing agency includes 
an appropriate and clear disclaimer on the information, as explained 
more fully below. Finally, the Bulletin does not directly cover 
information supplied to the government by third parties (e.g., studies 
by private consultants, companies and private, non-profit 
organizations, or research institutions such as universities). However, 
if an agency plans to disseminate information supplied by a third party 
(e.g., using this information as the basis for an agency's factual 
determination that a particular behavior causes a disease), the 
requirements of the Bulletin apply, if the dissemination is 
``influential''.
    In cases where a draft report or other information is released by 
an agency solely for purposes of peer review, a question may arise as 
to whether the draft report constitutes an official ``dissemination'' 
under information-quality guidelines. Section I instructs agencies to 
make this clear by presenting the following disclaimer in the report:

This information is distributed solely for the purpose of pre-
dissemination peer review under applicable information quality 
guidelines. It has not been formally disseminated by [the agency]. 
It does not represent and should not be construed to represent any 
agency determination or policy.

    In cases where the information is highly relevant to specific 
policy or regulatory deliberations, this disclaimer shall appear on 
each page of a draft report. Agencies also shall discourage state, 
local, international and private organizations from using information 
in draft reports that are undergoing peer review. Draft influential 
scientific information presented at scientific meetings or shared 
confidentially with colleagues for scientific input prior to peer 
review shall include the disclaimer: ``The Findings and Conclusions in 
This Report (Presentation) Have Not Been Formally Disseminated by [The 
Agency] and Should Not Be Construed to Represent Any Agency 
Determination or Policy.''
    An information product is not covered by the Bulletin unless it 
represents an official view of one or more departments or agencies of 
the Federal government. Accordingly, for the purposes of this Bulletin, 
``dissemination'' excludes research produced by government-funded 
scientists (e.g., those supported extramurally or intramurally by 
Federal agencies or those working in state or local governments with 
Federal support) if that information is not represented as the views of 
a department or agency (i.e., they are not official government 
disseminations). For influential scientific information that does not 
have the imprimatur of the Federal government, scientists employed by 
the Federal government are required to include in their information 
product a clear disclaimer that ``the findings and conclusions in this 
report are those of the author(s) and do not necessarily represent the 
views of the funding agency.'' A similar disclaimer is advised for non-
government employees who publish government-funded research.
    For the purposes of the peer review Bulletin, the term ``scientific 
information'' means factual inputs, data, models, analyses, technical 
information, or scientific assessments related to such disciplines as 
the behavioral and social sciences, public health and medical sciences, 
life and earth sciences, engineering, or physical sciences. This 
includes any communication or representation of knowledge such as facts 
or data, in any medium or form, including textual, numerical, graphic, 
cartographic, narrative, or audiovisual forms. This definition includes 
information that an agency disseminates from a Web page, but does not 
include the provision of hyperlinks on a Web page to information that 
others disseminate. This definition excludes opinions, where the 
agency's presentation makes clear that an individual's opinion, rather 
than a statement of fact or of the agency's findings and conclusions, 
is being offered.
    The term ``influential scientific information'' means scientific 
information the agency reasonably can determine will have or does have 
a clear and substantial impact on important public policies or private 
sector decisions. In the term ``influential scientific information,'' 
the term ``influential'' should be interpreted consistently with OMB's 
government-wide information quality guidelines and the information 
quality guidelines of the agency. Information dissemination can have a 
significant economic impact even if it is not part of a rulemaking. For 
instance, the economic viability of a technology can be influenced by 
the government's characterization of its attributes. Alternatively, the 
Federal government's assessment of risk can directly or indirectly 
influence the response actions of state and local agencies or 
international bodies.
    One type of scientific information is a scientific assessment. For 
the purposes of this Bulletin, the term ``scientific assessment'' means 
an evaluation of a body of scientific or technical knowledge, which 
typically synthesizes multiple factual inputs, data, models, 
assumptions, and/or applies best professional judgment to bridge 
uncertainties in the available information. These assessments include, 
but are not limited to, state-of-science reports; technology 
assessments; weight-of-evidence analyses; meta-analyses; health, 
safety, or ecological risk assessments; toxicological characterizations 
of substances; integrated assessment models; hazard determinations; or 
exposure assessments. Such assessments often draw upon knowledge from 
multiple disciplines. Typically, the data and models used in scientific 
assessments have already been subject to some form of peer review 
(e.g., refereed journal peer review or peer review under Section II of 
this Bulletin).

Section II: Peer Review of Influential Scientific Information

    Section II requires each agency to subject ``influential'' 
scientific information to peer review prior to dissemination. For 
dissemination of

[[Page 2668]]

influential scientific information, Section II provides agencies broad 
discretion in determining what type of peer review is appropriate and 
what procedures should be employed to select appropriate reviewers. 
Agencies are directed to chose a peer review mechanism that is 
adequate, giving due consideration to the novelty and complexity of the 
science to be reviewed, the relevance of the information to decision 
making, the extent of prior peer reviews, and the expected benefits and 
costs of additional review.
    The National Academy of Public Administration suggests that the 
intensity of peer review should be commensurate with the significance 
of the information being disseminated and the likely implications for 
policy decisions.\10\ Furthermore, agencies need to consider tradeoffs 
between depth of peer review and timeliness.\11\ More rigorous peer 
review is necessary for information that is based on novel methods or 
presents complex challenges for interpretation. Furthermore, the need 
for rigorous peer review is greater when the information contains 
precedent-setting methods or models, presents conclusions that are 
likely to change prevailing practices, or is likely to affect policy 
decisions that have a significant impact.
---------------------------------------------------------------------------

    \10\ National Academy of Public Administration, Setting 
Priorities, Getting Results: A New Direction for EPA, National 
Academy Press, Washington, DC, 1995:23.
    \11\ Presidential/Congressional Commission on Risk Assessment 
and Risk Management, Risk Commission Report, 1997.
---------------------------------------------------------------------------

    This tradeoff can be considered in a benefit-cost framework. The 
costs of peer review include both the direct costs of the peer review 
activity and those stemming from potential delay in government and 
private actions that can result from peer review. The benefits of peer 
review are equally clear: the insights offered by peer reviewers may 
lead to policy with more benefits and/or fewer costs. In addition to 
contributing to strong science, peer review, if performed fairly and 
rigorously, can build consensus among stakeholders and reduce the 
temptation for courts and legislators to second-guess or overturn 
agency actions.\12\ While it will not always be easy for agencies to 
quantify the benefits and costs of peer review, agencies are encouraged 
to approach peer review from a benefit-cost perspective.
---------------------------------------------------------------------------

    \12\ Mark R. Powell, Science at EPA: Information in the 
Regulatory Process, Resources for the Future, Washington, DC, 1999: 
148, 176; Sheila Jasanoff, The Fifth Branch: Science Advisors as 
Policy Makers, Harvard University Press, Boston, 1990: 242.
---------------------------------------------------------------------------

    Regardless of the peer review mechanism chosen, agencies should 
strive to ensure that their peer review practices are characterized by 
both scientific integrity and process integrity. ``Scientific 
integrity,'' in the context of peer review, refers to such issues as 
``expertise and balance of the panel members; the identification of the 
scientific issues and clarity of the charge to the panel; the quality, 
focus and depth of the discussion of the issues by the panel; the 
rationale and supportability of the panel's findings; and the accuracy 
and clarity of the panel report.'' ``Process integrity'' includes such 
issues as ``transparency and openness, avoidance of real or perceived 
conflicts of interest, a workable process for public comment and 
involvement,'' and adherence to defined procedures.\13\
---------------------------------------------------------------------------

    \13\ ILSI Risk Sciences Institute, ``Policies and Procedures: 
Model Peer Review Center of Excellence,'' 2002: 4. Available at 
https://rsi.ilsi.org/file/Policies&Procedures.pdf.
---------------------------------------------------------------------------

    When deciding what type of peer review mechanism is appropriate for 
a specific information product, agencies will need to consider at least 
the following issues: Individual versus panel review; timing; scope of 
the review; selection of reviewers; disclosure and attribution; public 
participation; disposition of reviewer comments; and adequacy of prior 
peer review.
Individual Versus Panel Review
    Letter reviews by several experts generally will be more 
expeditious than convening a panel of experts. Individual letter 
reviews are more appropriate when a draft document covers only one 
discipline or when premature disclosure of a sensitive report to a 
public panel could cause harm to government or private interests. When 
time and resources warrant, panels are preferable, as they tend to be 
more deliberative than individual letter reviews and the reviewers can 
learn from each other. There are also multi-stage processes in which 
confidential letter reviews are conducted prior to release of a draft 
document for public notice and comment, followed by a formal panel 
review. These more rigorous and expensive processes are particularly 
valuable for highly complex, multidisciplinary, and more important 
documents, especially those that are novel or precedent-setting.
Timing of Peer Review
    As a general rule, it is most useful to consult with peers early in 
the process of producing information. For example, in the context of 
risk assessments, it is valuable to have the choice of input data and 
the specification of the model reviewed by peers before the agency 
invests time and resources in implementing the model and interpreting 
the results. ``Early'' peer review occurs in time to ``focus attention 
on data inadequacies in time for corrections.
    When an information product is a critical component of rule-making, 
it is important to obtain peer review before the agency announces its 
regulatory options so that any technical corrections can be made before 
the agency becomes invested in a specific approach or the positions of 
interest groups have hardened. If review occurs too late, it is 
unlikely to contribute to the course of a rulemaking. Furthermore, 
investing in a more rigorous peer review early in the process ``may 
provide net benefit by reducing the prospect of challenges to a 
regulation that later may trigger time consuming and resource-draining 
litigation.'' \14\
---------------------------------------------------------------------------

    \14\ Fred Anderson, Mary Ann Chirba Martin, E Donald Elliott, 
Cynthia Farina, Ernest Gellhorn, John D. Graham, C. Boyden Gray, 
Jeffrey Holmstead, Ronald M. Levin, Lars Noah, Katherine Rhyne, 
Jonathan Baert Wiener, ``Regulatory Improvement Legislation: Risk 
Assessment, Cost-Benefit Analysis, and Judicial Review,'' Duke 
Environmental Law and Policy Forum, Fall 2000, vol. XI (1): 132.
---------------------------------------------------------------------------

Scope of the Review
    The ``charge'' contains the instructions to the peer reviewers 
regarding the objective of the peer review and the specific advice 
sought. The importance of the information, which shapes the goal of the 
peer review, influences the charge. For instance, the goal of the 
review might be to determine the utility of a body of literature for 
drawing certain conclusions about the feasibility of a technology or 
the safety of a product. In this context, an agency might ask reviewers 
to determine the relevance of conclusions drawn in one context for 
other contexts (e.g., different exposure conditions or patient 
populations).
    The charge to the reviewers should be determined in advance of the 
selection of the reviewers. In drafting the charge, it is important to 
remember the strengths and limitations of peer review. Peer review is 
most powerful when the charge is specific and steers the reviewers to 
specific technical questions while also directing reviewers to offer a 
broad evaluation of the overall product.
    Uncertainty is inherent in science, and in many cases individual 
studies do not produce conclusive evidence. Thus, when an agency 
generates a scientific

[[Page 2669]]

assessment, it is presenting its scientific judgment about the 
accumulated evidence rather than scientific fact.\15\ Specialists 
attempt to reach a consensus by weighing the accumulated evidence. Peer 
reviewers can make an important contribution by distinguishing 
scientific facts from professional judgments. Furthermore, where 
appropriate, reviewers should be asked to provide advice on the 
reasonableness of judgments made from the scientific evidence. However, 
the charge should make clear that the reviewers are not to provide 
advice on the policy (e.g., the amount of uncertainty that is 
acceptable or the amount of precaution that should be embedded in an 
analysis). Such considerations are the purview of the government.\16\
---------------------------------------------------------------------------

    \15\ Mark R. Powell, Science at EPA: Information in the 
Regulatory Process, Resources for the Future, Washington, DC, 1999: 
139.
    \16\ Ibid.
---------------------------------------------------------------------------

    The charge should ask that peer reviewers ensure that scientific 
uncertainties are clearly identified and characterized. Since not all 
uncertainties have an equal effect on the conclusions drawn, reviewers 
should be asked to ensure that the potential implications of the 
uncertainties for the technical conclusions drawn are clear. In 
addition, peer reviewers might be asked to consider value-of-
information analyses that identify whether more research is likely to 
decrease key uncertainties.\17\ Value-of-information analysis was 
suggested for this purpose in the report of the Presidential/
Congressional Commission on Risk Assessment and Risk Management.\18\ A 
description of additional research that would appreciably influence the 
conclusions of the assessment can help an agency assess and target 
subsequent efforts.
---------------------------------------------------------------------------

    \17\ Granger Morgan and Max Henrion, ``The Value of Knowing How 
Little You Know,'' Uncertainty: A Guide to Dealing with Uncertainty 
in Quantitative Risk and Policy Analysis, Cambridge University 
Press, 1990: 307.
    \18\ Presidential/Congressional Commission on Risk Assessment 
and Risk Management, Risk Commission Report, 1997, Volume 1: 39, 
Volume 2: 91.
---------------------------------------------------------------------------

Selection of Reviewers
    Expertise. The most important factor in selecting reviewers is 
expertise: ensuring that the selected reviewer has the knowledge, 
experience, and skills necessary to perform the review. Agencies shall 
ensure that, in cases where the document being reviewed spans a variety 
of scientific disciplines or areas of technical expertise, reviewers 
who represent the necessary spectrum of knowledge are chosen. For 
instance, expertise in applied mathematics and statistics is essential 
in the review of models, thereby allowing an audit of calculations and 
claims of significance and robustness based on the numeric data.\19\ 
For some reviews, evaluation of biological plausibility is as important 
as statistical modeling. Agencies shall consider requesting that the 
public, including scientific and professional societies, nominate 
potential reviewers.
---------------------------------------------------------------------------

    \19\ William W. Lowrance, Modern Science and Human Values, 
Oxford University Press, New York, NY 1985: 86.
---------------------------------------------------------------------------

    Balance. While expertise is the primary consideration, reviewers 
should also be selected to represent a diversity of scientific 
perspectives relevant to the subject. On most controversial issues, 
there exists a range of respected scientific viewpoints regarding 
interpretation of the available literature. Inviting reviewers with 
competing views on the science may lead to a sharper, more focused peer 
review. Indeed, as a final layer of review, some organizations (e.g., 
the National Academy of Sciences) specifically recruit reviewers with 
strong opinions to test the scientific strength and balance of their 
reports. The NAS policy on committee composition and balance \20\ 
highlights important considerations associated with perspective, bias, 
and objectivity.
---------------------------------------------------------------------------

    \20\ National Academy of Sciences, ``Policy and Procedures on 
Committee Composition and Balance and Conflicts of Interest for 
Committees Used in the Development of Reports,'' May 2003: Available 
at: https://www.nationalacademies.org/coi/.
---------------------------------------------------------------------------

    Independence. In its narrowest sense, independence in a reviewer 
means that the reviewer was not involved in producing the draft 
document to be reviewed. However, for peer review of some documents, a 
broader view of independence is necessary to assure credibility of the 
process. Reviewers are generally not employed by the agency or office 
producing the document. As the National Academy of Sciences has stated, 
``external experts often can be more open, frank, and challenging to 
the status quo than internal reviewers, who may feel constrained by 
organizational concerns.'' \21\ The Carnegie Commission on Science, 
Technology, and Government notes that ``external science advisory 
boards serve a critically important function in providing regulatory 
agencies with expert advice on a range of issues.'' \22\ However, the 
choice of reviewers requires a case-by-case analysis. Reviewers 
employed by other Federal and state agencies may possess unique or 
indispensable expertise.
---------------------------------------------------------------------------

    \21\ National Research Council, Peer Review in Environmental 
Technology Development Programs: The Department of Energy's Office 
of Science and Technology, National Academy Press, Washington, DC, 
1998: 3.
    \22\ Carnegie Commission on Science, Technology, and Government, 
Risk and the Environment: Improving Regulatory Decision Making, 
Carnegie Commission, New York, 1993: 90.
---------------------------------------------------------------------------

    A related issue is whether government-funded scientists in 
universities and consulting firms have sufficient independence from the 
federal agencies that support their work to be appropriate peer 
reviewers for those agencies.\23\ This concern can be mitigated in 
situations where the scientist initiates the hypothesis to be tested or 
the method to be developed, which effectively creates a buffer between 
the scientist and the agency. When an agency awards grants through a 
competitive process that includes peer review, the agency's potential 
to influence the scientist's research is limited. As such, when a 
scientist is awarded a government research grant through an 
investigator-initiated, peer-reviewed competition, there generally 
should be no question as to that scientist's ability to offer 
independent scientific advice to the agency on other projects. This 
contrasts, for example, to a situation in which a scientist has a 
consulting or contractual arrangement with the agency or office 
sponsoring a peer review. Likewise, when the agency and a researcher 
work together (e.g., through a cooperative agreement) to design or 
implement a study, there is less independence from the agency. 
Furthermore, if a scientist has repeatedly served as a reviewer for the 
same agency, some may question whether that scientist is sufficiently 
independent from the agency to be employed as a peer reviewer on 
agency-sponsored projects.
---------------------------------------------------------------------------

    \23\ Lars Noah, ``Scientific `Republicanism': Expert Peer Review 
and the Quest for Regulatory Deliberation, Emory Law Journal, 
Atlanta, Fall 2000:1066.
---------------------------------------------------------------------------

    As the foregoing suggests, independence poses a complex set of 
questions that must be considered by agencies when peer reviewers are 
selected. In general, agencies shall make an effort to rotate peer 
review responsibilities across the available pool of qualified 
reviewers, recognizing that in some cases repeated service by the same 
reviewer is needed because of essential expertise.
    Some agencies have built entire organizations to provide 
independent scientific advice while other agencies tend to employ ad 
hoc scientific panels on specific issues. Respect for the independence 
of reviewers may be enhanced if an agency collects names of potential 
reviewers (based on considerations of expertise and reputation for 
objectivity) from the

[[Page 2670]]

public, including scientific or professional societies. The Department 
of Energy's use of the American Society of Mechanical Engineers to 
identify potential peer reviewers from a variety of different 
scientific societies provides an example of how professional societies 
can assist in the development of an independent peer review panel.\24\
---------------------------------------------------------------------------

    \24\ American Society for Mechanical Engineers, Assessment of 
Technologies Supported by the Office of Science and Technology, 
Department of Energy: Results of the Peer Review for Fiscal Year 
2002, ASME Technical Publishing, Danvers, MA, 2003.
---------------------------------------------------------------------------

    Conflict of Interest. The National Academy of Sciences defines 
``conflict of interest'' as any financial or other interest that 
conflicts with the service of an individual on the review panel because 
it could impair the individual's objectivity or could create an unfair 
competitive advantage for a person or organization.\25\ This standard 
provides a useful benchmark for agencies to consider in selecting peer 
reviewers. Agencies shall make a special effort to examine prospective 
reviewers' potential financial conflicts, including significant 
investments, consulting arrangements, employer affiliations and grants/
contracts. Financial ties of potential reviewers to regulated entities 
(e.g., businesses), other stakeholders, and regulatory agencies shall 
be scrutinized when the information being reviewed is likely to be 
relevant to regulatory policy. The inquiry into potential conflicts 
goes beyond financial investments and business relationships and 
includes work as an expert witness, consulting arrangements, honoraria 
and sources of grants and contracts. To evaluate any real or perceived 
conflicts of interest with potential reviewers and questions regarding 
the independence of reviewers, agencies are referred to federal ethics 
requirements, applicable standards issued by the Office of Government 
Ethics, and the prevailing practices of the National Academy of 
Sciences. Specifically, peer reviewers who are Federal employees 
(including special government employees) are subject to Federal 
requirements governing conflicts of interest. See, e.g., 18 U.S.C. 208; 
5 CFR part 2635 (2004). With respect to reviewers who are not Federal 
employees, agencies shall adopt or adapt the NAS policy for committee 
selection with respect to evaluating conflicts of interest.\26\ Both 
the NAS and the Federal government recognize that under certain 
circumstances some conflict may be unavoidable in order to obtain the 
necessary expertise. See, e.g., 18 U.S.C. 208(b)(3); 5 U.S.C. App. 15 
(governing NAS committees). To improve the transparency of the process, 
when an agency determines that it is necessary to use a reviewer with a 
real or perceived conflict of interest, the agency should consider 
publicly disclosing those conflicts. In such situations, the agency 
shall inform potential reviewers of such disclosure at the time they 
are recruited.
---------------------------------------------------------------------------

    \25\ National Academy of Sciences, ``Policy and Procedures on 
Committee Composition and Balance and Conflicts of Interest for 
Committees Used in the Development of Reports,'' May 2003: Available 
at: https://www.nationalacademies.org/coi/.
    \26\ Ibid.
---------------------------------------------------------------------------

Disclosure and Attribution: Anonymous Versus Identified
    Peer reviewers must have a clear understanding of how their 
comments will be conveyed to the authors of the document and to the 
public. When peer review of government reports is considered, the case 
for transparency is stronger, particularly when the report addresses an 
issue with significant ramifications for the public and private 
sectors. The public may not have confidence in the peer review process 
when the names and affiliations of the peer reviewers are unknown. 
Without access to the comments of reviewers, the public is incapable of 
determining whether the government has seriously considered the 
comments of reviewers and made appropriate revisions. Disclosure of the 
slate of reviewers and the substance of their comments can strengthen 
public confidence in the peer review process. It is common at many 
journals and research funding agencies to disclose annually the slate 
of reviewers. Moreover, the National Academy of Sciences now discloses 
the names of its peer reviewers, without disclosing the substance of 
their comments. The science advisory committees to regulatory agencies 
typically disclose at least a summary of the comments of reviewers as 
well as their names and affiliations.
    For agency-sponsored peer review conducted under Sections II and 
III, this Bulletin strikes a compromise by requiring disclosure of the 
identity of the reviewers, but not public attribution of specific 
comments to specific reviewers. The agency has considerable discretion 
in the implementation of this compromise (e.g., summarizing the views 
of reviewers as a group or disclosing individual reviewer comments 
without attribution). Whatever approach is employed, the agency must 
inform reviewers in advance of how it intends to address this issue. 
Information about a reviewer retrieved from a record filed by the 
reviewer's name or other identifier may be disclosed only as permitted 
by the conditions of disclosure enumerated in the Privacy Act, 5 U.S.C. 
552a as amended, and as interpreted in OMB implementing guidance, 40 FR 
28,948 (July 9, 1975).
Public Participation
    Public comments can be important in shaping expert deliberations. 
Agencies may decide that peer review should precede an opportunity for 
public comment to ensure that the public receives the most 
scientifically strong product (rather than one that may change 
substantially as a result of peer reviewer suggestions). However, there 
are situations in which public participation in peer review is an 
important aspect of obtaining a high-quality product through a credible 
process. Agencies, however, should avoid open-ended comment periods, 
which may delay completion of peer reviews and complicate the 
completion of the final work product.
    Public participation can take a variety of forms, including 
opportunities to provide oral comments before a peer review panel or 
requests to provide written comments to the peer reviewers. Another 
option is for agencies to publish a ``request for comment'' or other 
notice in which they solicit public comment before a panel of peer 
reviewers performs its work.
Disposition of Reviewer Comments
    A peer review is considered completed once the agency considers and 
addresses the reviewers' comments. All reviewer comments should be 
given consideration and be incorporated where relevant and valid. For 
instance, in the context of risk assessments, the National Academy of 
Sciences recommends that peer review include a written evaluation made 
available for public inspection.\27\ In cases where there is a public 
panel, the agency should plan publication of the peer review report(s) 
and the agency's response to peer reviewer comments.
---------------------------------------------------------------------------

    \27\ National Research Council, Risk Assessment in the Federal 
Government: Managing the Process, National Academy Press, 
Washington, DC, 1983.
---------------------------------------------------------------------------

    In addition, the credibility of the final scientific report is 
likely to be enhanced if the public understands how the agency 
addressed the specific concerns raised by the peer reviewers. 
Accordingly, agencies should consider preparing a written response to 
the peer review report explaining: The agency's agreement or 
disagreement, the actions the agency has undertaken or will undertake 
in response to the report, and (if applicable) the reasons the agency 
believes those actions satisfy any key

[[Page 2671]]

concerns or recommendations in the report.
Adequacy of Prior Peer Review
    In light of the broad range of information covered by Section II, 
agencies are directed to choose a peer review mechanism that is 
adequate, giving due consideration to the novelty and complexity of the 
science to be reviewed, the relevance of the information to decision 
making, the extent of prior peer reviews, and the expected benefits and 
costs of additional review.
    Publication in a refereed scientific journal may mean that adequate 
peer review has been performed. However, the intensity of peer review 
is highly variable across journals. There will be cases in which an 
agency determines that a more rigorous or transparent review process is 
necessary. For instance, an agency may determine a particular journal 
review process did not address questions (e.g., the extent of 
uncertainty inherent in a finding) that the agency determines should be 
addressed before disseminating that information. As such, prior peer 
review and publication is not by itself sufficient grounds for 
determining that no further review is necessary.

Section III: Peer Review of Highly Influential Scientific Assessments

    Whereas Section II leaves most of the considerations regarding the 
form of the peer review to the agency's discretion, Section III 
requires a more rigorous form of peer review for highly influential 
scientific assessments. The requirements of Section II of this Bulletin 
apply to Section III, but Section III has some additional requirements, 
which are discussed below. In planning a peer review under Section III, 
agencies typically will have to devote greater resources and attention 
to the issues discussed in Section II, i.e., individual versus panel 
review; timing; scope of the review; selection of reviewers; disclosure 
and attribution; public participation; and disposition of reviewer 
comments.
    A scientific assessment is considered ``highly influential'' if the 
agency or the OIRA Administrator determines that the dissemination 
could have a potential impact of more than $500 million in any one year 
on either the public or private sector or that the dissemination is 
novel, controversial, or precedent-setting, or has significant 
interagency interest. One of the ways information can exert economic 
impact is through the costs or benefits of a regulation based on the 
disseminated information. The qualitative aspect of this definition may 
be most useful in cases where it is difficult for an agency to predict 
the potential economic effect of dissemination. In the context of this 
Bulletin, it may be either the approach used in the assessment or the 
interpretation of the information itself that is novel or precedent-
setting. Peer review can be valuable in establishing the bounds of the 
scientific debate when methods or interpretations are a source of 
controversy among interested parties. If information is covered by 
Section III, an agency is required to adhere to the peer review 
procedures specified in Section III.
    Section III(2) clarifies that the principal findings, conclusions 
and recommendations in official reports of the National Academy of 
Sciences that fall under this Section are generally presumed not to 
require additional peer review. All other highly influential scientific 
assessments require a review that meets the requirements of Section III 
of this Bulletin.
    With regard to the selection of reviewers, Section III(3)(a) 
emphasizes consideration of expertise and balance. As discussed in 
Section II, expertise refers to the required knowledge, experience and 
skills required to perform the review whereas balance refers to the 
need for diversity in scientific perspective and disciplines. We 
emphasize that the term ``balance'' here refers not to balancing of 
stakeholder or political interests but rather to a broad and diverse 
representation of respected perspectives and intellectual traditions 
within the scientific community, as discussed in the NAS policy on 
committee composition and balance.\28\
---------------------------------------------------------------------------

    \28\ National Academy of Sciences, ``Policy and Procedures on 
Committee Composition and Balance and Conflicts of Interest for 
Committees Used in the Development of Reports,'' May 2003: Available 
at: https://www.nationalacademies.org/coi/.
---------------------------------------------------------------------------

    Section III(3)(b) instructs agencies to consider barring 
participation by scientists with a conflict of interest. The conflict 
of interest standards for Sections II and III of the Bulletin are 
identical. As discussed under Section II, those peer reviewers who are 
Federal employees, including Special Government Employees, are subject 
to applicable statutory and regulatory standards for Federal employees. 
For non-government employees, agencies shall adopt or adapt the NAS 
policy for committee member selection with respect to evaluating 
conflicts of interest.
    Section III(3)(c) instructs agencies to ensure that reviewers are 
independent of the agency sponsoring the review. Scientists employed by 
the sponsoring agency are not permitted to serve as reviewers for 
highly influential scientific assessments. This does not preclude 
Special Government Employees, such as academics appointed to advisory 
committees, from serving as peer reviewers. The only exception to this 
ban would be the rare situation in which a scientist from a different 
agency of a Cabinet-level department than the agency that is 
disseminating the scientific assessment has expertise, experience and 
skills that are essential but cannot be obtained elsewhere. In 
evaluating the need for this exception, agencies shall use the NAS 
criteria for assessing the appropriateness of using employees of 
sponsors (e.g., the government scientist must not have had any part in 
the development or prior review of the scientific information and must 
not hold a position of managerial or policy responsibility).
    We also considered whether a reviewer can be independent of the 
agency if that reviewer receives a substantial amount of research 
funding from the agency sponsoring the review. Research grants that 
were awarded to the scientist based on investigator-initiated, 
competitive, peer-reviewed proposals do not generally raise issues of 
independence. However, significant consulting and contractual 
relationships with the agency may raise issues of independence or 
conflict, depending upon the situation.
    Section III(3)(d) addresses concerns regarding repeated use of the 
same reviewer in multiple assessments. Such repeated use should be 
avoided unless a particular reviewer's expertise is essential. Agencies 
should rotate membership across the available pool of qualified 
reviewers. Similarly, when using standing panels of scientific 
advisors, it is suggested that the agency rotate membership among 
qualified scientists in order to obtain fresh perspectives and 
reinforce the reality and perception of independence from the agency.
    Section III(4) requires agencies to provide reviewers with 
sufficient background information, including access to key studies, 
data and models, to perform their role as peer reviewers. In this 
respect, the peer review envisioned in Section III is more rigorous 
than some forms of journal peer review, where the reviewer is often not 
provided access to underlying data or models. Reviewers shall be 
informed of applicable access, objectivity, reproducibility and other 
quality standards under Federal information quality laws.

[[Page 2672]]

    Section III(5) addresses opportunity for public participation in 
peer review, and provides that the agency shall, wherever possible, 
provide for public participation. In some cases, an assessment may be 
so sensitive that it is critical that the agency's assessment achieve a 
high level of quality before it is publicized. In those situations, a 
rigorous yet confidential peer review process may be appropriate, prior 
to public release of the assessment. If an agency decides to make a 
draft assessment publicly available at the onset of a peer review 
process, the agency shall, whenever possible, provide a vehicle for the 
public to provide written comments, make an oral presentation before 
the peer reviewers, or both. When written public comments are received, 
the agency shall ensure that peer reviewers receive copies of comments 
that address significant scientific issues with ample time to consider 
them in their review. To avoid undue delay of agency activities, the 
agency shall specify time limits for public participation throughout 
the peer review process.
    Section III(6) requires that agencies instruct reviewers to prepare 
a peer review report that describes the nature and scope of their 
review and their findings and conclusions. The report shall disclose 
the name of each peer reviewer and a brief description of his or her 
organizational affiliation, credentials and relevant experiences. The 
peer review report should either summarize the views of the group as a 
whole (including any dissenting views) or include a verbatim copy of 
the comments of the individual reviewers (with or without attribution 
of specific views to specific names). The agency shall also prepare a 
written response to the peer review report, indicating whether the 
agency agrees with the reviewers and what actions the agency has taken 
or plans to take to address the points made by reviewers. The agency is 
required to disseminate the peer review report and the agency's 
response to the report on the agency's Web site, including all the 
materials related to the peer review such as the charge statement, peer 
review report, and agency response to the review. If the scientific 
information is used to support a final rule then, where practicable, 
the peer review report shall be made available to the public with 
enough time for the public to consider the implications of the peer 
review report for the rule being considered.
    Section III(7) authorizes but does not require an agency to 
commission an entity independent of the agency to select peer reviewers 
and/or manage the peer review process in accordance with this Bulletin. 
The entity may be a scientific or professional society, a firm 
specializing in peer review, or a non-profit organization with 
experience in peer review.

Section IV: Alternative Procedures

    Peer review as described in this Bulletin is only one of many 
procedures that agencies can employ to ensure an appropriate degree of 
pre-dissemination quality of influential scientific information. For 
example, Congress has assigned the NAS a special role in advising the 
Federal government on scientific and technical issues. The procedures 
of the NAS are generally quite rigorous, and thus agencies should 
presume that major findings, conclusions, and recommendations of NAS 
reports meet the performance standards of this Bulletin.
    As an alternative to complying with Sections II and III of this 
Bulletin, an agency may instead (1) rely on scientific information 
produced by the National Academy of Sciences, (2) commission the 
National Academy of Sciences to peer review an agency draft scientific 
information product, or (3) employ