Direct Grant Programs and Definitions That Apply to Department Regulations, 49337-49356 [2013-19390]

Agencies

• DEPARTMENT OF EDUCATION

[Federal Register Volume 78, Number 156 (Tuesday, August 13, 2013)]
[Rules and Regulations]
[Pages 49337-49356]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 2013-19390]



[[Page 49337]]

Vol. 78

Tuesday,

No. 156

August 13, 2013

Part II





Department of Education





-----------------------------------------------------------------------





34 CFR Parts 75 and 77





Direct Grant Programs and Definitions That Apply to Department 
Regulations; Final Rule

Federal Register / Vol. 78 , No. 156 / Tuesday, August 13, 2013 / 
Rules and Regulations

[[Page 49338]]


-----------------------------------------------------------------------

DEPARTMENT OF EDUCATION

34 CFR Parts 75 and 77

[Docket ID ED-2012-OII-0026]
RIN 1890-AA14


Direct Grant Programs and Definitions That Apply to Department 
Regulations

AGENCY: Department of Education.

ACTION: Final regulations.

-----------------------------------------------------------------------

SUMMARY: The Secretary amends the regulations in the Education 
Department General Administrative Regulations (EDGAR) to: improve the 
Department's ability to evaluate the performance of discretionary grant 
programs and grantee projects; support, where appropriate, projects 
supported by evidence of effectiveness; review grant applications using 
selection factors that promote the Secretary's policy objectives 
related to project evaluation, sustainability, productivity, and 
strategy to scale; and reduce burden on grantees in selecting 
implementation sites, implementation partners, or evaluation service 
providers for their proposed projects. These amendments will allow the 
Department to be more effective and efficient when selecting grantees 
in discretionary grant competitions, provide higher-quality data to the 
Congress and the public, and better focus applicants on the goals and 
objectives of the programs to which they apply for grants.

DATES: These regulations are effective September 12, 2013.

FOR FURTHER INFORMATION CONTACT: Erin McHugh, U.S. Department of 
Education, 400 Maryland Avenue SW., Room 4W319, LBJ, Washington, DC 
20202. Telephone: (202) 401-1304 or by email: erin.mchugh@ed.gov.
    If you use a telecommunications device for the deaf (TDD) or a text 
telephone (TTY), call the Federal Relay Service (FRS), toll free, at 1-
800-877-8339.

SUPPLEMENTARY INFORMATION: 

Executive Summary

    Purpose of This Regulatory Action: The purpose of this action is to 
amend EDGAR to improve the quality and effectiveness of grant-making 
decisions and reduce the burden on applicants and grantees. These 
amendments will help align the Department's grant process with the 
Secretary's policy objectives and allow Department programs to design 
grant competitions to achieve those objectives. These amendments will 
also increase the flexibility for applicants and grantees to both 
reduce burden on applicants and grantees and improve the quality of 
data generated and reported by grantees. The authority to amend EDGAR 
is 20 U.S.C. 1221e-3 and 3474.
    Summary of the Major Provisions of This Regulatory Action: These 
rules:
    1. Allow the Secretary, in the application notice for a grant 
competition, to establish performance measurement requirements for 
grantees (New Sec.  75.110);
    2. Revise requirements for project evaluations submitted to the 
Department by grantees and for continuation of a multi-year project to 
incorporate performance measurement requirements for grantees (Amended 
Sec. Sec.  75.253 and 75.590);
    3. Authorize grantees to procure implementation sites without 
regard to the procurement procedures in parts 74 and 80 and use small 
purchase procedures to procure evaluation service providers and 
providers of services that are essential to the success of a proposed 
grant, provided the site or service provider is identified in the grant 
application (New Sec.  75.135);
    4. Allow the Secretary, through an announcement in the Federal 
Register, to authorize grantees under particular programs to award 
subgrants to directly carry out programmatic activities. The possible 
subgrantees and the program activities they would carry out must be 
identified and described in the grantees' applications or selected 
through a competitive process set out in subgranting procedures 
established by the grantee (New Sec.  75.708);
    5. Add one new selection criterion and amend two existing selection 
criteria that the Department may use to evaluate applications. The new 
criterion is used to assess the extent to which a proposed project 
could be brought to scale. The amendments to the general selection 
criteria also include the addition of five new factors to Sec.  
75.210(h) (Quality of the Project Evaluation) that could be used to 
assess how well a proposed project evaluation would produce evidence 
about the project's effectiveness. We also revised one factor and added 
six new factors to Sec.  75.210(c) (Quality of the Project Design) 
(Amended Sec. Sec.  75.209 and 75.210);
    6. Authorize program offices to consider the effectiveness of 
proposed projects under a new priority that could be used as an 
absolute, competitive preference, or invitational priority (New Sec.  
75.266); and
    7. Allow the Secretary to fund data collection periods after the 
end of the substantive work of a project so that project outcomes could 
be assessed using data from the entire project period (Amended 
Sec. Sec.  75.250 and 75.251).
    Costs and Benefits: The Secretary believes that these regulations 
do not impose significant costs on entities that would receive 
assistance through Department of Education programs. Any costs imposed 
on applicants by these regulations are limited to the paperwork burden 
involved in preparing an application and keeping records needed to 
track progress on meeting performance measures. The benefits of 
implementing them outweigh any costs incurred by applicants.
    The benefits of the amendments in these regulations for the use of 
performance measures, baseline data, and performance targets 
established by the Department or by grantees themselves are that the 
Department would collect meaningful data that could be used to select 
applications for funding and assess the success of individual projects. 
The Department will also use these data to report to the Congress and 
the Office of Management and Budget (OMB) on the success of the grant 
programs in achieving their legislative objectives. The Department's 
strengthened capability to evaluate the success of Department programs 
should help improve the effectiveness of those programs and improve 
transparency about how public funds are expended, without imposing 
additional costs on grantees or other parties.
    Additionally, these final regulations add a new Sec.  75.135 and 
amend Sec.  75.708 regarding subgranting and competition exceptions. 
These sections will reduce costs, increase benefits, and potentially 
improve project quality by removing barriers that impede grantees from 
working with, either through a contract or a subgrant, implementation 
partners and service providers identified in funded applications. These 
final regulations will relieve grantees of the costs of administering 
competitions without reducing accountability or increasing the risk of 
improper use of or accounting for grant expenditures.
    These regulations also provide the Department with greater 
flexibility in conducting grant competitions by allowing for the use of 
selection criteria that: (1) Are closely aligned with program 
objectives and priorities, and (2) promote policy objectives such as 
project evaluation, sustainability, productivity, and strategy to 
scale. Thus, these amendments will benefit applicants, the Department, 
and the public by allowing the Secretary to establish selection 
criteria that are concise and closely aligned with the goals and 
objectives of a particular grant competition and are focused more

[[Page 49339]]

closely and coherently on the intended programmatic and policy 
outcomes. Because the new, more specific criteria will be used instead 
of the more generic criteria currently in EDGAR, the regulations will 
generate these benefits without increasing the costs for applicants, 
grantees, or the Department.
    On December 14, 2012, the Secretary published a notice of proposed 
rulemaking (NPRM) for these amendments in the Federal Register (77 FR 
74392).
    This document includes three revisions from the NPRM. We discuss 
changes from the NPRM in greater detail in the Analysis of Comments and 
Changes. Specifically, we have revised Sec.  75.135 to make it clear 
that grantees may exercise the competition exception when procuring 
services from entities in instances where the entity is identified in 
the funded application. We have also revised the definitions of 
``strong evidence of effectiveness'' and ``moderate evidence of 
effectiveness'' in Sec.  77.1 to clarify that only studies with 
unfavorable outcomes that were so substantial as to call into question 
the potential effectiveness of the proposed project would disqualify 
the evidence from meeting the condition in the definitions. We do not 
discuss minor technical or editorial changes.
    Public Comment: In response to our invitation in the NPRM, 38 
parties submitted comments on the proposed regulations. We group major 
issues according to subject. Analysis of Comments and Changes: An 
analysis of the comments and of any changes in the regulations since 
publication of the NPRM follows.

Information Regarding Performance Measurement--Sec.  75.110

    Comments: One commenter agreed with the proposed amendment to 
create Sec.  75.110, stating that establishing performance measurement 
requirements in a notice inviting applications for a competition would 
both increase the likelihood of obtaining more robust data on grantee 
performance and increase the number of rigorous evaluation studies in 
the field.
    Some commenters agreed with the proposed amendment but requested 
clarification on key points. One commenter expressed concerns that 
performance measures beyond those related to student achievement would 
not be considered. Another commenter suggested differentiating between 
performance measures and outcomes data, indicating that performance 
measures help grantees continuously improve their projects, while 
outcomes data are useful in evaluating the success of their projects. 
The commenter also suggested developing a list of indicators for 
applicants to use when defining and adopting their own measures of 
success. Another commenter noted the importance of aligning performance 
measures with program goals and taking into account the size and scope 
of each proposed project when evaluating the quality of the performance 
measures.
    Two commenters expressed support for the proposed amendment, but 
they suggested that special considerations be made for applicants with 
limited capacity to analyze and collect data and recommended that these 
applicants be permitted to use grant funds and additional planning time 
in order to meet the performance measurement requirement.
    One commenter expanded on the idea of allowing grantees to use 
grant funds for performance measurement by suggesting the inclusion of 
a provision for performance measurement expenses in part 75, subpart F.
    Discussion: We agree with the commenter that the proposed amendment 
on performance measurement will strengthen the quality of data provided 
by grantees on their projects. Grantees typically report performance 
measures specific to their projects. Because those performance measures 
vary significantly, even among projects supported under one 
competition, it is very difficult for the Department to track the 
overall success of a program without performance measures that apply to 
all projects funded under a particular program's competition. By 
requiring standard performance measurements in a notice inviting 
applications, and by retaining the applicant's ability to set 
additional project-specific measures, we are more likely to obtain data 
that are meaningful both to evaluate the overall program and the 
quality of each grant funded under a competition for that program. This 
allows us to more effectively measure each program's effectiveness, as 
required by the Government Performance and Results Act of 1993 (GPRA), 
and will provide the Department a mechanism to hold grantees 
accountable for their performance and their success in implementing 
their grants.
    We also agree that it is important to consider performance measures 
beyond those related to student achievement. While Department programs 
share a common focus on improving academic achievement for students, 
many programs focus on factors that may not directly relate to 
students, such as professional development for teachers or engaging 
parents and family members in the school community. With this in mind, 
Sec.  75.110 gives the Department flexibility to set standard 
performance measurement requirements for all types of programs, not 
just programs that measure student performance, while continuing to 
invite applicants to set additional project-specific measures. 
Therefore, this regulation will allow the use of a variety of 
performance measures. While we explicitly require that grantees collect 
and report on GPRA-mandated performance measures, which may be focused 
on student achievement, grantees retain the discretion to establish 
additional performance measures uniquely related to the objectives of 
their proposed projects.
    We recognize that some grantees may have limited capacity to meet 
the performance measurement requirement and acknowledge that this may 
appear to disadvantage small local educational agencies (LEAs), rural 
LEAs, community colleges, and small nonprofit organizations in 
particular. We suggest that when preparing an application, an applicant 
assess its needs and develop its proposed budget accordingly. For 
example, an applicant that lacks sufficient resources to collect and 
analyze data on its own may request funding to obtain data collection 
and evaluation services from external providers. Neither current 
regulations nor these new regulations prohibit an applicant from 
including in its project budget support for data collection and 
analysis. If an applicant decides to procure these services from a 
contractor, the applicant must meet the procurement requirements 
authorized under new Sec.  75.135(b), including identifying the 
proposed contractor in the application. See discussion of Sec.  75.135 
under Procurement and Subgrant Process for Entities Named in 
Applications.
    We agree that performance measures should be aligned with the goals 
of the Department program and that useful measures will take into 
account the expected scope and size of each proposed project. Regarding 
the comment suggesting that we amend Sec.  75.110 to list specific 
program-aligned performance measures, we do not think Sec.  75.110 is 
the most appropriate platform for enumerating specific, program-aligned 
performance measures. The purpose of the amendments to Sec.  75.110 is 
to permit the Department to establish performance measures in the 
notice inviting applications and to establish standard performance 
measurement requirements that all applicants for a particular 
Department program must use, while still allowing

[[Page 49340]]

applicants the flexibility to suggest other, more project-specific, 
performance measures. Also, given the variety of programs to which 
these regulations apply, we do not think it is appropriate to prescribe 
a list of indicators in the regulation. We think that Department 
program officials are in the best position to establish appropriate 
performance measurement indicators for particular grant competitions 
and need the discretion to change the measures as the program evolves. 
In addition, more detailed information on indicators for a particular 
Department program will be provided in each notice inviting 
applications than can be provided through the use of generic 
performance measures listed in a regulation.
    Finally, we agree that performance measures and outcomes data are 
two separate terms, but we want to clarify that both are necessary and 
important to the continuous improvement and success of a grant. 
``Performance measure'' is defined as any quantitative indicator, 
statistic, or metric used to gauge program or project performance. 
Thus, a performance measure is a unit for measuring outcome data. By 
selecting the appropriate measures, we can ensure that the outcome data 
collected by grantees are relevant to program performance and that the 
Department has the data needed to report program performance 
information to the Congress under GPRA. Further, we expect that 
grantees will collect outcome data not only at the end of a project, 
but in the interim as well. Formative outcome data are collected and 
analyzed throughout the project period and are useful for the 
continuous improvement of the project, while summative outcome data are 
collected and analyzed at the end of the project period and are useful 
when evaluating the project's overall impact. Performance measures are 
expected to inform both types of outcome data.
    Changes: None.
    Comments: Some commenters expressed concern that the proposed 
changes to Sec.  75.110 would unfairly disadvantage small, rural, and 
economically disadvantaged LEAs that may have limited access to data or 
limited resources to collect data. Commenters requested that we clarify 
how performance measures, baseline data, performance targets, and 
performance data will be set, and by whom they will be set.
    Some commenters expanded on this suggestion, requesting that we 
solicit LEAs or field experts for input on defining performance 
measures, baseline data, performance targets, and performance data.
    Some commenters did not agree with the proposed change and 
expressed concern that it would prove too costly and burdensome for 
grantees. One commenter did not agree with the proposed regulation 
because, according to the commenter, the performance measurement 
requirement would be too costly and would not ultimately improve 
services for students.
    Discussion: While we recognize that all applicants may not have 
equal resources to collect and report performance measurement data 
prior to receiving a grant, each applicant should assess its capacity 
when writing its application and develop its budget proposal 
accordingly. An applicant may include funds in its project budget to 
support data collection and analysis. Applicants can use the exception 
in Sec.  75.135 to procure the needed expertise to collect the 
appropriate data and evaluate the outcomes under the measures 
established for the competition.
    Performance measures must be aligned to the goals of the program, 
which are based on the law and the Department's regulations and 
policies. As such, performance measures for a particular program are 
generally set by the Department officials responsible for the program. 
We appreciate the opinions of LEA representatives and field experts and 
encourage interested parties to comment on notices of proposed 
priorities; however, performance measures must ultimately align with 
program goals so the Department can measure the effectiveness of its 
programs.
    Gathering reliable and valid information on project outcomes is an 
integral part of determining which processes, products, strategies, and 
practices are working for students and which are not. While these final 
regulations may require grantees to use a portion of project funds on 
measuring performance, we consider it to be an important investment 
that will ultimately lead grantees to more successful results and 
thereby improve results for students and help the Department report 
more meaningful information to the Congress on the benefits of the 
Department's programs, as required under GPRA.
    Changes: None.

Procurement and Subgrant Process for Entities Named in Applications--
Sec.  75.135 Competition Exception for Implementation Sites, 
Implementation Partners, or Evaluation Service Providers; and Sec.  
75.708 Prohibition on Subgrants

    Comments: Several commenters expressed strong support for the 
proposed changes to Sec. Sec.  75.135 and 75.708. Many of these 
commenters recommended that the Secretary allow grantees to make 
subgrants without approval from the Secretary. The commenters stated 
that individual grantees are better positioned than the Secretary to 
determine whether they need to make subgrants to carry out their 
projects, what types of entities may receive subgrants, and how the 
subgrants would be made. One commenter suggested revising the 
regulation to provide that subgrants should always be allowed unless 
the Department decides to prohibit it in certain circumstances. The 
commenter thought that formulating the regulation in this manner would 
encourage public-private partnerships while preserving the Secretary's 
authority to prohibit subgranting when necessary. One commenter argued 
that providing direct authority to grantees to identify and administer 
subgrants would reduce the administrative burden of seeking approval 
from the Department. Another commenter indicated this flexibility is 
necessary to mitigate implementation delays in instances when the 
publication of the notice inviting applications in the Federal Register 
is not timely. The commenter noted that State educational agencies 
(SEAs) particularly need this flexibility and suggested adding a new 
paragraph that allows a State's Chief School Officer to determine the 
types of entities that may receive subgrants and the procedures for 
making subgrants within the State.
    Some commenters also recommended that the regulation specifically 
identify SEAs, institutions of higher education (IHEs), and nonprofit 
organizations as types of entities that may be awarded a subgrant. One 
commenter proposed adding for-profit entities as a type of entity that 
may be awarded a subgrant. The commenter noted the inclusion of for-
profit entities is particularly important considering that many grants 
are designed around a product or service that will be provided by a 
for-profit entity and emphasized that grantees should identify partners 
or providers based on the needs of their projects without consideration 
for the corporate status of a partner or provider.

[[Page 49341]]

    Discussion: We appreciate the commenters' support for the changes 
regarding subgranting. However, we decline to make the revisions 
suggested because it is prudent and necessary for the Department to 
maintain control over when Sec.  75.708 is used. The Department must 
ensure that subgrants are only authorized and used in a manner and 
under circumstances that are consistent with the requirements and 
purposes of authorizing statutes. This objective can only be met if the 
Department retains control over authorizing the grantee's use of 
subgrants. We note however, that under Sec.  75.708 the Secretary will 
indicate through an announcement in the Federal Register whether 
subgrants can be made to entities identified in an approved application 
or can be made to entities selected through a competitive process 
included in the grantee's application. Thus, in lieu of requiring a 
grantee to identify all entities that will receive subgrants in the 
approved application, the Secretary may allow a grantee to use a 
competitive process that it describes in the approved application to 
determine the entities that will receive subgrants.
    With regard to the comment suggesting that for-profit entities be 
allowed to receive subgrants, we note that grantees already have the 
authority to enter into contracts with for-profit entities. 
Additionally, we think that procurements are the appropriate vehicle 
for grantees to use to secure goods and services from for-profit 
entities. For that reason, we decline to revise the regulations to 
allow subgrants to for-profit entities. However, we agree that there 
may be circumstances under which a product or service provided by a 
for-profit entity is integral to implementation of a project. As a 
result, we have revised Sec.  75.135 to include entities that will 
provide a product or service that would, if removed from the grant, 
have a detrimental effect on the successful implementation of the 
grant.
    Changes: We have revised Sec.  75.135(b) to clarify that when 
entering into a contract for data collection, data analysis, evaluation 
services, or essential services, as defined in paragraph (f) of this 
section, an applicant may select a provider using the informal, small-
purchase procurement procedures in 34 CFR 80.36(d)(1), regardless of 
whether that applicant would otherwise be subject to that part or 
whether the evaluation contract would meet the standards for a small 
purchase order, if--
    (1) The contract is with the data collection, data analysis, 
evaluation service, or essential service provider;
    (2) The data collection, data analysis, evaluation service, or 
essential service provider that the applicant proposes to use is 
identified in the application for the grant; and
    (3) The data collection, data analysis, evaluation service, or 
essential service provider is identified in the application in order to 
meet a statutory, regulatory, or priority requirement related to the 
competition.
    We have also added paragraph (f) to state that, for the purposes of 
this section, essential service means a product or service directly 
related to the grant that would, if not provided, have a detrimental 
effect on the grant.
    Comment: One commenter recommended that the Department broadly 
implement the authority to allow subgrants. The commenter suggested 
that the Ready To Learn (RTL) program is particularly well-suited for 
the use of subgrants given that subrecipients of RTL grantees are often 
responsible for the development and production of educational 
programming that is integral to the grant. According to the commenter, 
subgranting will both continue to ensure close monitoring of funds and 
foster close collaboration that will further project objectives.
    Discussion: As noted elsewhere in our responses to comments in this 
section, the Department cannot establish a universal rule allowing the 
use of subgrants because program statutes define differently whether 
subgrants may be used and in what circumstances they may be used. 
Additionally, subgrants, even when not prohibited by a statute, may not 
always be appropriate for a particular program.
    Changes: None.
    Comments: Two commenters stated that, although the proposed 
amendments in Sec.  75.135 that permit exceptions to the procurement 
procedures are beneficial and they support them as written, some 
grantees may be subject to State or local laws that require specific 
procurement procedures. One commenter explained that such State laws 
and requirements negate the benefits of the proposed amendment and 
suggested revising the language to minimize this consequence.
    Discussion: The proposed amendments in Sec.  75.135 relax 
requirements that otherwise apply to grantees under parts 74 and 80. 
They do not, as the commenters noted, eliminate a grantee's 
responsibilities to comply with their own procurement requirements and 
State and local laws that exceed those required by Department 
regulations. State governments may follow their own procurement 
requirements, subject only to the requirement that they must include in 
their contracts all clauses required by Federal statutes, Executive 
orders, and implementing regulations. We note that some SEAs have 
adopted some of the requirements in Sec.  80.36. Other State, local, 
and Indian tribal government grantees must comply with the minimum 
requirements in 34 CFR 80.36. Non-governmental grantees must comply 
with the minimum procurement requirements in 34 CFR 74.41-74.48. These 
final regulations do not change other applicable financial management 
and procurement requirements in 34 CFR parts 74 and 80, including those 
that require State agencies to follow their own procurement policies 
and procedures (34 CFR 80.36(a)) or that generally require grantees to 
maintain procurement procedures that prohibit conflicts of interest. 
The continued applicability of these requirements in parts 74 and 80 of 
EDGAR is crucial to ensuring accountability for the use of Federal 
funds by grantees.
    Changes: None.
    Comments: Many commenters expressed strong support for the 
revisions to Sec.  75.135 and Sec.  75.708 and agreed that the 
exemption from procurement requirements for selecting implementation 
sites or partner entities, and the use of small purchase procedures to 
select evaluation service providers identified in grant applications 
will improve the implementation and outcome of grants funded by the 
Department. One commenter noted that the proposed amendment would be 
particularly beneficial to SEAs because it will support more efficient 
use of resources and ensure grant activities are implemented on a 
timely basis.
    One commenter expressed general support for the proposed amendment 
but recommended revising it to include a competition exception for 
products or services identified in the application that are unique and 
essential, meaning that the use of an alternative product or service 
would be detrimental to the implementation of the project. The 
commenter also suggested adding a condition that allows the use of 
simplified procurement procedures for products or services that are not 
core to the implementation of the project when the costs of conducting 
a competition would be excessive in relation to the amount of grant 
funds that would be awarded in the contract. The commenter also stated 
that local and State procurement requirements would still apply, so 
these revisions would eliminate an additional Federal requirement in 
these instances but

[[Page 49342]]

would not remove those existing protections.
    Discussion: We appreciate the commenters' support for the proposed 
amendment to Sec.  75.135 and agree it will be beneficial to grantees. 
With regard to the recommendation that we revise the regulation to 
allow grantees to use the competition exception for products and 
services identified in the application that are unique, we recognize 
that grant projects may be designed around such products or services. 
Therefore, we have revised paragraph (b) of Sec.  75.135 to include an 
exception for entities that will provide a product or service that 
would, if not provided, have a detrimental effect on the grant. 
However, we decline to revise the regulation to reduce the competition 
requirements for products or services that are not identified in the 
application or core to the implementation of the project because we do 
not think such a revision is consistent with the intent of the change. 
We do not intend this change to limit competition in instances when 
full and open competition is practical. We note, however, that the 
simplified acquisition threshold already provides grantees some 
flexibility in competition requirements for procurements under 
$100,000. OMB has proposed to raise this threshold to $150,000 in its 
proposed amendments to title 2 of the CFR. See the OMB proposal, Reform 
of Federal Policies Relating to Grants and Cooperative Agreements; Cost 
Principles and Administrative Requirements (Including Single Audit 
Act), published on Friday, February 1, 2013, at 78 FR 7282. Thus, 
regardless of the exemption authorized in these final regulations, 
applicants will have greater flexibility to use small purchase 
procedures when the procurement threshold is raised. Therefore, it is 
not necessary or appropriate to make the changes suggested.
    Also, as noted earlier, these regulations relax certain procurement 
requirements that otherwise apply to grantees under parts 74 and 80. 
Grantees should be aware, however, that these amendments do not 
eliminate a grantee's responsibilities to comply with its own 
procurement requirements and State and local laws to the extent that 
those requirements and laws exceed the minimum requirement in parts 74 
and 80.
    Changes: We have revised Sec.  75.135(b) to clarify that when 
entering into a contract for data collection, data analysis, evaluation 
services, or essential services, as defined in paragraph (f) of this 
section, an applicant may select a provider using the informal, small-
purchase procurement procedures in 34 CFR 80.36(d)(1), regardless of 
whether that applicant would otherwise be subject to that part or 
whether the evaluation contract would meet the standards for a small 
purchase order, if--
    (1) The contract is with the data collection, data analysis, 
evaluation service, or essential service provider;
    (2) The data collection, data analysis, evaluation service, or 
essential service provider that the applicant proposes to use is 
identified in the application for the grant; and
    (3) The data collection, data analysis, evaluation service, or 
essential service provider is identified in the application in order to 
meet a statutory, regulatory, or priority requirement related to the 
competition.
    We have also added paragraph (f) to state that, for the purposes of 
this section, essential service means a product or service directly 
related to the grant that would, if not provided, have a detrimental 
effect on the grant.
    Comment: None.
    Discussion: Based on the comments received, we revised Sec.  
75.135(b) to expand the circumstances under which grantees may use 
small purchase procedures. We have also revised Sec.  75.135(c) to 
require applicants, who utilize this exception, to certify that they 
followed the small purchase procedures. The small purchase procedures, 
while not as extensive as the full procurement requirements set out in 
Parts 74 and 80, provide important protections to the Federal interest 
in the prudent and allowable use of grant funds. By requiring 
applicants that utilize this exception to certify that they followed 
the small purchase procedures, we provide increased assurance that the 
protections to Federal grant funds offered by those procedures are, in 
fact, in place.
    Changes: We have revised the certification requirement in Sec.  
75.135(c) to require grantees that relied on the exceptions of Sec.  
75.135(b) to certify that they used small purchase procedures to obtain 
a product or service if the applicant relied on the exception 
authorized in this section to procure the product or service.

Amendments Relating to Evidence--Sec.  75.210(c) Quality of the Project 
Design, Sec.  75.210(h) Quality of the Evaluation, Sec.  75.266 
Consideration for Applications Supported by Strong or Moderate 
Evidence, Sec.  75.590 Evaluation by the Grantee, and Sec.  77.1 
Definitions That Apply to All Department Programs

    Comments: Many commenters strongly supported the proposed 
definitions of ``strong evidence of effectiveness'' and ``moderate 
evidence of effectiveness'' in Sec.  77.1(c). One commenter applauded 
the Department for expanding the focus on evidence-based practices and 
stated that this effort will result in higher quality grant 
applications and outcomes. One commenter noted that the strength of 
evidence used to support a project's effectiveness should be rigorous, 
objective, and pertinent to the goals of the project.
    A few of these commenters suggested amendments or clarifications to 
these definitions. One commenter recommended clarifying that a study, 
in order to meet these definitions, need only meet the appropriate 
standards outlined in the What Works Clearinghouse (WWC) and not 
necessarily be reviewed by the WWC or posted on the WWC Web site. Three 
commenters noted that these definitions, as proposed, present a risk 
that a study could meet the definition even if the effects are: (1) On 
trivial or developer-created outcomes; (2) artificially inflated or 
likely a result of chance; or (3) so small in size as to be of little 
importance. These commenters recommended revising the definitions to 
clarify that the study must be of sufficient duration and sample size 
to represent a valid test and to require that the study find a 
significant favorable outcome based on a measure of clear policy 
importance. One of these commenters further suggested requiring that 
the study have a substantial and important effect on improving student 
achievement or student growth, closing achievement gaps, decreasing 
dropout rates, increasing high school graduation rates, or increasing 
college enrollment and competition rates. The commenters stated that 
such changes avoid the loophole of classifying programs that lack 
policy or practical importance as evidence-based programs.
    Discussion: We agree with the commenters that the evidence used to 
support a project's effectiveness should be objective and pertinent to 
the goals of the project. However, we also recognize that at the 
various stages of a proposed project's development,

[[Page 49343]]

different types of evidence are available to assess the effectiveness 
of a project. That is why we include definitions for four levels of 
evidence: ``Strong evidence of effectiveness,'' ``moderate evidence of 
effectiveness,'' ``evidence of promise,'' and ``strong theory.'' We 
establish these definitions in order to develop an understanding for 
applicants of what is required to meet each level of evidence. 
Combined, these four levels of evidence allow the Department to support 
effective projects (strong or moderate evidence of effectiveness) or 
projects that have a high potential to be effective (evidence of 
promise or strong theory).
    We appreciate the commenters' concerns regarding the definitions of 
``strong evidence of effectiveness'' and ``moderate evidence of 
effectiveness.'' However, we do not consider the proposed changes 
necessary as the definitions already safeguard against the risks 
identified by the commenters.
    With regard to ensuring the outcome has policy and practical 
importance, both of these definitions refer to the effect on a 
``relevant outcome.'' The definition of ``relevant outcome'' explains 
that it is the ultimate outcome of the proposed process, product, 
strategy, or practice and should be consistent with the specific goals 
of the Department program. Thus, the references to ``relevant outcome'' 
would not allow studies that only show effects on trivial outcomes to 
meet the requirements of the definitions.
    Further, the WWC Procedures and Standards Handbook \1\ explains 
that a study that does not include a valid or reliable outcome measure, 
or does not provide adequate information to determine whether it uses 
an outcome that is valid or reliable, would not meet WWC Evidence 
Standards. Because the WWC Evidence Standards are incorporated in these 
definitions, a study that only includes an outcome measure created by 
the evaluator or developer with weak or no validity or reliability data 
would not meet the requirements of the definitions.
---------------------------------------------------------------------------

    \1\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), currently found at the following 
link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    With regard to requiring that a study has an adequate sample size 
to ensure that the effect is not artificially inflated, a result of 
chance, or so small it is of little importance, both definitions refer 
to the definitions of a ``large sample'' and a ``multi-site sample.'' 
In order for any study to meet the requirements of ``strong evidence of 
effectiveness,'' that study would need to have used a ``large sample'' 
and a ``multi-site sample.'' Under the definition of ``moderate 
evidence of effectiveness,'' a study must meet, among other 
requirements, one of the following two conditions: (1) WWC Evidence 
Standards without reservations; or (2) WWC Evidence Standards with 
reservations.\2\ Although a small study that meets WWC Evidence 
Standards ``without reservations'' would meet one of these two 
conditions, a study that meets WWC Evidence Standards ``with 
reservations'' could only meet the requirements of ``moderate evidence 
of effectiveness'' if it used a ``large sample'' and a ``multi-site 
sample.'' We think the inclusion of ``large sample'' and ``multi-site 
sample'' in the definitions of ``strong evidence of effectiveness'' and 
``moderate evidence of effectiveness'' appropriately mitigate the risks 
identified by the commenters.
---------------------------------------------------------------------------

    \2\ See definition for full description of the two possible 
conditions.
---------------------------------------------------------------------------

    With regard to ensuring that studies are of sufficient duration to 
meet the requirements, we note that the WWC Evidence Standards do not 
require a minimum study length. More importantly, because it is not 
clear that requiring a minimum study length is appropriate or 
necessary, we decline to revise the definitions to include such a 
requirement.
    Changes: None.
    Comment: None.
    Discussion: In Sec.  77.1(c), we have revised the definitions of 
``moderate evidence of effectiveness'' and ``strong evidence of 
effectiveness'' by adding the phrase ``and overriding'' to the second 
parenthetical in paragraphs (a) and (b) of both definitions. We add 
this phrase to clarify that only studies with unfavorable outcomes that 
were so substantial as to call into question the potential 
effectiveness of the proposed project would disqualify the evidence 
from meeting the condition in the definitions.
    Changes: We have revised the second parenthetical in paragraphs (a) 
and (b) of the definitions of ``moderate evidence of effectiveness'' 
and ``strong evidence of effectiveness'' to add the phrase ``and 
overriding.'' The parenthetical now reads ``with no statistically 
significant and overriding unfavorable impacts on that outcome for 
relevant populations in the study or in other studies of the 
intervention reviewed by and reported on by the What Works 
Clearinghouse.''
    Comments: One commenter expressed concern that the proposed 
definitions related to evidence would stifle innovation and that 
providing special consideration for projects supported by evidence of 
effectiveness would limit the pool of applications for a competition. 
Another commenter stated that such consideration is not appropriate for 
all programs and the proposed amendment appears to be an attempt to 
turn all projects funded by the Department into Investing in Innovation 
(i3) projects.
    Discussion: The establishment of procedures to provide special 
consideration for projects supported by strong or moderate evidence of 
effectiveness provides the Secretary a mechanism to support effective 
projects and offer incentives to the field for building an evidence 
base on the effectiveness of the processes, products, strategies, and 
practices that are, or will be used, in education. However, as noted in 
our response to other comments in this discussion, we recognize that 
different types of evidence are available at the various stages of a 
proposed project's development and that there are some areas where 
strong or moderate evidence of effectiveness is not yet available. As 
such, we agree that it would not be appropriate for the Secretary to 
consider whether a project is supported by strong or moderate evidence 
of effectiveness for all Department programs. The Secretary will only 
provide special consideration for projects supported by strong or 
moderate evidence of effectiveness in programs where such evidence 
exists or where such incentives are meaningful. When such levels of 
evidence do not exist, Department program officials may consider 
whether using ``evidence of promise'' or ``strong theory'' would be 
more appropriate for spurring innovation. Thus, we do not think 
providing special consideration in program areas that do have these 
levels of evidence would preclude robust competition or stifle 
innovation.
    Changes: None.
    Comments: Three commenters expressed support for the establishment 
of procedures to provide special consideration for projects supported 
by strong or moderate evidence of effectiveness. However, these 
commenters suggested clarifying that the special consideration be given 
to both existing projects supported by strong or moderate evidence of 
effectiveness and new projects that are proposing to adopt or adapt 
models supported by strong or moderate evidence of effectiveness.
    Discussion: The definitions of ``strong evidence of effectiveness'' 
and ``moderate evidence of effectiveness'' in Sec.  77.1(c) indicate 
that the study needs to be of the effectiveness of the proposed 
process, product, strategy, or practice.

[[Page 49344]]

These definitions also clarify that the study must overlap with the 
populations and settings in the proposed project. Therefore, a new 
project that is adopting the model of the process, product, strategy, 
or practice in the study meets the definitions. An applicant proposing 
a new process, product, strategy, or practice that is adapting or 
changing the model from what was in the study would need to explain how 
the study supports the adapted version of the model. Thus, a study may 
be used to support an adaptation of the model in the study so long as 
the applicant can provide a justification that the proposed project's 
efficacy necessitates the adaptation, and is based upon the evidence 
and theory supported by the original study.
    Given the variety of programs to which these regulations apply, we 
do not think it is appropriate for the Department to determine at what 
single point an adaptation would make the study no longer credible for 
supporting the effectiveness of the proposed project. However, any 
programs providing special consideration for projects supported by 
strong or moderate evidence of effectiveness would provide instructions 
to applicants on the information they need to submit to demonstrate 
that they meet the applicable evidence level.
    Changes: None.
    Comment: One commenter recommended providing special consideration 
only for projects supported by strong or moderate evidence of 
effectiveness through the establishment of a separate competition, as 
opposed to ``an across the board competitive preference.''
    Discussion: Section 75.266 authorizes the Secretary to establish a 
separate competition or provide a competitive preference for 
applications supported by strong or moderate evidence of effectiveness. 
We decline to limit the Secretary to providing special consideration 
through a separate competition because that process may not be 
appropriate for all Department programs. Given the variety of programs 
to which these regulations apply, it is important that we provide 
sufficient flexibility for determining which programs require, and how 
the Secretary would consider, strong or moderate evidence of 
effectiveness.
    Change: None.
    Comment: One commenter noted that projects funded by the Department 
should produce evaluations that meet defined standards but questioned 
whether the WWC Evidence Standards were appropriate considering the 
burden associated with conducting evaluations that are designed to meet 
those standards. Specifically, the commenter expressed concern that 
small or rural LEAs would not have the capacity to conduct such 
evaluations and that the Department's use of selection factors 
promoting WWC Evidence Standards would favor large research 
organizations over LEAs. The commenter further stated that it is 
contradictory for the Department to use selection factors that promote 
evaluations more rigorous than required by the program. To address 
these concerns, the commenter recommended revising Sec.  
75.210(h)(2)(viii)-(x) to require that the proposed project evaluation 
meets the next level higher or equivalent level of the prior evidence 
supporting the proposed project's effectiveness.
    Discussion: The WWC is an initiative of the Department's Institute 
of Education Sciences (IES) and serves as a central and trusted source 
of scientific evidence for what works in education. Although we 
recognize the WWC Evidence Standards primarily refer to randomized 
controlled trial (RCT) and quasi-experimental design (QED) studies, we 
also note that these designs are the most rigorous and defensible 
methods for producing unbiased evidence of project effectiveness.
    We agree with the commenter that conducting project evaluations 
that are designed to meet the WWC Evidence Standards requires planning 
and resources. However, because an applicant may obtain an evaluation 
service provider to conduct the project evaluation through a contract 
and may include these activities and costs in its proposed project 
budget, the use of the factors in Sec.  75.210(h)(2)(viii)-(x) would 
not inherently disadvantage an applicant that lacks the internal 
capacity to conduct such evaluations. We also note that Sec.  75.210(h) 
(Quality of the Project Evaluation) is only one criterion among 
multiple criteria used to evaluate applications. The Department 
considers each program's purpose, goals, and applicant pool when 
deciding which selection criteria and factors to use in a given year's 
competition. By creating these factors under Sec.  75.210(h) (Quality 
of the Project Evaluation), the Department has the option--not the 
obligation--to use them to encourage applicants to propose project 
evaluations that would meet WWC Evidence Standards. Consequently, the 
Department will use these factors when appropriate for a particular 
competition and will not use them when doing so would conflict with 
required program evaluations.
    We decline to replace these factors with a factor that would allow 
a proposed project evaluation to be the equivalent level of the prior 
evidence supporting the proposed project's effectiveness. In general, 
to provide the public the greatest return on its investment, 
evaluations funded by the Department should build on prior research, as 
appropriate. Although we recognize the importance of replicating 
results of a past study, we think it is important for applicants to 
propose project evaluations that increase the level of evidence of the 
proposed project's effectiveness, as appropriate. By providing the 
flexibility to select among the various factors under Sec.  75.210(h) 
(Quality of the Project Evaluation), the Department has the discretion 
to select factors that are appropriate for the areas of study and 
research goals for a particular program.
    Changes: None.
    Comments: Two commenters indicated they had no objections to the 
proposed changes to Sec. Sec.  75.210, 75.266, and 77.1 regarding 
evidence of effectiveness and WWC Evidence Standards, but cautioned the 
Department to be prudent in their use in discretionary grant 
competitions. One commenter stated that lack of evidence should not be 
the sole rationale for deciding not to make a grant to a particular 
applicant and suggested that the new regulations should not be used to 
establish a high threshold for evidence of effectiveness in areas where 
the amount of evidence on existing practice is not strong, particularly 
in areas that are difficult to measure, such as school climate or 
efforts to reduce administrative burden or build capacity. Similarly, 
another commenter recommended that programs establish thresholds for 
evidence of effectiveness that are commensurate with the quantity and 
quality of existing evidence in the field.
    Discussion: We agree that the new regulations in Sec. Sec.  75.210, 
75.266, and 77.1 regarding evidence of effectiveness should only be 
used when appropriate for a particular program. We are making changes 
to these regulations to support evidence-based grant making in areas 
where evidence exists and to provide incentives and opportunities to 
build the body of evidence of effectiveness in education.
    Changes: None.
    Comments: Several commenters noted the distinction between a 
``project'' and a ``strategy within a project.'' The commenters stated 
that it might be more appropriate for the Department to evaluate the 
effectiveness of an individual strategy used by a grantee

[[Page 49345]]

rather than attempt to measure the effectiveness of a project as a 
whole. The commenters suggested revising the proposed selection 
criteria to clarify that programs or strategies could be used when 
evaluating a project's effectiveness.
    Discussion: An applicant may propose to evaluate different 
strategies within a project using different evaluation design methods. 
For example, an applicant may propose a pre-post analysis to assess 
progress of one strategy within its project and a more rigorous 
evaluation design for another strategy within its project. Despite the 
flexibility we allow an applicant in designing the proposed project 
evaluation, under Sec.  75.590 (Evaluation by the grantee), the entire 
project being supported by Federal funds must be evaluated.
    Changes: None.
    Comments: A few commenters stated that they could not support the 
inclusion of selection factors that consider evidence of effectiveness 
because it would competitively disadvantage certain types of 
applicants, including those with limited resources or those that serve 
student populations that have unique needs.
    Two commenters specifically suggested that the consideration of 
effectiveness would present a disadvantage to community colleges. One 
commenter discussed three challenges for community colleges that make 
it difficult for them to conduct rigorous evaluations. According to the 
commenter, rigorous evaluation designs (1) distract from community 
colleges' missions to provide access to education for all students; (2) 
often require approval of an Institutional Review Board (IRB), and many 
community colleges have not established IRBs; and (3) require signed 
consent from participants, which creates additional complications and 
concerns regarding student access to educational programs or support 
services. Both commenters stated that most studies posted on the WWC 
Web site focus on K-12 education and that existing research around 
community colleges is insufficient for them to compete if factors 
related to evidence of effectiveness are used by the Department. To 
address this concern, one commenter recommended creating a special 
track of priority funding for empirical research on community colleges 
in all of the Department's postsecondary programs.
    One commenter suggested modifying the new selection factors to 
exempt programs with ``historical evidence of benefit to students.'' 
The commenter stated that programs that equalize educational 
opportunity among low-income, first-generation college students who, in 
large part, are from underrepresented groups, should not be required to 
use quantitative research to determine their effectiveness.
    One commenter stated that programs providing parental training and 
engagement services would be disadvantaged by selection factors related 
to evidence of effectiveness because such programs require a focus on 
individual parent and family needs. The commenter expressed concern 
that the use of these selection factors, or any special consideration 
given for evidence of effectiveness, would limit which entities could 
apply to a particular program without providing a clear benefit to 
children and their families. Another commenter suggested that the 
selection factors referring to ``evidence of promise'' and ``strong 
theory'' be the only selection factors related to evidence of 
effectiveness used for implementation-based grants.
    Discussion: As noted elsewhere in our response to comments in this 
discussion, we agree that the selection factors relating to evidence of 
effectiveness, whether they fall under Sec.  75.210(c) (Quality of the 
Project Design) or Sec.  75.210(h) (Quality of the Project Evaluation), 
should only be used when appropriate for a particular program. We 
include these selection factors to support evidence-based grant making 
in areas where evidence exists and to provide incentives and 
opportunities to build the body of evidence of effectiveness in 
education.
    Because the Department has the discretion to select factors that 
are appropriate for the areas of study and research goals of a 
particular program, and therefore would not select factors that would 
require applicants to provide evidence of effectiveness in areas that 
have not been widely researched, we decline to remove these factors. 
Additionally, we do not think the amount of research for a particular 
area of education on the WWC Web site is a reason not to add these 
factors to the regulations. A study does not need to be reviewed by the 
WWC or posted on the WWC Web site to meet the WWC Evidence Standards. 
Department program officials could use research available on the WWC 
Web site or from other sources to inform their decision on whether 
these selection factors are appropriate for the particular program.
    Further, we note that these factors address evidence of 
effectiveness and evaluation of effectiveness at various levels. Two of 
the factors refer to grantees proposing evaluation designs that meet 
WWC Evidence Standards with or without reservations, but we also 
include two other factors that refer to ``evidence of promise'' and 
``strong theory.'' Including four levels of evidence provides the 
Department the opportunity to consider the level of evidence available 
in the field for the types of projects to be funded by the relevant 
program and the capacity of potential applicants to design evaluations 
that would assess the effectiveness of a project at these different 
levels.
    With regard to the other issues raised by the commenters, we 
recognize that rigorous evaluation designs require grantees to compare 
individuals participating in the project to those who are not 
participating in the evaluation. However, requiring more rigorous 
evaluation designs does not contradict the educational mission of 
serving all students because evaluating the effectiveness of a 
particular intervention is necessary to understand which interventions 
effectively improve student outcomes. Although funds that support 
evaluation services cannot also support direct services to students, 
investing in evaluation allows entities to focus finite resources on 
only those processes, products, strategies, or practices that are most 
effective in improving student outcomes. Therefore, we do not think 
evaluating the effectiveness of a project using a rigorous design would 
impede an entity from carrying out its educational mission. Further, 
because an applicant may contract to obtain an evaluation service 
provider that has access to an IRB to conduct the project evaluation, 
and because the applicant may include these activities or activities 
related to accessing an independent IRB or establishing its own IRB to 
support the project evaluation and their costs in its proposed budget, 
we do not think applicants that lack their own IRBs are disadvantaged. 
Similarly, because under the Common Rule for the Protection of Human 
Subjects, an IRB can modify or waive requirements for written consent, 
and the costs for activities to obtain written consent from 
participants may be included in the proposed budget, we do not think a 
specific type of entity is inherently disadvantaged by the use of 
selection factors that encourage applicants to propose rigorous 
evaluations of their projects' effectiveness.
    With regard to the comments recommending that these factors not be 
used for programs that historically have benefited students and that we 
only allow the use of ``evidence of promise'' and ``strong theory'' for 
implementation grants, we reiterate the importance of the Department 
supporting the

[[Page 49346]]

improvement of information available to practitioners and policymakers 
about which practices work, for which types of students, and in which 
contexts. These selection factors support that goal by providing 
incentives to applicants for grants to build an evidence base on the 
effectiveness of the processes, products, strategies, and practices 
that are, or will be used, in education.
    Changes: None.
    Comments: A few commenters stated that the Department should not 
support evaluations using research designs that include control groups. 
Two commenters stated that experimental designs are unethical because 
they require grantees to withhold treatment from students in public 
education. Another commenter felt that it was inappropriate to deny 
services to students simply to accommodate research and evaluation, 
particularly when the Congress authorized and funded the program to 
provide services to students. One commenter further expressed concern 
that favoring such designs would provide an advantage to large research 
organizations over LEAs that lack the capacity to conduct such 
evaluations.
    In addition to concerns about placing students or teachers in 
``experimental'' versus ``control'' groups, one commenter cited 
challenges regarding the use of RCTs. Specifically, the commenter noted 
that a pure control condition is rare because fidelity of 
implementation can significantly impact the effectiveness of a process, 
product, strategy, or practice. The commenter indicated that how well a 
particular process, product, strategy, or practice works depends on the 
conditions under which it is implemented, and the costs of observation 
and metrics to determine the fidelity of that implementation are 
significant. The commenter also noted that, because products and 
services are constantly changing and improving, the products or 
services are frequently no longer available in the format or version 
that was studied by the time an evaluation is complete. The commenter 
concluded that equal consideration of alternative study designs, such 
as regression discontinuity designs, is needed to ensure the Department 
does not limit the development, implementation, and evaluation of 
innovative projects.
    One commenter expressed concern that the selection factors that 
refer to ``evidence of promise'' and ``strong theory'' would be used to 
make all implementation projects into randomized research projects with 
control groups. According to the commenter, these selection factors 
would skew successful applications toward projects conducting research 
studies and away from projects providing services to students and 
teachers. The commenter stated that to require such project evaluations 
is not consistent with legislative intent and would not result in 
improved student outcomes. Another commenter made a similar statement 
that the proposed amendments regarding evidence of effectiveness and 
evaluation should not be used for the TRIO programs because they would 
undermine the intent of the Higher Education Opportunity Act of 2008.
    Discussion: We appreciate the commenters' concerns about whether it 
is ethical to evaluate the effectiveness of a project using a 
randomized experimental design. In order to ensure ethical research, 
the Department, under the Common Rule for the Protection of Human 
Subjects in Research (34 CFR 97), does not permit covered human 
subjects research to be initiated until it has been reviewed by an IRB 
and receives the Department's protection of human subjects clearance. 
Although we recognize that conducting these types of evaluations can be 
difficult, we also recognize that random assignment of entities 
(students, teachers, schools, or other units of analysis) to a 
treatment or control group is the most effective way to eliminate 
plausible competing explanations for observed differences between 
treated and non-treated individuals or groups (i.e., the estimated 
treatment effect). Two common strategies used in randomized experiments 
in education that are designed to address this ethical concern are the 
use of a ``wait-list'' control group and the assignment of schools, 
rather than students, to the treatment group. Despite the challenges in 
conducting such evaluations, we consider it important to provide a 
mechanism for the Department, when appropriate, to use these selection 
factors to encourage grantees to conduct evaluations of their projects 
that will improve the information available to practitioners and 
policymakers about which processes, products, strategies, and practices 
work, for which types of students, and in which contexts.
    We disagree that the regulation would favor large research 
institutions over other types of applicants. Applicants that do not 
feel they possess adequate resources to carry out a rigorous evaluation 
of their proposed projects may contract with an evaluation service 
provider. Applicants can use the exception in Sec.  75.135 to procure 
the needed expertise to implement a rigorous evaluation.
    We disagree with the comment that the use of selection factors 
referring to ``evidence of promise'' and ``strong theory'' requires all 
implementation projects to become randomized research projects with 
control groups. We define ``strong theory'' to mean a rationale for the 
proposed process, product, strategy, or practice that includes a 
``logic model'' (as defined in Sec.  77.1(c)). The development of a 
logic model and the associated rationale does not require a grantee to 
conduct a randomized experiment. Similarly, although a grantee may use 
a QED or RCT to meet the ``evidence of promise'' definition, a grantee 
could also use a correlational study with statistical controls. Thus, 
neither evidence level requires the use of a treatment and control 
group. See Sec.  77.1(c) for definitions of these terms.
    Although we appreciate that products and services change over time 
and may no longer be available in the format or version that was 
studied by the time an evaluation is complete, these selection factors 
are intended to provide incentives to applicants for grants to build an 
evidence base on the effectiveness of the processes, products, 
strategies, and practices that are, or will be used, in education. As 
noted elsewhere in our discussion of comments related to evidence of 
effectiveness, an applicant may use a study of a product or service to 
support an adaptation of it so long as the applicant can provide a 
justification that the proposed project's efficacy necessitates the 
adaptation, and is based upon the evidence and theory supported by the 
original study. This same concept applies to the potential for learning 
from a project evaluation of a product or service that may no longer be 
available in the format or version that was studied by the time an 
evaluation is complete. Moreover, the selection factor regarding 
``evidence of promise'' does allow consideration of alternative study 
designs.
    With regard to commenters' concerns about the Department requiring 
the use of evaluation designs that are in conflict with a program's 
statute, the Department does not propose or require grantees to use 
grant funds in a manner that is prohibited by statute. As noted 
elsewhere in our responses to comments in this section, the Department 
has discretion in determining which selection factors are most 
appropriate for a particular program's purpose and goals. Therefore, 
the Department would not use a selection factor that is in conflict 
with a program's governing statute, purpose, or goals.
    Changes: None.

[[Page 49347]]

    Comments: Several commenters recommended including a selection 
factor under Sec.  75.210(h) (Quality of the Project Evaluation) to 
promote evaluative methods for small service delivery programs. 
Specifically, the commenters requested the addition of a selection 
factor for studies that assess or use best practices for service 
delivery strategies using small-scale pilots.
    Discussion: We agree that a selection factor that encourages 
project evaluations of pilot strategies that may be best practices is 
of value, and we have included Sec.  75.210(h)(2)(xii) for this 
purpose. This selection factor supports project evaluations that 
clearly articulate the key components and outcomes of the grant-
supported process, product, strategy, or practice, as well as the 
measurable threshold for acceptable implementation. Implementation 
studies that articulate the key components of the proposed project and 
the measureable threshold for acceptable implementation of the key 
components are necessary to disseminate information about and replicate 
best practices. These studies also could be used to evaluate a pilot of 
service delivery strategy. Because Sec.  75.210(h) (Quality of the 
Project Evaluation) and other existing factors under this criterion 
provide for the type of evaluation proposed by the commenter, we do not 
think it is necessary to create a factor for the evaluation of a 
specific type of project.
    Changes: None.

Selection Criteria Based on Statutory or Regulatory Provisions--Sec.  
75.209

    Comment: One commenter agreed with the proposed changes to Sec.  
75.209 but suggested including a clause that explicitly allows for 
successful applicants to be able to adjust their projects based on a 
continuous improvement model. The commenter noted that this change 
would allow grantees to use formative evidence and research to adjust 
their projects as needed, resulting in better outcomes overall.
    Discussion: We agree that continuous improvement models are useful. 
In fact, grantees currently address their lessons learned during the 
implementation of the project and discuss how they can continuously 
improve their projects in their annual performance reports to the 
Department. The regulations do not prohibit a grantee from adjusting 
its project as needed, provided that the scope or objectives of the 
project are not changed. Our intent in the changes to Sec.  75.209 is 
to provide the Secretary the flexibility to use selection criteria 
related to a program's statute or regulations in notices inviting 
applications.
    Changes: None.

General Selection Criteria--Sec.  75.210

    Comments: Many commenters expressed general concern over the 
proposed changes to Sec.  75.210(c) (Quality of the Project Design) 
without focusing on any one proposed factor. Some noted that the 
proposed selection factors under Sec.  75.210(c) are not widely 
applicable to all Department programs and that some selection factors 
may unfairly disadvantage some applicants. For example, one commenter 
asserted that the proposed selection factors will not improve student 
outcomes and are therefore unnecessary. Another commenter expressed 
concern that the proposed selection factors allow the Secretary too 
much discretion when designing competitions and that the intent and 
purpose of some already-established programs could be compromised.
    Alternatively, one commenter suggested that many of the proposed 
selection factors rely too much on peer reviewer subjectivity and 
further that inter-rater reliability between peer reviewers would be 
difficult to achieve if these factors are used in a competition.
    Discussion: We agree that each selection factor in Sec.  75.210(c) 
(Quality of the Project Design) is not applicable to each Department 
program. There is no requirement that the Department use each selection 
factor listed in Sec.  75.210(c) (Quality of the Project Design) for 
every program or competition. We rely on Department program officials 
to choose the selection factors for their programs prudently, with the 
capacity of applicants in mind, such that the selection factors used 
will appropriately match the goals of the program.
    As part of the discretionary grant process, we depend on peer 
reviewers for their objectivity and professional expertise. The 
Department trains peer reviewers on the details of the particular 
program, and monitors peer reviewer discussions to ensure that 
reviewers make scoring decisions based solely on the selection criteria 
provided in the notice inviting applications. While the Department 
takes these steps to support inter-rater reliability, we also rely on 
the professional judgment and expertise of peer reviewers when 
evaluating applications.
    Although some factors may not directly relate to student 
achievement, we disagree with the comment that the new selection 
factors in Sec.  75.210(c) (Quality of the Project Design) will not 
improve student achievement. Each factor requires applicants to 
approach the design of their projects in ways that will increase 
efficiency, productivity, and overall success. Increased student 
achievement will result from a Department competition's use of 
selection factors that better allow grantees to implement their 
projects effectively.
    Changes: None.
    Comment: One commenter agreed with the proposed change to Sec.  
75.210(c)(2)(xvi) regarding integration of a proposed project with 
similar or related efforts. The commenter stated that emphasizing 
integration efforts within the grantee's community would increase the 
likelihood of a successful project. The commenter noted that this 
proposed factor is particularly appropriate for public charter schools, 
given their unique positions in their communities.
    Some commenters expressed concern that the proposed change to Sec.  
75.210(c)(2)(xvi) would disadvantage nonprofit entities. These 
commenters noted that nonprofit entities do not necessarily have 
control over State or Federal funding streams and may have difficulty 
securing willing community partners. One commenter expanded on this 
concern, and suggested that we clarify that the integration of existing 
funding streams only be considered to the extent practicable for 
applicants. Similarly, another commenter noted that in some high-need 
areas, opportunities to partner with funding organizations simply do 
not exist.
    One commenter suggested that Sec.  75.210(c)(2)(xvi) regarding 
increased efficiency and productivity and (xxvii) regarding using 
nonpublic funds or resources to build on similar or related efforts be 
used only as competitive preference priorities. The commenter stressed 
that the proposed selection factors could disadvantage small and rural 
LEAs, and potentially eliminate applicants with otherwise strong 
responses to the criterion due solely to their inability to secure 
other sources of funding. The commenter also warned that a nonpublic 
entity may have its own agenda when agreeing to partner with an 
applicant, possibly complicating the nature of the partnership.
    One commenter expressed concern that nonpublic funds and resources 
were given favor in Sec.  75.210(c)(2)(xxvii). The commenter argued 
that integration of resources is important regardless of their source, 
be it public or private. Based on this argument, the commenter 
suggested the selection factor be removed.

[[Page 49348]]

    Discussion: We agree that the proposed selection factors will 
increase the likelihood of grantee success in implementing their 
projects. Requiring an applicant to create partnerships with community, 
State, or Federal partners establishes early on that project 
sustainability is an important factor for success, in the event that 
the proposed project is awarded grant funds.
    We recognize that establishing partnerships with community 
organizations may be challenging for some grantees, particularly those 
based in high-need areas. However, we would like to clarify that, under 
amended selection factor Sec.  75.210(c)(2)(xvi), a grantee is not 
limited to using organizations based in its community when selecting 
partners. Grantees may choose to integrate or build on the related 
efforts of other programs based anywhere in the country, assuming that 
the goals and efforts of such programs align appropriately with those 
of the grant.
    We also recognize that small and rural LEAs may experience 
challenges when responding to the selection factor. We would like to 
stress that this change to the selection criterion in Sec.  75.210(c) 
(Quality of Project Design) does not require its use in each Department 
competition. The Department chooses appropriate selection factors based 
on the intended goals of the program and the expected capacity of 
applicants.
    We disagree that changing the proposed selection factors into 
competitive preference priorities would prove more favorable for small 
and rural LEAs. Competitive preference priorities, by providing 
grantees who address them with an advantage over those who choose not 
to do so, can be decisive in determining which applicants receive 
grants. Therefore, it is not clear that considering the integration of 
other funding streams as a competitive preference priority, as opposed 
to a selection factor, would address the commenter's concern.
    Nonprofit entities, while typically not in a position to control 
State or Federal funding streams, do have options available to them to 
address Sec.  75.210(c)(2)(xvi). This selection factor considers the 
extent to which an applicant has secured partners to build on similar 
efforts. A nonprofit entity, if it were eligible to apply for funding 
in a Department program, could collaborate with an LEA, which is likely 
receiving State and Federal funding, on a mutually beneficial project 
that aligns with the goals of the Department program.
    While we agree that Sec.  75.210(c)(2)(xxvii) regarding integrating 
with or building on related efforts may not be appropriate for some 
Department programs, we are retaining it because there are Department 
programs in which it would be beneficial. Because the use of this 
selection factor is not required for use in all Department programs or 
competitions, we think the best approach to addressing this concern is 
for the Department to use the selection factor in only those programs 
for which it is appropriate.
    In response to the concern that a nonpublic entity may take 
advantage of an applicant and complicate the nature of the partnership 
to promote its own agenda, applicants applying to a competition in 
which Sec.  75.210(c)(2)(xxvii) is a selection factor should use their 
best judgment in evaluating potential partners and only enter into 
formal relationships with entities that share their goals. This is 
generally a prudent practice, whether the applicant is choosing to 
partner with a public or a nonpublic entity, and should be followed in 
any competition that requires an applicant to work with a partner, even 
if Sec.  75.210(c)(2)(xxvii) is not a selection factor.
    Changes: None.
    Comment: Some commenters praised the proposed selection factor 
Sec.  75.210(c)(2)(xxiv) regarding resources for operating a project 
beyond the length of the grant. Commenters also supported Sec.  
75.210(c)(2)(xxv), which asks applicants to describe the potential and 
planning for the incorporation of project purposes and activities into 
the ongoing work of the grant. These commenters stated that grant funds 
should not be awarded in cases where long-term funding is needed but 
not secured and that asking an applicant to explicitly address how it 
plans to continue the project after the completion of a grant award 
will help to ensure long-term success.
    Two commenters expressed agreement with the proposed changes to 
Sec.  75.210(c)(2)(xxiv) and (xxv), but suggested some further 
modifications. One commenter suggested that we consider whether 
applicants have effectively worked to build a market for educational 
services. Another commenter suggested that when considering the extent 
to which an applicant has secured resources to sustain the project 
beyond the grant period, we also consider whether the applicant has 
demonstrated evidence of broad stakeholder commitment to the project.
    One commenter agreed that it is critical that grantees plan their 
projects with sustainability in mind but did not agree with the 
addition of selection factors Sec.  75.210(c)(2)(xxiv) and (xxv), 
arguing that the current state of the economy is not stable enough to 
ensure that resources committed at the time an award is made would 
still be available at the end of a grant period. Another commenter 
noted that the proposed changes could disadvantage community colleges 
and proposed that we consider an applicant's cost per student when 
using proposed selection factors related to sustainability. The 
commenter stated that applicants working with fewer resources per 
student need more flexibility in adhering to the requirements outlined 
in selection factor Sec.  75.210(c)(2)(xxiv).
    Discussion: We agree that long-term planning and broad stakeholder 
support are integral to a grantee's successful project. The amendments 
to Sec.  75.210(c) (Quality of the Project Design) will allow for 
flexibility when assessing an applicant's plan to sustain its project 
after the grant period ends. With added flexibility in Sec.  75.210(c), 
the Department may choose to fund applications that have a strong focus 
in effective and sustainable practices.
    We recognize that some applicants, such as community colleges, may 
operate with fewer resources per student than other types of 
applicants. However, the regulations do not prohibit such an applicant 
from requesting funds in its budget proposal to support the proposed 
project fully. If an applicant assesses its resources and finds that it 
requires more funds per student to carry out the project and address 
selection factor Sec.  75.210(c)(2)(xxiv), that applicant should plan 
its budget accordingly. It is important that an applicant have the 
discretion to determine how best to address its sustainability needs. 
For example, an applicant may design its project to include strategies 
that build its capacity to implement project activities more 
efficiently, which in turn would support sustainability after the 
grant.
    The proposed selection factors related to sustainability are 
designed with the current economic climate in mind. As a few commenters 
noted, applicants should be actively planning on how those realities 
will affect their proposed projects. The intent of selection factors 
Sec.  75.210(c)(2)(xxiv) and (xxv) is to encourage applicants to engage 
in this planning process and comprehensively plan how their projects 
could be implemented beyond the grant period if such projects were 
funded.
    Finally, regarding the recommendation that we include a factor 
considering whether an applicant

[[Page 49349]]

effectively worked to build a market for educational services, we note 
that we have added a new selection criterion, Sec.  75.210(i) (Strategy 
to Scale), which includes selection factor Sec.  75.210(i)(2)(v), that 
considers whether an applicant demonstrates an unmet demand for the 
process, product, strategy, or practice that will enable the applicant 
to reach the level of scale that is proposed in the application. This 
factor recognizes work that an applicant would do in advance of the 
project to build a market for educational services.
    Changes: None.
    Comment: Two commenters noted specifically the importance of 
productivity and efficiency, stating that selection factor Sec.  
75.210(c)(2)(xxvi) is especially appropriate given the current climate 
of limited resources and high expectations for success.
    Discussion: We agree that productivity and efficiency have become 
increasingly important factors to consider in recent years. Allowing 
the Secretary to evaluate whether a proposed project is efficient and 
productive will ensure that Department funds are used as effectively 
and prudently as possible.
    Changes: None.
    Comments: Two commenters agreed with the addition of a new 
selection criterion, Sec.  75.210(i) (Strategy to Scale), to consider 
an applicant's ability to successfully scale a project at the regional 
or national level. One commenter noted that the proposed addition would 
specifically benefit charter management organizations and support them 
in scaling successful strategies, and the other noted the importance of 
sharing best practices broadly.
    Another commenter expressed support for the selection criterion in 
Sec.  75.210(i) but requested that we allow for-profit entities, as 
well as nonprofit entities, to partner with grantees to bring their 
projects to scale during the grant period. The commenter stated that 
scaling has not historically been an area of expertise for entities 
providing educational services and that for-profit entities are well-
suited to provide needed support.
    Discussion: We agree that, in many Department programs, an 
applicant's ability to scale a proposed process, product, strategy, or 
program is very important. As the Department begins and continues to 
use this selection criterion, we expect potential applicants will 
devote resources and supports to focus on the processes, products, 
strategies, and programs that have greater potential to scale.
    Should a grantee decide that it needs additional help in the area 
of scalability, that grantee is not obligated to seek assistance from 
only nonprofit entities. The proposed selection criterion, as written, 
does not explicitly refer to the types of entities with which a grantee 
may choose to work. We recognize that some for-profit entities may be 
particularly well-positioned to help grantees achieve scale, and 
encourage each grantee, to the extent it requires additional support, 
to seek out partners that are best suited to meet the needs of their 
projects.
    Changes: None.
    Comments: Many commenters noted that while the strategy to scale is 
an important criterion to consider for new projects, it is not 
applicable to programs that have already established successful 
practices at a national scale or to programs that are already widely 
available to students.
    Conversely, some commenters expressed concern that the proposed 
selection criterion Sec.  75.210(i) (Strategy to Scale) would not be 
feasible for small LEAs, rural LEAs, or community colleges. One 
commenter requested revising the language of the proposed criterion to 
``Feasibility of Replication'' and placing it as a selection factor 
under Sec.  75.210(h) (Quality of the Project Evaluation). This 
commenter also suggested that the subject of scaling a project is 
better suited to an IES grant. Another commenter noted that the 
selection criterion should instead be used only as a selection factor 
in specific circumstances because an applicant's capacity to scale is 
not a useful consideration for all Department programs. Another 
commenter added that some programs are, by definition, small and 
community based and that the use of this criterion would adversely 
affect such programs.
    One commenter did not agree with the proposed selection criterion 
concerning an applicant's strategy to scale, noting that increasing 
focus on scaling projects to regional and national levels would 
decrease focus on student outcomes at the local level. The commenter 
also points out that many projects are effective because they are 
planned with a specific place in mind, and scaling such projects could 
prove ineffective.
    Discussion: We recognize that the proposed selection criterion may 
not be applicable to Department programs that are already well 
established. We stress that Department program officials are in the 
best position to decide which selection criteria and factors fit the 
goals of their programs. When preparing notices inviting applications, 
the Department will continue to consider the strengths and needs of 
likely applicants, and will choose selection criteria that are 
appropriate to the program's purpose, goals, and applicant pool. 
Likewise, if the Department concludes that the nature of the program or 
types of applicants are not conducive to scaling, then the Department 
may decide not to include the selection criterion in the notice 
inviting applications for the program.
    If the Department concludes the use of this criterion is consistent 
with the program's purpose and goals then applicants that better 
address the criterion will likely receive more points for the criterion 
than applicants that address it poorly. We recognize that some types of 
applicants may not typically design or implement projects that include 
activities to support effectively scaling a proposed process, product, 
strategy, or practice; however, any applicant responding to a notice 
inviting applications that includes this criterion may consider 
partnering with others to take the proposed process, product, strategy, 
or practice to scale.
    We do not agree with the suggestion to change selection criterion 
Sec.  75.210(i) (Strategy to Scale) into a selection factor under 
selection criterion Sec.  75.210(h) (Quality of the Project 
Evaluation). The nuances needed to make the free-standing selection 
criterion useful and adaptable to a variety of Department programs 
would be lost if it were re-written as a selection factor under another 
criterion. It is important that a grantee experiencing success be able 
to share information about its project and support broad implementation 
of it to ensure that best practices are widely accessible and more 
easily replicated in the field. We think that by including Sec.  
75.210(i) (Strategy to Scale) as a selection criterion, as opposed to a 
selection factor within a selection criterion, we are able to provide 
clearer guidance to applicants on effective scaling methodology and 
feasible replication.
    We disagree that including a selection criterion that considers an 
applicant's ability to effectively scale its proposed process, product, 
strategy, or practice would decrease focus on student outcomes at the 
local level. By choosing to consider one selection criterion, the 
Department does not diminish the influence of other selection criteria 
under consideration. For example, if Department program officials 
choose to consider Sec.  75.210(c) (Quality of the Project Design) and 
Sec.  75.210(i) (Strategy to Scale), a successful applicant would be 
expected to respond effectively to both criteria. That applicant would

[[Page 49350]]

therefore need to explain why its project design is effective in 
increasing student outcomes in its current setting and explain its 
capacity to scale. While Sec.  75.210(i) (Strategy to Scale) primarily 
considers how well an applicant could implement its proposed process, 
product, strategy, or practice in a variety of settings and 
populations, it remains one piece among many to be considered as part 
of the competition process.
    Changes: None.

Maximum Funding Period--Sec.  75.250

    Comments: Many commenters expressed support for the change to this 
regulation because it will improve the quality of the data available to 
determine whether educational activities improve teaching and learning. 
However, one of these commenters stated that the option for funding for 
continued evaluation should be guaranteed. The commenter also suggested 
that grantees be allowed to use funds to hire qualified data management 
personnel and consultants to develop data architecture and data storage 
capacity.
    Discussion: We appreciate the commenters' support for this 
regulation. However, we cannot guarantee continued data collection 
periods for all programs and grantees because this option is only 
possible in cases where there is authority for evaluation activities 
and sufficient funds have been appropriated for the program. Because 
these conditions may not be met for all programs or in all years, we 
cannot guarantee a continued data collection period for all programs 
and grantees.
    With regard to the recommendation that the Department allow 
grantees to use grant funds to support personnel or contracts to assist 
with data collection, we note that, under the current regulations and 
cost principles, applicants may include such costs in their proposed 
budgets to contract for these services so long as they are necessary to 
the performance measurement and evaluation of the project.
    Changes: None.
    Comment: One commenter recommended clarifying whether the Secretary 
could approve a data collection period without providing additional 
funds. The commenter explained that in some cases grantees may need 
both additional time and funds in order to complete performance 
measurement activities but that there are also instances when a grantee 
may need only additional time.
    Discussion: Under Sec.  75.261, a grantee may request a no-cost 
extension of its project period to complete approved project 
activities. Thus, the regulations already allow grantees to request 
additional time to complete performance measure and other project 
activities without additional funding, so long as the appropriation 
accounts remain available. Funds obligated on a fiscal year basis 
remain available in grant accounts for five fiscal years after the 
expiration of the fiscal year for which the funds were obligated by the 
Federal government. 31 U.S.C. 1552(a). Thus, both obligated and 
unobligated grant funds generally remain available to grantees during 
no cost extensions to fund continued collection of data after the end 
of a project period.
    The amended regulations in Sec.  75.250 allow the Secretary to 
approve a data collection period with or without additional funds. 
Prior to the approval of a data collection period, we would assess with 
the grantee the appropriate duration of the data collection period and 
whether additional funds are necessary to complete the data collection, 
reporting, and analysis that would occur during that period.
    Changes: None.
    Comment: One commenter stated that data collection is not the only 
valid reason for extending a grant period and suggested revising the 
regulations to allow extensions on the basis of effectiveness and to 
aid in a project's transition to a new funding stream.
    Discussion: These amendments apply to discretionary grant programs 
that award funds on the basis of a competitive process. As such, it 
would not be appropriate for the Department to award additional funds 
to a grantee to conduct a new project or transition to a new funding 
stream outside of the competitive process.
    Changes: None.

Continuation of a Multi-Year Project After the First Budget Period--
Sec.  75.253

    Comments: Three commenters expressed support for the change in this 
regulation. One commenter stated that the change would improve the use 
of performance measurement and evaluation. However, one of these 
commenters requested additional information on the meaning of 
``substantial progress.'' Another commenter urged establishing program-
specific evaluation requirements that balance the need for valid 
evidence of effectiveness with the need to limit burden on grantees.
    Discussion: We appreciate the commenters' support for the amended 
regulation. Given the variety of programs to which these regulations 
apply, a more detailed definition of ``substantial progress'' would not 
be practical or helpful. The Department will establish program-specific 
evaluation requirements in the notice inviting applications and will 
consider the program's purpose, goals, and applicant pool when deciding 
the evaluation requirements to use in a given year's competition. As 
part of this process, the Department must consider the burden of the 
information collection associated with the application and program 
requirements and receive approval under the Paperwork Reduction Act of 
1995 from OMB to collect that information. Because current law requires 
programs to consider the burden associated with information collection, 
we do not think it is necessary to make the change requested by the 
commenter.
    Changes: None.

Other Comments Not Directly Related to the Proposed Rule

    Comment: One commenter stated that the amendments were unclear and 
would produce nothing of value for students in this country.
    Discussion: Although we recognize these amendments may not directly 
affect students, we disagree with the assessment that they would not 
produce anything of value. These amendments are designed to support the 
successful implementation of projects funded by the Department and 
improve their performance measures, which will in turn benefit students 
served by the projects. The proposed amendments also allow the 
Department to be more effective and efficient when selecting 
discretionary grantees, to provide higher-quality data to the Congress 
and the public about the effectiveness of Department programs, and to 
reduce administrative burden on applicants and grantees.
    Changes: None.
    Comment: One commenter expressed concern that students with 
disabilities are underserved.
    Discussion: We appreciate the commenter's concern and note that 
section 504 of the Rehabilitation Act of 1973 and the Department's 
section 504 implementing regulations prohibit discrimination on the 
basis of disability for entities receiving financial assistance from 
the Department. In addition, the Department enforces Title II of the 
Americans with Disabilities Act (ADA), as well as the regulations 
implementing Title II of the ADA, which prohibit discrimination on the 
basis of disability by public entities. Finally, section 427 of the 
General Education Provisions Act (GEPA) addresses equitable access by 
requiring all applicants to provide a statement that identifies access 
barriers

[[Page 49351]]

to participation, which can include barriers to participation by 
individuals with disabilities, in their projects and identifies 
solutions to overcome those barriers.
    Moreover, the Department's Office of Special Education and 
Rehabilitative Services (OSERS) is committed to improving results and 
outcomes for people with disabilities of all ages. OSERS provides a 
wide array of supports to parents and individuals, school districts, 
and states in three main areas: Special education, vocational 
rehabilitation, and research. Within OSERS, the Office of Special 
Education Programs (OSEP) supports a comprehensive array of programs 
and projects authorized by the Individuals with Disabilities Education 
Act (IDEA) that improve results for infants, toddlers, children, and 
youth with disabilities.
    Changes: None.
    Comment: One commenter recommended adding language to Sec. Sec.  
76.722 and 80.40 to clarify that, although a grantee may require 
subrecipients to submit reports in a manner and format that enable the 
grantee to comply with Department requirements, an SEA must not do so 
in a manner that would place financial or programmatic burden on the 
subrecipient or require a subrecipient to provide data that is readily 
available to the SEA by other means. The commenter noted that 
monitoring subrecipients is vital to the successful implementation of a 
grant, but a grantee should not use it to usurp autonomy of 
subrecipients or to require the use of specific financial software that 
could be costly or burdensome to small entities.
    Discussion: In the preamble of the NPRM, we discussed on page 74392 
the Department's retrospective analysis of existing regulations and 
requested comment on other regulations within EDGAR that may be in need 
of modification. We appreciate this commenter's concerns regarding 
Sec. Sec.  76.722 and 80.40 and will use the feedback to further inform 
and plan our retrospective review efforts.
    Changes: None.

Executive Orders 12866 and 13563

Regulatory Impact Analysis

    Under Executive Order 12866, the Secretary must determine whether 
this regulatory action is ``significant'' and, therefore, subject to 
the requirements of the Executive order and subject to review by the 
Office of Management and Budget (OMB). Section 3(f) of Executive Order 
12866 defines a ``significant regulatory action'' as an action likely 
to result in a rule that may--
    (1) Have an annual effect on the economy of $100 million or more, 
or adversely affect a sector of the economy, productivity, competition, 
jobs, the environment, public health or safety, or State, local, or 
tribal governments or communities in a material way (also referred to 
as an ``economically significant'' rule);
    (2) Create serious inconsistency or otherwise interfere with an 
action taken or planned by another agency;
    (3) Materially alter the budgetary impacts of entitlement grants, 
user fees, or loan programs or the rights and obligations of recipients 
thereof; or
    (4) Raise novel legal or policy issues arising out of legal 
mandates, the President's priorities, or the principles stated in the 
Executive order.
    This final regulatory action is a significant regulatory action 
subject to review by OMB under section 3(f) of Executive Order 12866.
    We have also reviewed these regulations under Executive Order 
13563, which supplements and explicitly reaffirms the principles, 
structures, and definitions governing regulatory review established in 
Executive Order 12866. To the extent permitted by law, Executive Order 
13563 requires that an agency--
    (1) Propose or adopt regulations only on a reasoned determination 
that their benefits justify their costs (recognizing that some benefits 
and costs are difficult to quantify);
    (2) Tailor its regulations to impose the least burden on society, 
consistent with obtaining regulatory objectives and taking into 
account--among other things and to the extent practicable--the costs of 
cumulative regulations;
    (3) In choosing among alternative regulatory approaches, select 
those approaches that maximize net benefits (including potential 
economic, environmental, public health and safety, and other 
advantages; distributive impacts; and equity);
    (4) To the extent feasible, specify performance objectives, rather 
than the behavior or manner of compliance a regulated entity must 
adopt; and
    (5) Identify and assess available alternatives to direct 
regulation, including economic incentives--such as user fees or 
marketable permits--to encourage the desired behavior, or provide 
information that enables the public to make choices.
    Executive Order 13563 also requires an agency ``to use the best 
available techniques to quantify anticipated present and future 
benefits and costs as accurately as possible.'' The Office of 
Information and Regulatory Affairs of OMB has emphasized that these 
techniques may include ``identifying changing future compliance costs 
that might result from technological innovation or anticipated 
behavioral changes.''
    We are issuing these final regulations only on a reasoned 
determination that their benefits justify their costs. In choosing 
among alternative regulatory approaches, we selected those approaches 
that maximize net benefits. Based on the analysis that follows, the 
Department believes that these proposed regulations are consistent with 
the principles in Executive Order 13563.
    We also have determined that this regulatory action does not unduly 
interfere with State, local, or tribal governments in the exercise of 
their governmental functions.
    In accordance with both Executive orders, the Department has 
assessed the potential costs and benefits, both quantitative and 
qualitative, of this regulatory action. The potential costs associated 
with this regulatory action are those resulting from statutory 
requirements and those we have determined as necessary for 
administering the Department's programs and activities.

Paperwork Reduction Act of 1995

    These regulations do not contain any information collection 
requirements.

Intergovernmental Review

    These regulations subject to the requirements of Executive Order 
12372 and the regulations in 34 CFR part 79. One of the objectives of 
the Executive order is to foster an intergovernmental partnership and a 
strengthened federalism. The Executive order relies on processes 
developed by State and local governments for coordination and review of 
proposed Federal financial assistance.
    This document provides early notification of our specific plans and 
actions for these regulations.

Assessment of Educational Impact

    In the NPRM we requested comments on whether the proposed 
regulations would require transmission of information that any other 
agency or authority of the United States gathers or makes available.
    Based on the response to the NPRM and on our review, we have 
determined that these final regulations do not require transmission of 
information that any other agency or authority of the United States 
gathers or makes available.

[[Page 49352]]

    Accessible Format: Individuals with disabilities can obtain this 
document in an accessible format (e.g., braille, large print, 
audiotape, or compact disc) on request to the program contact person 
listed under FOR FURTHER INFORMATION CONTACT.
    Electronic Access to This Document: The official version of this 
document is the document published in the Federal Register. Free 
Internet access to the official edition of the Federal Register and the 
Code of Federal Regulations is available via the Federal Digital System 
at: www.gpo.gov/fdsys. At this site you can view this document, as well 
as all other documents of this Department published in the Federal 
Register, in text or Adobe Portable Document Format (PDF). To use PDF 
you must have Adobe Acrobat Reader, which is available free at the 
site.
    You may also access documents of the Department published in the 
Federal Register by using the article search feature at: 
www.federalregister.gov. Specifically, through the advanced search 
feature at this site, you can limit your search to documents published 
by the Department. You may also view this document in text [or PDF] at 
the following site:

List of Subjects

34 CFR Part 75

    Accounting, Copyright, Education, Grant programs--education.

34 CFR Part 77

    Education, Grant programs--education.

    Dated: August 6, 2013.
Arne Duncan,
Secretary of Education.
    For the reasons discussed in the preamble, the Secretary amends 
parts 75 and 77 of title 34 of the Code of Federal Regulations as 
follows:

PART 75--DIRECT GRANT PROGRAMS

0
1. The authority citation for part 75 continues to read as follows:

    Authority: 20 U.S.C. 1221e-3 and 3474, unless otherwise noted.


0
2. Add a new Sec.  75.110 to read as follows:


Sec.  75.110  Information regarding performance measurement.

    (a) The Secretary may establish in an application notice for a 
competition one or more performance measurement requirements, including 
requirements for performance measures, baseline data, or performance 
targets, and a requirement that applicants propose in their 
applications one or more of their own performance measures, baseline 
data, or performance targets.
    (b) If an application notice requires applicants to propose 
project-specific performance measures, baseline data, or performance 
targets, the application must include the following, as required by the 
application notice:
    (1) Performance measures. How each proposed performance measure 
would accurately measure the performance of the project and how the 
proposed performance measure would be consistent with the performance 
measures established for the program funding the competition.
    (2) Baseline data. (i) Why each proposed baseline is valid; or
    (ii) If the applicant has determined that there are no established 
baseline data for a particular performance measure, an explanation of 
why there is no established baseline and of how and when, during the 
project period, the applicant would establish a valid baseline for the 
performance measure.
    (3) Performance targets. Why each proposed performance target is 
ambitious yet achievable compared to the baseline for the performance 
measure and when, during the project period, the applicant would meet 
the performance target(s).
    (c) If the application notice establishes performance measurement 
requirements, the applicant must also describe in the application--
    (1)(i) The data collection and reporting methods the applicant 
would use and why those methods are likely to yield reliable, valid, 
and meaningful performance data; and
    (ii) If the Secretary requires applicants to collect data after the 
substantive work of a project is complete regarding the attainment of 
certain performance targets, the data collection and reporting methods 
the applicant would use during the post-performance period and why 
those methods are likely to yield reliable, valid, and meaningful 
performance data.
    (2) The applicant's capacity to collect and report reliable, valid, 
and meaningful performance data, as evidenced by high-quality data 
collection, analysis, and reporting in other projects or research.

(Authority: 20 U.S.C. 1221e-3 and 3474)


0
3. Add a new undesignated center heading ``Competition Exceptions'' in 
subpart C immediately before the undesignated center heading ``State 
Comment Procedures''.

0
4. Add a new Sec.  75.135 to subpart C under the undesignated center 
heading ``Competition Exceptions'' to read as follows:


Sec.  75.135  Competition exception for proposed implementation sites, 
implementation partners, or service providers.

    (a) When entering into a contract with implementation sites or 
partners, an applicant is not required to comply with the competition 
requirements in 34 CFR 74.43 or 80.36(c), as applicable, if--
    (1) The contract is with an entity that agrees to provide a site or 
sites where the applicant would conduct the project activities under 
the grant;
    (2) The implementation sites or partner entities that the applicant 
proposes to use are identified in the application for the grant; and
    (3) The implementation sites or partner entities are included in 
the application in order to meet a regulatory, statutory, or priority 
requirement related to the competition.
    (b) When entering into a contract for data collection, data 
analysis, evaluation services, or essential services, an applicant may 
select a provider using the informal, small-purchase procurement 
procedures in 34 CFR 80.36(d)(1), regardless of whether that applicant 
would otherwise be subject to that part or whether the evaluation 
contract would meet the standards for a small purchase order, if--
    (1) The contract is with the data collection, data analysis, 
evaluation service, or essential service provider;
    (2) The data collection, data analysis, evaluation service, or 
essential service provider that the applicant proposes to use is 
identified in the application for the grant; and
    (3) The data collection, data analysis, evaluation service, or 
essential service provider is identified in the application in order to 
meet a statutory, regulatory, or priority requirement related to the 
competition.
    (c) If the grantee relied on the exceptions under paragraph (a) or 
(b) of this section, the grantee must certify in its application that 
any employee, officer, or agent participating in the selection, award, 
or administration of a contract is free of any real or apparent 
conflict of interest and, if the grantee relied on the exceptions of 
paragraph (b) of this section, that the grantee used small purchase 
procedures to obtain the product or service.
    (d) A grantee must obtain the Secretary's prior approval for any 
change to an implementation site, implementation partner, or data 
collection, data analysis, evaluation service, or essential service 
provider, if the grantee relied on the exceptions

[[Page 49353]]

under paragraph (a) or (b) of this section to select the entity.
    (e) The exceptions in paragraphs (a) and (b) of this section do not 
extend to the other procurement requirements in 34 CFR part 74 and 34 
CFR part 80 regarding contracting by grantees and subgrantees.
    (f) For the purposes of this section, essential service means a 
product or service directly related to the grant that would, if not 
provided, have a detrimental effect on the grant.

(Authority: 20 U.S.C. 1221e-3 and 3474)


0
5. Revise Sec.  75.209 to read as follows:


Sec.  75.209  Selection criteria based on statutory or regulatory 
provisions.

    The Secretary may establish selection criteria and factors based on 
statutory or regulatory provisions that apply to the authorized 
program, which may include, but are not limited to criteria and factors 
that reflect--
    (a) Criteria contained in the program statute or regulations;
    (b) Criteria in Sec.  75.210;
    (c) Allowable activities specified in the program statute or 
regulations;
    (d) Application content requirements specified in the program 
statute or regulations;
    (e) Program purposes, as described in the program statute or 
regulations; or
    (f) Other pre-award and post-award conditions specified in the 
program statute or regulations.

(Authority: 20 U.S.C. 1221e-3 and 3474)


0
6. Amend Sec.  75.210 by:
0
A. Revising the introductory text.
0
B. Revising paragraph (c)(2)(xvi).
0
C. Adding paragraphs (c)(2)(xxiv) through (xxix).
0
D. Adding paragraphs (h)(2)(viii) through (xii).
0
E. Adding a new paragraph (i).
    The revisions and additions read as follows.


Sec.  75.210  General selection criteria.

    In determining the selection criteria to evaluate applications 
submitted in a grant competition, the Secretary may select one or more 
of the following criteria and may select from among the list of 
optional factors under each criterion. The Secretary may define a 
selection criterion by selecting one or more specific factors within a 
criterion or assigning factors from one criterion to another criterion.
* * * * *
    (c) * * *
    (2) * * *
    (xvi) The extent to which the proposed project will integrate with 
or build on similar or related efforts to improve relevant outcomes (as 
defined in 34 CFR 77.1(c)), using existing funding streams from other 
programs or policies supported by community, State, and Federal 
resources.
* * * * *
    (xxiv) The extent to which the applicant demonstrates that it has 
the resources to operate the project beyond the length of the grant, 
including a multi-year financial and operating model and accompanying 
plan; the demonstrated commitment of any partners; evidence of broad 
support from stakeholders (e.g., State educational agencies, teachers' 
unions) critical to the project's long-term success; or more than one 
of these types of evidence.
    (xxv) The potential and planning for the incorporation of project 
purposes, activities, or benefits into the ongoing work of the 
applicant beyond the end of the grant.
    (xxvi) The extent to which the proposed project will increase 
efficiency in the use of time, staff, money, or other resources in 
order to improve results and increase productivity.
    (xxvii) The extent to which the proposed project will integrate 
with or build on similar or related efforts in order to improve 
relevant outcomes (as defined in 34 CFR 77.1(c)), using nonpublic funds 
or resources.
    (xxviii) The extent to which the proposed project is supported by 
evidence of promise (as defined in 34 CFR 77.1(c)).
    (xxix) The extent to which the proposed project is supported by 
strong theory (as defined in 34 CFR 77.1(c)).
* * * * *
    (h) * * *
    (2) * * *
    (viii) The extent to which the methods of evaluation will, if well-
implemented, produce evidence about the project's effectiveness that 
would meet the What Works Clearinghouse Evidence Standards without 
reservations.\1\
---------------------------------------------------------------------------

    \1\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    (ix) The extent to which the methods of evaluation will, if well-
implemented, produce evidence about the project's effectiveness that 
would meet the What Works Clearinghouse Evidence Standards with 
reservations.\2\
---------------------------------------------------------------------------

    \2\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    (x) The extent to which the methods of evaluation will, if well-
implemented, produce evidence of promise (as defined in 34 CFR 
77.1(c)).
    (xi) The extent to which the methods of evaluation will provide 
valid and reliable performance data on relevant outcomes.
    (xii) The extent to which the evaluation plan clearly articulates 
the key components, mediators, and outcomes of the grant-supported 
intervention, as well as a measurable threshold for acceptable 
implementation.
    (i) Strategy to scale. (1) The Secretary considers the applicant's 
strategy to scale the proposed project.
    (2) In determining the applicant's capacity to scale the proposed 
project, the Secretary considers one or more of the following factors:
    (i) The applicant's capacity (e.g., in terms of qualified 
personnel, financial resources, or management capacity) to bring the 
proposed project to scale on a national or regional level (as defined 
in 34 CFR 77.1(c)) working directly, or through partners, during the 
grant period.
    (ii) The applicant's capacity (e.g., in terms of qualified 
personnel, financial resources, or management capacity) to further 
develop and bring to scale the proposed process, product, strategy, or 
practice, or to work with others to ensure that the proposed process, 
product, strategy, or practice can be further developed and brought to 
scale, based on the findings of the proposed project.
    (iii) The feasibility of successful replication of the proposed 
project, if favorable results are obtained, in a variety of settings 
and with a variety of populations.
    (iv) The mechanisms the applicant will use to broadly disseminate 
information on its project so as to support further development or 
replication.
    (v) The extent to which the applicant demonstrates there is unmet 
demand for the process, product, strategy, or practice that will enable 
the applicant to reach the level of scale that is proposed in the 
application.
    (vi) The extent to which the applicant identifies a specific 
strategy or strategies that address a particular barrier or barriers 
that prevented the applicant, in the past, from reaching the level of 
scale that is proposed in the application.

0
7. Revise Sec.  75.250 to read as follows:


Sec.  75.250  Maximum funding period.

    (a) The Secretary may approve a project period of up to 60 months 
to perform the substantive work of a grant.
    (b) The Secretary may approve a data collection period for a grant 
for a period of up to 72 months after the end of the project period and 
provide funding for

[[Page 49354]]

the data collection period for the sole purpose of collecting, 
analyzing, and reporting performance measurement data regarding the 
project. The Secretary may inform applicants of the Secretary's intent 
to approve data collection periods in the application notice published 
for a competition or may decide to fund data collection periods after 
grantees have started their project periods.

(Authority: 20 U.S.C. 1221e-3 and 3474.)


0
8. Amend Sec.  75.251 by revising the section heading and adding a new 
paragraph (c) to read as follows:


Sec.  75.251  Budget periods.

* * * * *
    (c) If the Secretary funds a multi-year data collection period, the 
Secretary may fund the data collection period through separate budget 
periods and fund those budget periods in the same manner as those 
periods are funded during the project period.

0
9. Amend Sec.  75.253 by--
0
A. Revising paragraph (a)(2).
0
B. Adding a new paragraph (a)(5).
0
C. Redesignating paragraphs (b) through (e) as paragraphs (c) through 
(f).
0
D. Adding a new paragraph (b).
0
E. Revising newly redesignated paragraph (f).
    The revisions and additions read as follows:


Sec.  75.253  Continuation of a multi-year project after the first 
budget period.

    (a) * * *
    (2) The grantee has either--
    (i) Made substantial progress in achieving--
    (A) The goals and objectives of the project; and
    (B) If the Secretary established performance measurement 
requirements for the grant in the application notice, the performance 
targets in the grantee's approved application; or
    (ii) Obtained the Secretary's approval for changes to the project 
that--
    (A) Do not increase the amount of funds obligated to the project by 
the Secretary; and
    (B) Enable the grantee to achieve the goals and objectives of the 
project and meet the performance targets of the project, if any, 
without changing the scope or objectives of the project.
* * * * *
    (5) The grantee has maintained financial and administrative 
management systems that meet the requirements in 34 CFR 74.21 or 80.20, 
as appropriate.
    (b) In deciding whether a grantee has made substantial progress, 
the Secretary may consider any information relevant to the authorizing 
statute, a criterion, a priority, or a performance measure, or to a 
financial or other requirement that applies to the selection of 
applications for new grants.
* * * * *
    (f) Unless prohibited by the program statute or regulations, a 
grantee that is in the final budget period of its project period may 
seek continued assistance for the project as required under the 
procedures for selecting new projects for grants.
* * * * *

0
10. Add Sec.  75.266 to subpart D to read as follows:


Sec.  75.266  What procedures does the Secretary use if the Secretary 
decides to give special consideration to applications supported by 
strong or moderate evidence of effectiveness?

    (a) As used in this section, ``strong evidence of effectiveness'' 
is defined in 34 CFR 77.1(c);
    (b) As used in this section, ``moderate evidence of effectiveness'' 
is defined in 34 CFR 77.1(c); and
    (c) If the Secretary determines that special consideration of 
applications supported by strong or moderate evidence of effectiveness 
is appropriate, the Secretary may establish a separate competition 
under the procedures in 34 CFR 75.105(c)(3), or provide competitive 
preference under the procedures in 34 CFR 75.105(c)(2), for 
applications supported by:
    (1) Evidence of effectiveness that meets the conditions set out in 
paragraph (a) of the definition of ``strong evidence of effectiveness'' 
in 34 CFR 77.1;
    (2) Evidence of effectiveness that meets the conditions set out in 
either paragraph (a) or (b) of the definition of ``strong evidence of 
effectiveness'' in 34 CFR 77.1; or
    (3) Evidence of effectiveness that meets the conditions set out in 
the definition of ``moderate evidence of effectiveness.''

(Authority: 20 U.S.C. 1221e-3 and 3474.)


0
11. Revise Sec.  75.590 to read as follows.


Sec.  75.590  Evaluation by the grantee.

    (a) If the application notice for a competition required applicants 
to describe how they would evaluate their projects, each grantee under 
that competition must demonstrate to the Department that--
    (1) The evaluation meets the standards of the evaluation in the 
approved application for the project; and
    (2) The performance measurement data collected by the grantee and 
used in the evaluation meet the performance measurement requirements of 
the approved application.
    (b) If the application notice for a competition did not require 
applicants to describe how they would evaluate their projects, each 
grantee must provide information in its performance report 
demonstrating--
    (1) The progress made by the grantee in the most recent budget 
period, including progress based on the performance measurement 
requirements for the grant, if any;
    (2) The effectiveness of the grant, including fulfilling the 
performance measurement requirements of the approved application, if 
any; and
    (3) The effect of the project on the participants served by the 
project, if any.

(Authority: 20 U.S.C. 1221e-3 and 3474.)


0
12. Amend Sec.  75.708 by:
0
A. Revising the section heading.
0
B. Revising paragraph (a).
0
C. Redesignating paragraph (b) as paragraph (e).
0
D. Adding new paragraphs (b), (c) and (d).
    The revision and additions read as follows.


Sec.  75.708  Subgrants.

    (a) A grantee may not make a subgrant under a program covered by 
this part unless authorized by statute or by paragraph (b) of this 
section.
    (b) The Secretary may, through an announcement in the Federal 
Register, authorize subgrants when necessary to meet the purposes of a 
program. In this announcement, the Secretary will--
    (1) Designate the types of entities, e.g., State educational 
agencies, local educational agencies, institutions of higher education, 
and nonprofit organizations, to which subgrants can be awarded; and
    (2) Indicate whether subgrants can be made to entities identified 
in an approved application or, without regard to whether the entity is 
identified in an approved application, have to be selected through a 
competitive process set out in subgranting procedures established by 
the grantee.
    (c) If authorized under paragraph (b) of this section, a subgrant 
is allowed if it will be used by that entity to directly carry out 
project activities described in that application.
    (d) The grantee, in awarding subgrants under paragraph (b) of this 
section, must--
    (1) Ensure that subgrants are awarded on the basis of an approved 
budget that is consistent with the grantee's approved application and 
all applicable Federal statutory, regulatory, and other requirements;

[[Page 49355]]

    (2) Ensure that every subgrant includes any conditions required by 
Federal statute and executive orders and their implementing 
regulations; and
    (3) Ensure that subgrantees are aware of requirements imposed upon 
them by Federal statute and regulation, including the Federal anti-
discrimination laws enforced by the Department.
* * * * *

PART 77-- DEFINITIONS THAT APPLY TO DEPARTMENT REGULATIONS

0
13. The authority citation for part 77 is revised to read as follows:

    Authority:  20 U.S.C. 1221e-3 and 3474, unless otherwise noted.


0
14. Amend Sec.  77.1(c) by adding the following definitions in 
alphabetical order:


Sec.  77.1  Definitions that apply to all Department programs.

* * * * *
    (c) * * *
    Ambitious means promoting continued, meaningful improvement for 
program participants or for other individuals or entities affected by 
the grant, or representing a significant advancement in the field of 
education research, practices, or methodologies. When used to describe 
a performance target, whether a performance target is ambitious depends 
upon the context of the relevant performance measure and the baseline 
for that measure.
* * * * *
    Baseline means the starting point from which performance is 
measured and targets are set.
* * * * *
    Evidence of promise means there is empirical evidence to support 
the theoretical linkage(s) between at least one critical component and 
at least one relevant outcome presented in the logic model for the 
proposed process, product, strategy, or practice. Specifically, 
evidence of promise means the conditions in paragraphs (a) and (b) of 
this section are met:
    (i) There is at least one study that is a--
    (A) Correlational study with statistical controls for selection 
bias;
    (B) Quasi-experimental study that meets the What Works 
Clearinghouse Evidence Standards with reservations; \1\ or
---------------------------------------------------------------------------

    \1\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    (C) Randomized controlled trial that meets the What Works 
Clearinghouse Evidence Standards with or without reservations.\2\
---------------------------------------------------------------------------

    \2\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    (ii) The study referenced in paragraph (a) found a statistically 
significant or substantively important (defined as a difference of 0.25 
standard deviations or larger), favorable association between at least 
one critical component and one relevant outcome presented in the logic 
model for the proposed process, product, strategy, or practice.
* * * * *
    Large sample means an analytic sample of 350 or more students (or 
other single analysis units) who were randomly assigned to a treatment 
or control group or 50 or more groups (such as classrooms or schools) 
that contain 10 or more students (or other single analysis units) and 
that were randomly assigned to a treatment or control group.
* * * * *
    Logic model (also referred to as theory of action) means a well-
specified conceptual framework that identifies key components of the 
proposed process, product, strategy, or practice (i.e., the active 
``ingredients'' that are hypothesized to be critical to achieving the 
relevant outcomes) and describes the relationships among the key 
components and outcomes, theoretically and operationally.
* * * * *
    Moderate evidence of effectiveness means one of the following 
conditions is met:
    (i) There is at least one study of the effectiveness of the 
process, product, strategy, or practice being proposed that meets the 
What Works Clearinghouse Evidence Standards without reservations,\3\ 
found a statistically significant favorable impact on a relevant 
outcome (with no statistically significant and overriding unfavorable 
impacts on that outcome for relevant populations in the study or in 
other studies of the intervention reviewed by and reported on by the 
What Works Clearinghouse), and includes a sample that overlaps with the 
populations or settings proposed to receive the process, product, 
strategy, or practice.
---------------------------------------------------------------------------

    \3\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    (ii) There is at least one study of the effectiveness of the 
process, product, strategy, or practice being proposed that meets the 
What Works Clearinghouse Evidence Standards with reservations,\4\ found 
a statistically significant favorable impact on a relevant outcome 
(with no statistically significant and overriding unfavorable impacts 
on that outcome for relevant populations in the study or in other 
studies of the intervention reviewed by and reported on by the What 
Works Clearinghouse), includes a sample that overlaps with the 
populations or settings proposed to receive the process, product, 
strategy, or practice, and includes a large sample and a multi-site 
sample (Note: multiple studies can cumulatively meet the large and 
multi-site sample requirements as long as each study meets the other 
requirements in this paragraph).
---------------------------------------------------------------------------

    \4\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

* * * * *
    Multi-site sample means more than one site, where site can be 
defined as an LEA, locality, or State.
* * * * *
    National level describes the level of scope or effectiveness of a 
process, product, strategy, or practice that is able to be effective in 
a wide variety of communities, including rural and urban areas, as well 
as with different groups (e.g., economically disadvantaged, racial and 
ethnic groups, migrant populations, individuals with disabilities, 
English learners, and individuals of each gender).
* * * * *
    Performance measure means any quantitative indicator, statistic, or 
metric used to gauge program or project performance.
    Performance target means a level of performance that an applicant 
would seek to meet during the course of a project or as a result of a 
project.
* * * * *
    Quasi-experimental design study means a study using a design that 
attempts to approximate an experimental design by identifying a 
comparison group that is similar to the treatment group in important 
respects. These studies, depending on design and implementation, can 
meet What Works Clearinghouse Evidence Standards with reservations \5\ 
(they cannot meet What Works Clearinghouse Evidence Standards without 
reservations).
---------------------------------------------------------------------------

    \5\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    Randomized controlled trial means a study that employs random 
assignment of, for example, students, teachers,

[[Page 49356]]

classrooms, schools, or districts to receive the intervention being 
evaluated (the treatment group) or not to receive the intervention (the 
control group). The estimated effectiveness of the intervention is the 
difference between the average outcome for the treatment group and for 
the control group. These studies, depending on design and 
implementation, can meet What Works Clearinghouse Evidence Standards 
without reservations.\6\
---------------------------------------------------------------------------

    \6\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    Regional level describes the level of scope or effectiveness of a 
process, product, strategy, or practice that is able to serve a variety 
of communities within a State or multiple States, including rural and 
urban areas, as well as with different groups (e.g., economically 
disadvantaged, racial and ethnic groups, migrant populations, 
individuals with disabilities, English learners, and individuals of 
each gender). For an LEA-based project, to be considered a regional-
level project, a process, product, strategy, or practice must serve 
students in more than one LEA, unless the process, product, strategy, 
or practice is implemented in a State in which the State educational 
agency is the sole educational agency for all schools.
    Relevant outcome means the student outcome(s) (or the ultimate 
outcome if not related to students) the proposed process, product, 
strategy, or practice is designed to improve; consistent with the 
specific goals of a program.
* * * * *
    Strong evidence of effectiveness means one of the following 
conditions is met:
    (i) There is at least one study of the effectiveness of the 
process, product, strategy, or practice being proposed that meets the 
What Works Clearinghouse Evidence Standards without reservations,\7\ 
found a statistically significant favorable impact on a relevant 
outcome (with no statistically significant and overriding unfavorable 
impacts on that outcome for relevant populations in the study or in 
other studies of the intervention reviewed by and reported on by the 
What Works Clearinghouse), includes a sample that overlaps with the 
populations and settings proposed to receive the process, product, 
strategy, or practice, and includes a large sample and a multi-site 
sample (Note: multiple studies can cumulatively meet the large and 
multi-site sample requirements as long as each study meets the other 
requirements in this paragraph).
---------------------------------------------------------------------------

    \7\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    (ii) There are at least two studies of the effectiveness of the 
process, product, strategy, or practice being proposed, each of which: 
Meets the What Works Clearinghouse Evidence Standards with 
reservations,\8\ found a statistically significant favorable impact on 
a relevant outcome (with no statistically significant and overriding 
unfavorable impacts on that outcome for relevant populations in the 
studies or in other studies of the intervention reviewed by and 
reported on by the What Works Clearinghouse), includes a sample that 
overlaps with the populations and settings proposed to receive the 
process, product, strategy, or practice, and includes a large sample 
and a multi-site sample.
---------------------------------------------------------------------------

    \8\ What Works Clearinghouse Procedures and Standards Handbook 
(Version 2.1, September 2011), which can currently be found at the 
following link: https://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19.
---------------------------------------------------------------------------

    Strong theory means a rationale for the proposed process, product, 
strategy, or practice that includes a logic model.

[FR Doc. 2013-19390 Filed 8-12-13; 8:45 am]
BILLING CODE 4000-01-P