Patent Quality Metrics for Fiscal Year 2017 and Request for Comments on Improving Patent Quality Measurement, 16142-16145 [2016-06851]

Agencies

• DEPARTMENT OF COMMERCE
• United States Patent and Trademark Office

[Federal Register Volume 81, Number 58 (Friday, March 25, 2016)]
[Notices]
[Pages 16142-16145]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2016-06851]


-----------------------------------------------------------------------

DEPARTMENT OF COMMERCE

United States Patent and Trademark Office

[Docket No.: PTO-P-2016-0006]


Patent Quality Metrics for Fiscal Year 2017 and Request for 
Comments on Improving Patent Quality Measurement

AGENCY: United States Patent and Trademark Office, Commerce.

ACTION: Request for comments.

-----------------------------------------------------------------------

SUMMARY: The United States Patent and Trademark Office (USPTO) is 
revising its patent quality metrics to better identify quality-related 
issues and more clearly communicate its quality measurements to the 
public. The new patent quality metrics are part of the USPTO's Enhanced 
Patent Quality Initiative (EPQI), which was launched in 2015 to engage 
patent stakeholders in enhancing patent quality. As part of the 
Enhanced Patent Quality Initiative, the prior patent quality metrics 
have been reassessed, and new patent quality metrics are now being 
designed for adoption for fiscal year 2017. The new patent quality 
metrics for use in fiscal year 2017 are planned to focus on the 
correctness and clarity of Office actions and will be applied through a 
newly unified review process using a standardized review form that will 
permit data from a significantly larger number of finished product 
quality reviews conducted at the agency to be aggregated and mined for 
information. The USPTO will also mine data on transactions during 
patent prosecution (e.g., the types of actions taken by the applicant 
and the USPTO) to assess examination processes and identify potential 
quality issues requiring further study. The review process will apply 
the new quality metrics and standardized form to increase the accuracy, 
consistency, transparency, clarity, and simplicity of USPTO quality 
review procedures. The USPTO is seeking comment from its stakeholders 
on further improvements to the changes proposed herein.

DATES: Comment Deadline Date: To be ensured of consideration in the 
development of the next iteration of metrics, written comments must be 
received on or before May 24, 2016.

ADDRESSES: Comments should be sent by electronic mail message over the 
Internet addressed to: QualityMetrics2017@uspto.gov. Comments may also 
be submitted by postal mail addressed to: Mail Stop Comments--Patents, 
Commissioner for Patents, P.O. Box 1450, Alexandria, VA, 22313-1450, 
marked to the attention of Michael Cygan, Senior Legal Advisor, Office 
of Patent Legal Administration, Office of the Deputy Commissioner for 
Patent Examination Policy.
    Although comments may be submitted by postal mail, the Office 
prefers to receive comments by electronic mail message over the 
Internet because sharing comments with the public is more easily 
accomplished. Electronic comments are preferred to be submitted in 
plain text, but also may be submitted in ADOBE[supreg] portable 
document format or MICROSOFT WORD[supreg] format. Comments not 
submitted electronically should be submitted on paper in a format that 
facilitates convenient digital scanning into ADOBE[supreg] portable 
document format.
    Timely filed comments will be available for public inspection at 
the Office of the Commissioner for Patents, currently located in 
Madison East, Tenth Floor, 600 Dulany Street, Alexandria, Virginia 
22314. Comments also will be available for viewing via the Office's 
Internet Web site (https://www.uspto.gov/patent/laws-and-regulations/comments-public/comments-improving-patent-quality-measurement). Because 
comments will be made available for public inspection, information that 
the submitter does not desire to make public, such as an address or 
phone number, should not be included in the comments. It would be 
helpful to the USPTO if comments included information about: (1) The 
name and affiliation of the individual responding; and (2) an 
indication of whether the comments represent views of the respondent's 
organization or are the respondent's personal views.

FOR FURTHER INFORMATION CONTACT: Michael T. Cygan, Senior Legal 
Advisor, at (571) 272-7700. Inquiries regarding this notice may be 
directed to the Office of Patent Legal Administration, by telephone at 
(571) 272-7701, or by electronic mail at PatentPractice@uspto.gov.

SUPPLEMENTARY INFORMATION:

I. Background

    Prior to fiscal year 2005, the USPTO quality metric was solely 
directed to the correctness of the final output of the examination 
process that would result in a patent: An allowed application. During 
fiscal years 2005 through 2009, the USPTO expanded its review efforts, 
employing two official metrics of examination quality: (1) The 
correctness of the examiner's determination of allowance of an 
application; and (2) the quality of the actions taken during the

[[Page 16143]]

course of examination. In fiscal year 2010, the first metric was 
modified to include final Office actions, and the second metric was 
modified to focus on the quality of non-final Office actions during 
prosecution. All quality analysis was performed by random selection of 
actions for review by a dedicated Office of Patent Quality Assurance 
(OPQA) team of reviewers, which reviewed each selected action to 
determine whether all required claim rejections were properly made in 
compliance with the patent statutes.
    In 2011, based on stakeholder input, the USPTO adopted a new 
``Composite Quality Metric'' for fiscal years 2011-2015 to track 
performance of those aspects that affect quality and provide a single 
comprehensive metric representing the overall state of patent 
examination quality. The Composite Quality Metric was composed of seven 
total factors: (1) The final disposition review, (2) the in-process 
review, (3) the first action on the merits (FAOM) search review, (4) 
the complete FAOM review, (5) the external quality survey, (6) the 
internal quality survey, and (7) an aggregation of five factors from 
the USPTO's Quality Index Report (QIR). The first four factors 
continued the USPTO's focus on the statutory compliance of work 
product; i.e., the correctness of the Office actions. The first four 
factors were derived from the results of reviews of randomly selected 
Office actions that were conducted by OPQA. These reviews continued the 
USPTO's focus on the statutory compliance of work product; i.e., the 
correctness of the Office actions, with only a basic assessment of 
whether the examiner had sufficiently set forth his or her position for 
any claim rejections. The next two factors were derived from surveys 
that assessed both internal and external stakeholder views on USPTO 
quality. The final factor was based on the USPTO's QIR, which measures 
the degree to which actions in the prosecution of patent applications 
reveal trends indicative of quality concerns and uses a statistical 
analysis of occurrences of certain types of events (e.g., reopening 
after final Office actions, consecutive non-final Office actions, 
consecutive restriction requirements) based on data available through 
the USPTO's Patent Application Locating and Monitoring (PALM) system. 
Performance in the overall Composite Quality Metric and in each of the 
component metric factors has been published on the USPTO dashboard Web 
site on a quarterly basis. The information from the Composite Quality 
Metric has been used to identify trends and areas of concern and to 
target those areas in need of increased training and/or resources.
    On February 5, 2015, the USPTO launched the Enhanced Patent Quality 
Initiative to improve the quality of patents issued by the USPTO. This 
initiative began with a request for public comments on a set of six 
proposals outlined in a Federal Register Notice. See Request for 
Comments on Enhancing Patent Quality, 80 FR 6475 (Feb. 5, 2015). The 
USPTO also held a two-day ``Quality Summit'' on March 25 and 26, 2015, 
at the USPTO headquarters in Alexandria, Virginia, to discuss the 
quality concerns of patent stakeholders and to receive feedback on the 
USPTO's proposals. Following the Quality Summit, the USPTO has 
continued its engagement with the public through numerous roadshows, 
events, and stakeholder meetings to further refine the steps that may 
be taken to improve quality.
    The Enhanced Patent Quality Initiative targets three pillars of 
patent quality: (1) Excellence in work products; (2) excellence in 
measuring patent quality; and (3) excellence in customer service. In 
furtherance of the second pillar of patent quality, the USPTO is 
focusing on improving the internal metrics used to evaluate patent 
examination quality and on improving the communication of its patent 
examination quality measurements to the public. Through this 
initiative, the USPTO has received numerous comments on establishing 
appropriate quality metrics. The USPTO has considered all of the 
comments received through the Summit, the Federal Register Notice, and 
numerous quality outreach events. Based on the information received to 
date, the USPTO has identified key aspects of quality measurement 
essential to developing more effective quality metrics.
    First, the clarity of the examiner's determinations and the 
rationale underlying the decisions made in Office actions is an 
important part of overall patent examination quality and should be 
emphasized in reviews of USPTO work product. Second, individual metrics 
that clearly reflect individual aspects of USPTO work product would 
better communicate patent quality than a single quality composite 
number that combines scores from unrelated sources such as surveys, 
procedural efficiency statistics, and substantive patentability 
compliance reviews. Third, improving the granularity of work product 
quality measurement to monitor compliance with each statutory provision 
and enable meaningful data at the work group and art unit level is 
highly desirable for providing targeted training resulting in greater 
consistency. Fourth, monitoring the process of examination, i.e., the 
type and number of actions taken during prosecution as reflected in the 
QIR, remains a high priority that is best used to spot unusual trends 
or occurrences that deserve further attention. Lastly, capturing a 
larger number of finished product quality reviews conducted at the 
agency and using a standardized review form will lead to a 
significantly greater number of data points, which will allow for 
greater consistency in the review of application quality within the 
Patents Organization. More information on the public comments received 
on the metrics, and how those are being used to identify improvements 
to the metrics, is available at https://www.uspto.gov/patent/initiatives/quality-metrics. In view of these guideposts, a new set of 
metrics is now being proposed to incorporate these and other 
improvements to the collection of data and reporting of metrics.

II. Improving Measurement of Patent Examination Quality

    As the next step in advancing the second pillar of the Enhanced 
Patent Quality Initiative, the proposed fiscal year 2017 patent quality 
metrics refocus the USPTO's measurement of the quality of the work 
products produced from first Office action through final disposition. 
The proposed metrics continue to assess the correctness of an 
examiner's determinations in a given Office action with increased 
attention on assessing whether the examiner clearly set forth his or 
her reasoning in a given Office action. In addition, the Office will 
continue to review the transactions taken during patent prosecution 
through the QIR, but this information will be used to identify the need 
for further investigation rather than being measured against a goal. 
Additionally, the USPTO is changing its reporting of the quality 
metrics to provide simpler and clearer communication of results to the 
public.

A. Measurement of Statutory Compliance and Clarity in Work Products

    The patent quality metrics of work product proposed here for fiscal 
year 2017 provide a tighter focus on measuring two foundational 
characteristics of patent examination: Statutory compliance and clarity 
of decision making in Office actions. These proposed patent quality 
metrics continue to measure correctness of actions in terms of their 
compliance

[[Page 16144]]

with each of the statutory requirements for issuance of a patent. To 
this end, a sampling of Office actions will continue to be reviewed 
both for improperly made rejections and for failure to make rejections 
where required by statute. The substantive review items will also 
include other items, for example, the propriety of the examiner's 
search, any interpretation of claim language under 35 U.S.C 112(f), any 
determination that an action is made final, any restriction or election 
of species requirement.
    Furthermore, the new metrics greatly enhance the review of the 
clarity of the components of Office actions by including new clarity 
review items specifically designed for each of the substantive 
patentability determinations made in Office actions. For example, when 
reviewing an Office action containing an obviousness rejection under 35 
U.S.C. 103, the review items consider not only whether the obviousness 
rejection was proper, but also whether the statement of the rejection 
mapped the elements identified in the prior art to the claim 
limitations, and whether the statement of the rejection explained the 
reasons for the rejection in a clear manner. The new clarity review 
items will also include, for example, items directed to the sufficiency 
of the recordation of any interview and the propriety of any reasons 
for allowance of an application.
    For fiscal year 2017, the USPTO is proposing to capture the 
correctness and clarity review items with a single standardized review 
form as a repository for all of the review items, replacing the review-
specific forms used in the 2011-2015 Composite Quality Metric. The 
review questions on such a standardized form, colloquially referred to 
as the ``Master Review Form,'' is planned to be used by all USPTO 
reviewers for finished product quality reviews of actions at every 
stage of prosecution. This Master Review Form will contain the above-
described criteria for recording correctness for each of the 
substantive patentability requirements and for recording the clarity of 
each of those decisions and the supporting rationales set forth in the 
Office action under review. The full list of correctness and clarity 
items in the draft proposed version of the Master Review Form is 
available for viewing at https://www.uspto.gov/patentquality. The USPTO 
welcomes and appreciates feedback on the elements of this form through 
this notice, and will use the input to help finalize the Master Review 
Form that will be deployed throughout the USPTO in fiscal year 2017.
    This draft proposed ``Master Review Form'' was developed as part of 
the Clarity and Correctness Data Capture program, which is part of the 
USPTO's Enhanced Patent Quality Initiative. The Clarity and Correctness 
Data Capture Program has been instituted to better capture the data 
produced through the different types of reviews within the Patents 
Organization. Historically, reviews have been performed not only by the 
quality assurance team, but also by other Technology Center personnel, 
with each reviewing area setting its own reviewing criteria. Moreover, 
the only work product reviews recorded for identification of trends 
were those undertaken by the Office of Patent Quality Assurance. The 
Master Review Form is designed to provide standardized reviewing 
criteria for quality reviews of finished work product. Through 
application of standardized reviewing criteria, the USPTO can better 
leverage the results from the many levels of review conducted at the 
agency. The improvements to the data capture process will enable 
meaningful data analysis at a more granular level than previously 
possible, permitting valid inferences to be drawn at the workgroup and 
art unit levels. Through this process, the USPTO and the stakeholders 
in the patent system will be able to gain a greater understanding of 
the state of patent prosecution and to work better together towards its 
improvement.

B. Measurement of Transactions During Patent Prosecution

    A further aspect of the new patent quality metrics will be the 
leveraging of the data representing the thousands of transactions made 
by the USPTO during prosecution to reveal information on the quality of 
the patent prosecution process itself. Transactions during prosecution, 
such as restrictions, first Office actions, and allowances, are 
monitored through the USPTO's PALM system. The USPTO monitors many of 
these transactions through its QIR. Since 2011, the USPTO has included 
some of these transactions, such as the number of occurrences of 
consecutive non-final rejections, as part of its reported quality data. 
For the proposed 2017 quality metrics, transactional data from the QIR 
will be used to identify information that can be used to prevent 
reopening of prosecution, reduce rework, and improve the consistency of 
decision making throughout the USPTO. Key indicators of the efficiency 
of prosecution will be instances of reopening of prosecution and 
repeated non-final Office actions, as well as other instances of rework 
(e.g., consecutive final Office actions, consecutive restrictions). 
These indicators do not, by themselves, provide a numerical measure of 
quality. Rather, these indicators will reveal trends and outlier 
behavior that will draw attention to potential quality concerns.

C. Clearer Reporting of the Metrics

    In presenting the results of the quality data, the USPTO will seek 
to further improve the usefulness and transparency of our quality 
reporting and to communicate the results in a clear and simple manner. 
The 2011-2015 Composite Quality Metric, which combined seven different 
quality variables into a single composite number, will be discontinued. 
The Quality Index Report will be used to identify potential areas of 
concern, rather than as providing a single, reportable number. While 
internal and external surveys will still be performed, the results will 
not be part of the quality metric, but instead will serve as 
independent checks on the quality metrics.

D. Refinement of Proposed Quality Metrics in FY 2016

    Fiscal year 2016 will represent a transitional period for the 
quality metrics, emphasizing the fine-tuning of the fiscal year 2017 
patent quality metrics. The USPTO will test and refine its proposed 
Master Review Form. This Master Review Form will contain new items, 
such as additional clarity review items, that will require a period of 
data collection to create numerical baselines for these items. The 
Master Review Form will initially be used in targeted reviews to 
determine the effectiveness of each individual clarity and correctness 
review item. The transactional data from the QIR will also be reviewed 
during 2016 to optimize the data analysis therein. Stakeholder comments 
on the Master Review Form in response to this notice will also form an 
important part of the process of optimizing the components of the 
patent quality metrics. During this transitional period, the 
information gleaned during fiscal year 2016 will be used to produce a 
finalized set of quality metrics for fiscal year 2017 that will 
represent the next phase of quality measurement, analysis, tracking, 
and reporting at the USPTO.

III. Feedback Sought on Improving Metrics of Patent Examination Quality

    The USPTO seeks input and comments from the public through this 
notice and through public outreach on the following:
    (1) Is the USPTO moving in the right direction by choosing to focus 
on two

[[Page 16145]]

core metrics: A work product metric representing correctness of 
actions, and a clarity metric that more thoroughly explores the 
sufficiency of the examiner's reasoning in an Office action, thus 
moving away from the prior goal-based quality ``score'' that reflected 
not only quality of work product but also results of surveys, used to 
discover both internal and external stakeholder opinions, and QIR 
process indicators? Which of the proposed clarity and correctness 
review items in the proposed standardized ``Master Review Form,'' 
available at https://www.uspto.gov/patentquality, should be used as the 
key drivers of patent examination quality metrics?
    (2) How can patent metrics best provide objective, rather than 
subjective, measurements of quality-related features in clarity and 
correctness reviews?
    In addition to the three questions posed above, the USPTO welcomes 
comments on any and all areas of quality measurement. Suggestions for 
rephrased or additional quality metrics review items, especially 
clarity indicators, are welcomed. The USPTO will consider all submitted 
comments as it develops the next iteration of quality metrics.
    For the most current information on this and other patent quality 
initiatives, please visit the Enhanced Patent Quality Initiative micro 
site at https://www.uspto.gov/patentquality.

    Dated: March 22, 2016.
Michelle K. Lee,
Under Secretary of Commerce for Intellectual Property and Director of 
the United States Patent and Trademark Office.
[FR Doc. 2016-06851 Filed 3-24-16; 8:45 am]
 BILLING CODE 3510-16-P