Agency Information Collection Activities: Proposed Collection; Comment Request, 6194-6197 [2020-02057]

Agencies

• Agency for Healthcare Research and Quality
• DEPARTMENT OF HEALTH AND HUMAN SERVICES

[Federal Register Volume 85, Number 23 (Tuesday, February 4, 2020)]
[Notices]
[Pages 6194-6197]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2020-02057]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Agency for Healthcare Research and Quality


Agency Information Collection Activities: Proposed Collection; 
Comment Request

AGENCY: Agency for Healthcare Research and Quality, HHS.

ACTION: Notice

-----------------------------------------------------------------------

SUMMARY: This notice announces the intention of the Agency for 
Healthcare Research and Quality (AHRQ) to request that the Office of 
Management and Budget (OMB) approve the proposed information collection 
project ``Evaluating the Implementation of Products by Learning Health 
Systems to Inform and Encourage Use of AHRQ Evidence Reports.''

DATES: Comments on this notice must be received by 60 days after date 
of publication.

ADDRESSES: Written comments should be submitted to: Doris Lefkowitz, 
Reports Clearance Officer, AHRQ, by email at 
[email protected].
    Copies of the proposed collection plans, data collection 
instruments, and specific details on the estimated burden can be 
obtained from the AHRQ Reports Clearance Officer.

FOR FURTHER INFORMATION CONTACT: Doris Lefkowitz, AHRQ Reports 
Clearance Officer, (301) 427-1477, or by email at 
[email protected].

SUPPLEMENTARY INFORMATION: 

Proposed Project

Evaluating the Implementation of Products by Learning Health Systems To 
Inform and Encourage Use of AHRQ Evidence Reports

    AHRQ's Evidence-based Practice Center (EPC) Program has 20 years of 
experience in synthesizing research to inform evidence-based health 
care practice, delivery, policies, and research. The AHRQ EPC program 
is committed to partnering with

[[Page 6195]]

organizations to make sure its evidence reports can be used in 
practice. Historically, most of its evidence reports have been used by 
clinical professional organizations to support the development of 
clinical practice guidelines or Federal agencies to inform their 
program planning and research priorities. To improve the uptake and 
relevance of the AHRQ EPC's evidence reports, specifically for health 
systems, AHRQ has contracted with the American Institutes for Research 
(AIR) to obtain feedback from learning health systems (LHSs) to assist 
the AHRQ EPC program in developing and disseminating evidence reports 
that can be used to improve the quality and effectiveness of patient 
care.
    Even if an EPC evidence report topic addresses LHS-specific 
evidence needs, the density of the information in an evidence report 
may preclude its easy review by busy LHS leaders and decisionmakers. 
AHRQ understands that to facilitate use by LHSs, complex evidence 
reports must be translated into a format that promotes LHS evidence-
based decision making and can be contextualized within each LHS' own 
system-generated evidence. Such translational products, for the 
purposes of this notice, are referred to simply as ``products.''
    The purpose of this information collection is to support a process 
evaluation of use and implementation of two such products into LHS 
decisionmaking processes, workflows, and clinical care. The evaluation 
has the following goals:
    1. Document how LHSs prioritize filling evidence gaps, make 
decisions about using evidence, and implement tools to support and 
promote evidence use in clinical care.
    2. Assess the contextual factors that may influence implementation 
success; associated implementation resources, barriers and 
facilitators; and satisfaction of LHS leaders and clinical staff.
    3. Provide the AHRQ EPC program with necessary insights about the 
perspectives, needs, and preferences of LHS leaders and clinical staff 
as related to decisions and implementation of products into practice.
    This study is being conducted by AHRQ through its contractor, the 
American Institutes for Research (AIR), pursuant to AHRQ's statutory 
authority to conduct and support research on, and disseminate 
information on, health care and on systems for the delivery of such 
care, including activities with respect to the quality, effectiveness, 
efficiency, appropriateness and value of healthcare services. 42 U.S.C 
299a(a)(1).

Method of Collection

    To achieve the goals of this project, the following data collection 
activities will be implemented:
    1. Key informant interviews with health system leaders, clinicians 
and staff; and
    2. compilation and coding of notes from ``implementation support'' 
meetings (``check-ins'') between an implementation facilitator and site 
champions who are implementing the products.

Brief Background on the Products To Be Implemented by LHSs in This 
Study

    AHRQ is funding the development of two products that are 
specifically intended to make the findings from EPC evidence reports 
more accessible and usable by health systems. These are the products 
that will be offered to LHSs for potential implementation during this 
project. They include a ``triage tool'' and a ``data visualization 
tool'' that have been designed to support LHS use of AHRQ evidence 
reports. The LHS triage tool presents high-level results of evidence 
reports that enable leaders within LHSs to quickly understand the 
relevance of the reports to their organization, share high-level 
information with key stakeholders (e.g., healthcare executives), and 
link to more granular data from the report. The data visualization tool 
presents data from the evidence review and individual studies in a 
dynamic, interactive website. The evaluation will capture the 
anticipated variation in how the LHS might use the products and the 
unique experience of LHSs.

Key Informant Interviews

    There will be two rounds of key informant interviews: (1) In-person 
preliminary interviews will be conducted early in the implementation 
period (months 1-3) with LHS leaders and clinicians and will focus on 
health systems' rationale for selecting each product and early 
experiences with its roll-out into practice; (2) remote follow-up 
interviews will be conducted via telephone later in the implementation 
period (months 10-11) with two sets of stakeholders: (a) LHS leaders 
and (b) clinicians/staff (hereafter, ``clinical staff'') actively 
implementing the product. These follow-up interviews will focus on 
health systems' experiences implementing their selected product(s). All 
interviews (preliminary and follow-up) will be 60 minutes in duration, 
recorded with permission of the key informants, and transcribed for 
analysis. Up to 88 total interviews will be conducted across the two 
rounds of key informant interviews. Assuming the same LHS leaders 
participate in the preliminary and follow-up interviews, the key 
informant interviews will involve 4-5 LHS leaders and clinical staff 
from each of the eleven LHSs implementing the study. Additional detail 
about the information collection components is provided below.

    1. In-person preliminary interviews. The preliminary interviews 
will include 2-3 LHS leaders/decisionmakers at each of eleven 
implementation sites for a maximum of 33 interviews in the first 
round of data collection. The interviews will be conducted during 
implementation site visits that are occurring early in the project 
to support the health systems' testing and/or roll out of the 
products into clinical workflows. Specific topics explored in the 
preliminary interviews include LHSs' decision to participate in 
implementation, decision considerations for the selected product, 
experiences leading the implementation, and early experiences and 
perceptions of the selected product(s). To limit respondent burden, 
we will use the implementation site visits as an opportunity for 
conducting the preliminary interviews, thereby limiting the need to 
schedule additional time with respondents for a phone interview. If 
a respondent has limited availability during the site visit, 
however, we may need to do the preliminary interview remotely or 
substitute the respondent with another qualified staff member who is 
available during the implementation site visit.
    2. Remote follow-up interviews. The follow-up interviews will 
include the 2-3 LHS leaders/decisionmakers from the preliminary 
interviews (maximim n = 33), along with 2 additional clinical staff 
(n = 22) at each of eleven implementation sites for a maximum of 55 
follow-up interviews. Specific topics explored in the follow-up 
interviews include LHS leaders' and clinical staff's experiences 
with each product as well as their perceptions of the relative 
advantage, acceptability/compatibility, appropriateness, and 
feasibility of using the product; implementation fidelity (i.e., if 
the implementation went as planned), reach, barriers and 
facilitators, and associated costs; any outcomes of implementing the 
product (e.g., achieved any intended systemic changes); and likely 
sustainability of continuing to use the product in practice.

    The two sets of in-depth qualitative interviews will allow for a 
nuanced exploration of both what LHSs value about the products and what 
it takes to successfully implement such tools into practice. The 
research on implementation and uptake of products to promote use of 
evidence in LHS settings is sparse, thus it is important to use a data 
collection strategy for the evaluation that will yield rich information 
about the experience of health systems, LHS decisionmakers, and the 
staff implementing the tools into practice. A quantitative survey would 
not yield the depth of individual

[[Page 6196]]

feedback that is needed to capture the experience of implementing these 
tools and the unique contexts of the health systems. Thus, interviews 
are the preferred method of systematically collecting these data.

Implementation Support Meetings/``Check-ins''

    In addition to key informant interviews, which will be conducted 
only at the beginning and end of implementation, AHRQ will gather 
information throughout the implementation period by using monthly 
implementation support meetings between implementation facilitators and 
site champions as an ongoing opportunity to ask key questions about 
implementation progress. Although the primary goal of these check-in 
meetings is to provide technical assistance with implementation and 
recommendations for handling emergent challenges in the implementation 
process, they will also be a source of rich information for the 
evaluation. Because these meetings occur in real time as the 
implementation unfolds, they will reduce the potential biases (e.g., 
selective memory, recency effects, forgetting details about key events 
and their sequence) associated with only collecting data at the 
beginning or end of the implementation period.
    These check-in meetings will occur by telephone and are intended to 
monitor implementation progress, provide support to health systems, and 
discuss next steps. AIR implementation facilitators for each site will 
schedule telephone conference calls with site champions (N = 11), 
during which structured notes will be taken. These notes will be 
supplemented with relevant information from other touchpoints between 
the facilitators and champions (e.g., ad hoc calls, email exchanges, 
and voluntary participation in monthly shared learning events) as they 
naturally occur. Notetakers will capture and document information 
related to key implementation domains as these topics arise in check-in 
meetings and other facilitator/champion encounters throughout 
implementation.

Estimated Annual Respondent Burden

    Exhibit 1 shows the total estimated annualized burden of 214.5 
hours for the two rounds of key informant interviews and implementation 
``check-ins'' combined. For the key informant interviews (totaling 154 
hours), burden is included for: (1) LHS leaders/decisionmakers 
participating in the preliminary interviews (a maximum of 33 hours), 
(2) LHS leaders/decisionmakers participating in the follow-up 
interviews (a maximum of 33 hours), (3) clinical staff participating in 
the follow-up interviews (a maximum of 22 hours), (4) interviewee 
review of materials, consent forms, and logistics in advance of their 
respective interviews (i.e., 16.5 + 5.5 = 22 hours) and (5) time for 
designated LHS staff (e.g., the LHS member, a designated site liaison, 
selected interviewees) to recommend key informants, coordinate 
implementation support, and help with scheduling of in-person 
preliminary interviews and remote follow-up interviews (44 hours). Also 
included in Exhibit 1 is the estimated annualized burden hours for 
monthly check-ins between implementation facilitators and LHS champions 
for informal technical assistance support and the quick status probes 
on implementation progress (a maximum of 60.5 hours). These annualized 
burden estimates for the key informant interviews and the coaching 
sessions are further explained below.

Key Informant Interviews: Expanded Detail on Burden Estimates

    We estimate 1 hour for each key informant interview for: (1) LHS 
leaders/decisionmakers participating in the preliminary interviews (a 
maximum of 33 hours), (2) LHS leaders/decisionmakers participating in 
the follow-up interviews (a maximum of 33 hours), (3) clinical staff 
participating in the follow-up interviews (a maximum of 22 hours), 
(Total interview burden = 1.00 hour x maximum of 88 interviews = 88 
hours). We estimate an additional 15 minutes (0.25 hours) will be 
needed for key informants to prepare for their respective interview(s) 
(Total interview preparation burden = 0.25 hours x maximum of 88 
interviews = 22 hours; of which 16.5 hours is for leaders/
decisionmakers to prepare for both preliminary and follow-up interviews 
and 5.5 is for clinical staff to prepare for their participation in the 
follow-up interviews only). Finally we estimate time for LHS leaders 
and staff to identify interview candidates, facilitate recruitment, 
coordinate implementation support, and assist with interview scheduling 
(4.00 hours per each of 11 LHSs; Total staff assistance burden = 4.00 
hours x 11 sites = 44 hours). The ``staff assistance'' burden involves 
the following:
     In each of the eleven LHS organizations implementing the 
product(s), the LHS member (and/or site liaison/champion) will identify 
prospective key informants (i.e., other LHS leaders/decisionmakers and 
appropriate clinical staff), with additional key informants 
subsequently identified through snowball sampling.
     Designated LHS staff (i.e., LHS member, designee and/or 
site liaison/champion) will provide needed contact information to the 
AIR evaluation team for outreach and recruitment of the prospective key 
informant interview candidates, assist with interview scheduling, and 
coordinate implementation support with the AIR team.
    We will develop standardized email messages to reach out to 
interview candidates and a written overview of the project, the 
evaluation, and the purpose of the interview. We will coordinate 
scheduling of both the implementation support check-ins and the 60-
minute interviews at the most convenient time, considering the needs of 
the LHS leadership and staff. For the preliminary interviews, if 
prospective interviewees are not available during our site visit, we 
will ask for suggestions of other LHS staff who meet our recruitment 
criteria or arrange a telephone interview, if needed.

Implementation Support Meetings/Check-Ins: Expanded Detail on Burden 
Estimates

    We estimate 60.5 hours for the monthly check-ins between 
implementation facilitators and LHS champions. This includes an average 
of 30 minutes of implementation support/check-in meetings per each of 
the 11 LHSs for each month of implementation (11 months). (11 months x 
0.5 hours = 5.5 hours). Across LHSs, the estimated burden associated 
with check-ins is approximately 61 hours across the implementation 
period (5.5 hours x 11 LHSs = 60.5 hours).

                                  Exhibit 1--Estimated Annualized Burden Hours
----------------------------------------------------------------------------------------------------------------
                                                                     Number of
                    Form name                        Number of     responses per     Hours per     Total burden
                                                   respondents *    respondent       response          hours
----------------------------------------------------------------------------------------------------------------
In-person preliminary interviews with LHS                  ** 33               1            1.00              33
 leaders/decisionmakers.........................

[[Page 6197]]

 
Remote follow-up interviews with LHS leaders/              ** 33               1            1.00              33
 decisionmakers.................................
Remote follow-up interviews with clinical staff.              22               1            1.00              22
Review of materials prior to BOTH preliminary                 33               2            0.25            16.5
 and follow-up interviews--LHS leaders/
 decisionmakers.................................
Review of materials prior to interviews--                     22               1            0.25             5.5
 clinical staff.................................
Interview scheduling and other staff assistance.              11               1            4.00              44
Implementation check-ins: Brief monthly                       11              11             0.5            60.5
 implementation progress checks, documented for
 the evaluation as structured notes on
 implementation topics naturally occurring in
 coach/champion encounters......................
rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
    Total.......................................  ..............  ..............  ..............       *** 214.5
----------------------------------------------------------------------------------------------------------------
* The numbers in this column give the maximum number of respondents for each listed activity based on a range in
  the number of recruits per site (e.g., ``2-3 LHS leaders/decisionmakers''). The balance may shift some between
  LHS leaders/decisionmakers and clinical staff depending on implementation team and leadership composition at
  each site. In any case, 88 interviews (33+33+22=88) is a maximum possible in the event each of the 11 sites
  contributes 3 ``LHS leaders/decsionmakers'' (likely the same people for preliminary and follow-up interviews)
  and 2 additional clinical staff (for follow-up interviews only) as key informants. It is more likely that the
  total number of interviews will be around 80.
** These are likely to be the same 33 respondents in both preliminary and follow-up interviews.
*** Total maximum burdened hours estimate based on maximum of 88 interviews.

    Costs associated with the estimated annualized burden hours are 
provided in Exhibit 2.

                                   Exhibit 2--Estimated Annualized Cost Burden
----------------------------------------------------------------------------------------------------------------
                                                                                      Average
                    Form name                        Number of     Total burden     hourly wage     Total cost
                                                   respondents *       hours          rate **         burden
----------------------------------------------------------------------------------------------------------------
In-person preliminary interviews with leaders/                33              33      \a\ $94.47       $3,117.51
 decisionmakers.................................
Remote follow-up interviews with leaders/                     33              33       \a\ 94.47        3,117.51
 decisionmakers.................................
Remote follow-up interviews with clinical staff.              22              22       \b\ 52.13        1,146.86
Review of materials prior to BOTH preliminary                 33            16.5       \a\ 94.47        1,558.76
 and follow-up interviews--LHS leaders/
 decisionmakers.................................
Review of materials prior to interviews--                     22             5.5       \b\ 52.13          286.72
 clinical staff.................................
Interview scheduling and other staff assistance               11              44       \c\ 20.34          894.96
 \c\............................................
Implementation check-ins (documented for the                  11            60.5       \a\ 94.47        5,715.44
 evaluation as structured notes on
 implementation progress).......................
                                                 ---------------------------------------------------------------
    Total.......................................  ..............  ..............  ..............       15,837.76
----------------------------------------------------------------------------------------------------------------
* The numbers in this column give the maximum number of respondents for each listed activity based on a range in
  the number of recruits per site (e.g., ``2-3 LHS leaders/decisionmakers''). As noted in the comment to Exhibit
  1, the balance may shift some between LHS leaders/decisionmakers and clinical staff depending on
  implementation team and leadership composition at each site. In any case, 88 interviews (33+33+22=88) is a
  maximum possible.
** National Compensation Survey: Occupational wages in the United States May 2018 ``U.S. Department of Labor,
  Bureau of Labor Statistics.''
\a\ Based on the mean wages for Internists, General. 29-1063; annual salary of $196,490.
\b\ Based on the mean wages for Physician Assistants, 29-1071; annual salary of $108,430.
\c\ Based on the mean wages for Secretaries and Administrative Assistants, 43-6010; annual salary of $42,320.

Request for Comments

    In accordance with the Paperwork Reduction Act, comments on AHRQ's 
information collection are requested with regard to any of the 
following: (a) Whether the proposed collection of information is 
necessary for the proper performance of AHRQ's health care research and 
health care information dissemination functions, including whether the 
information will have practical utility; (b) the accuracy of AHRQ's 
estimate of burden (including hours and costs) of the proposed 
collection(s) of information; (c) ways to enhance the quality, utility 
and clarity of the information to be collected; and (d) ways to 
minimize the burden of the collection of information upon the 
respondents, including the use of automated collection techniques or 
other forms of information technology.
    Comments submitted in response to this notice will be summarized 
and included in the Agency's subsequent request for OMB approval of the 
proposed information collection. All comments will become a matter of 
public record.

    Dated: January 29, 2020.
Virginia L. Mackay-Smith,
Associate Director.
[FR Doc. 2020-02057 Filed 2-3-20; 8:45 am]
 BILLING CODE 4160-90-P