Statewide Data Indicators for Child and Family Services Reviews: Request for Public Comment, 60874-60876 [2018-25835]
Download as PDF
<![CDATA[
60874
Federal Register / Vol. 83, No. 228 / Tuesday, November 27, 2018 / Notices
amozie on DSK3GDR082PROD with NOTICES1
Form Number: CMS–179 (OMB control
number 0938–0193); Frequency:
Occasionally; Affected Public: State,
Local, and Tribal Governments; Number
of Respondents: 56; Total Annual
Responses: 1,120; Total Annual Hours:
22,400. (For policy questions regarding
this collection contact Annette Pearson
at 410–786–6958.)
2. Type of Information Collection
Request: Extension; Title of Information
Collection: Home Health Change of Care
Notice; Use: The purpose of the Home
Health Change of Care Notice (HHCCN)
is to notify original Medicare
beneficiaries receiving home health care
benefits of plan of care changes. Home
health agencies (HHAs) are required to
provide written notice to Original
Medicare beneficiaries under various
circumstances involving the reduction
or termination of items and/or services
consistent with Home Health Agencies
Conditions of Participation (COPs).
The home health COP requirements
are set forth in § 1891[42 U.S.C.
1395bbb] of the Social Security Act (the
Act). The implementing regulations
under 42 CFR 484.10(c) specify that
Medicare patients receiving HHA
services have rights. The patient has the
right to be informed, in advance about
the care to be furnished, and of any
changes in the care to be furnished. The
HHA must advise the patient in advance
of the disciplines that will furnish care,
and the frequency of visits proposed to
be furnished. The HHA must advise the
patient in advance of any change in the
plan of care before the change is made.’’
Notification is required for covered
and non-covered services listed in the
plan of care (POC). The beneficiary will
use the information provided to decide
whether or not to pursue alternative
options to continue receiving the care
noted on the HHCCN. Form Number:
CMS–10280 (OMB control number:
0938–1196); Frequency: Yearly; Affected
Public: Private Sector (Business or other
for-profits, Not-for-Profit Institutions);
Number of Respondents: 12,149; Total
Annual Responses: 13,640,524; Total
Annual Hours: 908,459. (For policy
questions regarding this collection
contact Jennifer McCormick at 410–786–
2852.)
Dated: November 21, 2018.
William N. Parham, III,
Director, Paperwork Reduction Staff, Office
of Strategic Operations and Regulatory
Affairs.
[FR Doc. 2018–25858 Filed 11–26–18; 8:45 am]
BILLING CODE 4120–01–P
VerDate Sep<11>2014
17:45 Nov 26, 2018
Jkt 247001
DEPARTMENT OF HEALTH AND
HUMAN SERVICES
Centers for Medicare & Medicaid
Services
[Document Identifier: CMS–179]
Agency Information Collection
Activities: Proposed Collection;
Comment Request
ACTION:
Notice; partial withdrawal.
On October 1, 2018 (83 FR
49389), the Centers for Medicare &
Medicaid Services (CMS) published a
notice entitled, ‘‘Agency Information
Collection Activities: Proposed
Collection; Comment Request.’’ That
notice invited public comments on three
separate information collection requests
specific to document identifiers: CMS–
10142, CMS–R–262, and CMS–179. Due
to technical difficulties associated with
CMS–179 (OMB control number: 0938–
0193) titled ‘‘Medicaid State Plan Base
Plan Pages,’’ through the publication of
this document we are withdrawing
CMS–179 from our October 1, 2018,
information collection request. While
the technical issues associated with
CMS–179 have recently been resolved,
because of the delay we are publishing
a new 60-day information collection
request elsewhere in today’s Federal
Register. The November 30, 2018,
comment due date for the remaining
two collections [CMS–10142 (OMB
control number: 0938–0944) titled, ‘‘Bid
Pricing Tool (BPT) for Medicare
Advantage (MA) Plans and Prescription
Drug Plans (PDP)’’ and CMS–R–262
(OMB control number: 0938–0763),
titled ‘‘Contract Year 2020 Plan Benefit
Package (PBP) Software and Formulary
Submission’’] remains in effect without
change.
SUMMARY:
Dated: November 21, 2018.
William N. Parham, III,
Director, Paperwork Reduction Staff, Office
of Strategic Operations and Regulatory
Affairs.
[FR Doc. 2018–25840 Filed 11–26–18; 8:45 am]
BILLING CODE 4120–01–P
DEPARTMENT OF HEALTH AND
HUMAN SERVICES
Administration for Children and
Families
Statewide Data Indicators for Child and
Family Services Reviews: Request for
Public Comment
Children’s Bureau (CB),
Administration for Children and
Families (ACF), Administration on
AGENCY:
PO 00000
Frm 00054
Fmt 4703
Sfmt 4703
Children, Youth and Families (ACYF),
Department of Health and Human
Services (HHS).
Request for public comment on
revised syntax used to calculate
Statewide Data Indicators for Child and
Family Services Reviews (CFSRs).
ACTION:
In October 2014, the
Administration for Children and
Families (ACF) published a notice in the
Federal Register (79 FR 61241) with the
final plan to replace the statewide data
indicators used to determine a state’s
substantial conformity with titles IV–B
and IV–E of the Social Security Act
through the Child and Family Services
Reviews (CFSRs). In May 2015, ACF
published a notice in the Federal
Register (80 FR 27263) to correct some
of the calculations and language used in
the October 2014 notice. After the May
2015 notice, additional technical errors
in the syntax and formulation of the
statewide data indicators were
identified. Some of those errors were
discovered by states and other
interested parties that operationalized
the indicators for continuous quality
improvement. Based on the amount of
time required to complete a
comprehensive review of the syntax,
make revisions and validate the
accuracy of the calculations; the
Children’s Bureau (CB) decided to
suspend use of the indicators in
determinations of substantial
conformity during the third round of
CFSRs. In October 2016, CB published
CFSR Technical Bulletin #9 to inform
states of the decision to limit use of the
CFSR statewide data indicators and
national performance to context
information for round three of CFSRs.
Since then, CB has thoroughly
reviewed, revised, tested, and obtained
independent review and validation of
the syntax used to perform data quality
checks and calculate state performance
on the statewide data indicators. Before
CB finalizes the revised syntax, and
publishes revised performance results
for all states and revised national
standards, we invite state child welfare
agencies, partner organizations, and the
public to review, test, and provide
comments on the revised syntax used to
perform data quality checks and
calculate observed performance on the
statewide data indicators.
SUMMARY:
Comments are due by February
25, 2019.
DATES:
You may send comments,
identified by Docket Number ACF–
2018–0007, by one of the following
methods:
ADDRESSES:
E:\FR\FM\27NON1.SGM
27NON1
Federal Register / Vol. 83, No. 228 / Tuesday, November 27, 2018 / Notices
• Federal eRulemaking Portal:
https://www.regulations.gov. Follow the
instructions for submitting comments.
• Email: CBComments@acf.hhs.gov.
Include Docket Number ACF–2018–
0007 in subject line of the message.
FOR FURTHER INFORMATION CONTACT:
James Gregory, (202) 260–4668,
James.Gregory@acf.hhs.gov.
SUPPLEMENTARY INFORMATION: This
Federal Register announcement has
four sections: Background describes the
purpose, history, and authority for the
CFSRs and the use of statewide data
indicators; Overview of Review and
Validation of Syntax to Calculate CFSR
3 Statewide Data Indicator Performance
describes the internal and independent
review and validation process and
findings; Supporting Documents
provides a brief description of five
resources available to aid interested
parties in the syntax review process;
and Invitation to Comment wherein we
solicit public comment on the revised
syntax that is pending finalization.
amozie on DSK3GDR082PROD with NOTICES1
Background
CB implemented the CFSRs in 2001 in
response to a mandate in the Social
Security Amendments of 1994. The
reviews are required for CB to determine
if such programs are in substantial
conformity with title IV–B and IV–E
plan requirements. The review process,
as regulated at 45 CFR 1355.31–37, grew
out of extensive consultation with
interested groups, individuals, and
experts in the field of child welfare and
related areas.
The CFSRs enable CB to: (1) Ensure
conformity with federal child welfare
requirements; (2) determine what is
actually happening to children and
families as they are engaged in child
welfare services; and (3) assist states to
enhance their capacity to help children
and families achieve positive outcomes.
CB conducts the reviews in partnership
with state child welfare agency staff and
other partners and stakeholders
involved in the provision of child
welfare services. We have structured the
reviews to help states identify strengths
as well as areas needing improvement
within their agencies and programs.
We use the CFSR to assess state
performance on seven outcomes and
seven systemic factors. The seven
outcomes focus on key items measuring
safety, permanency, and well-being. The
seven systemic factors focus on key state
plan requirements of titles IV–B and IV–
E that provide a foundation for child
outcomes. If we determine a state has
not achieved substantial conformity in
one or more of the areas assessed in the
review, the state is required to develop
VerDate Sep<11>2014
17:45 Nov 26, 2018
Jkt 247001
and implement a program improvement
plan addressing the areas of
nonconformity within 2 years. CB
supports the states with technical
assistance and monitors implementation
of their program improvement plans. If
the state is unable to complete its
program improvement plan
successfully, a portion of the state’s
federal title IV–B and IV–E funds is
withheld.
In the April 23, 2014, Federal
Register (79 FR 22604), we provided a
detailed review of the consultation with
the field and information considered in
developing the third round of CFSRs,
and proposed a set of statewide data
indicators for public comment. We
considered all public comments and
issued a final plan in the October 10,
2014, Federal Register (79 FR 61241)
that included national standards for
state performance on statewide data
indicators and criteria CB would use to
determine whether a state is in
substantial conformity with certain
child welfare outcomes.
Simultaneously, CB released CFSR
Technical Bulletin #8 which provided
more details on calculation methods,
and a workbook that showed individual
state performance on the indicators and
preliminary findings of whether the
state met national standards. In
responding to state and other
stakeholder questions after the release of
those publications, we discovered errors
in our descriptions and calculations. As
a result, on May 13, 2015, we released
a notice in the Federal Register (80 FR
27263) to correct some calculations and
language used in the October 2014 final
notice. We also amended Technical
Bulletin #8 that provided technical
detail on calculations, how CB
determined whether a state met national
standards, and how program
improvement goals relative to the
statewide data indicators would be
established for states not meeting
national standards.
After the May 2015 notice, we
discovered additional technical errors in
the syntax and formulation of the
statewide data indicators. Some of the
changes were identified by states and
other interested parties that
operationalized the indicators for
continuous quality improvement. CB
determined additional time was
necessary to review, correct, and
thoroughly test revised syntax used to
calculate national and state performance
on the data indicators. Recognizing
challenges associated with the delay
and implementation of revised
indicators, especially for states already
involved in the CFSR and program
improvement planning process, CB
PO 00000
Frm 00055
Fmt 4703
Sfmt 4703
60875
decided to suspend use of the indicators
in determinations of substantial
conformity and as a basis for potentially
imposing financial penalties during the
third round of CFSRs. We are
authorized by the regulations at 45 CFR
1355.34(b)(4) and (5) to add, amend, or
suspend any of the statewide data
indicators and to adjust the national
standards when appropriate. CB issued
Technical Bulletin #9 on October 11,
2016, to notify states of the change in
the use of the statewide data indicators.
Technical Bulletin #9, which rescinded
Technical Bulletin #8A, provided
guidance on how state performance on
the CFSR 3 statewide data indicators
would be used as context, revised
related program improvement plan
requirements, and re-issued the
description of program improvement
measurement methods.
We recently completed the
comprehensive syntax review, revision,
and validation process for calculating
statewide data indicator performance
information. The revisions do not
change the statewide data indicator
measures as defined in prior Federal
Register notices. Revisions are limited
to operationalizing the measures and
calculation methods contained in the
statistical syntax used to generate
performance information.
Following the commencement of this
Federal Register Announcement, we
will review public comments to
determine readiness to finalize the
revised syntax or the potential need for
additional revisions. We will issue a
Federal Register to provide notice when
the revised syntax is finalized. The
notification will address applicable
revisions and clarifications necessary to
the Federal Register document from
May 2015, and include a summary of,
and our responses to, public comments
we received. We will also publish
revised performance results for all
states, revised national standards, and
the final revised syntax and supporting
files to calculate data quality, observed
performance, and risk-standardized
performance for the CFSR 3 statewide
data indicators.
Overview of Review and Validation of
Syntax To Calculate CFSR 3 Statewide
Data Indicator Performance
After the discovery of technical errors
in the syntax and formulation of the
statewide data indicators, federal staff
completed a comprehensive internal
review to identify and make syntax
revisions. The internal in-depth linelevel review of all syntax related to
computing performance on the
statewide indicators was completed in
May 2017. Staff verified each step of
E:\FR\FM\27NON1.SGM
27NON1
amozie on DSK3GDR082PROD with NOTICES1
60876
Federal Register / Vol. 83, No. 228 / Tuesday, November 27, 2018 / Notices
indicator computation to determine
whether the syntax was functioning
accurately and as designed. Additional
data quality tests were performed to
confirm the integrity of the population
of child records eligible for analysis,
and syntax functioned correctly across a
variety of scenarios, including outlier
cases with high error risk. The
comprehensive review resulted in the
identification of additional syntax
revisions, changes to how measurement
methods are operationalized, and new
methods to screen and address data
quality problems in AFCARS and
NCANDS submissions.
In addition to our internal review, we
contracted for an independent review
and validation of the syntax used to
complete data quality checks and
calculate observed and riskstandardized performance on the seven
statewide data indicators. The
independent reviewers used the
following multi-step process to
systematically review and verify all of
the syntax:
1. Reviewed syntax for completeness,
functionality on systems external to
the Children’s Bureau
2. Performed reliability checks against
performance indicators and state
data profiles
3. Performed line-level validity checks
of SPSS and STATA syntax files for
accurate task execution, data
quality verification
a. Reviewed the syntax and assessed
its function/purpose
b. Selected and applied the
appropriate validity check(s)
c. Ran all syntax and inspected
dataset(s) and output
d. Debriefed on similar code segments
across syntax files
The independent reviewers
determined the syntax can validly and
reliably develop source data files,
perform data quality checks, and
calculate state observed performance
and risk-standardized performance on
the seven statewide data indicators
according to the definitions set forth in
the Federal Register Notices and
associated CB guidance documents. The
reviewers also verified there are no
errors in the revised syntax for observed
performance and data quality checks, or
the original syntax for risk-standardized
performance that would cause
inaccuracies in the calculations. The
independent review also helped
demonstrate the feasibility for states and
partner agencies to replicate the syntax
with several additional steps to
calculate data quality and observed
performance on their own. Additional
information about the independent
VerDate Sep<11>2014
17:45 Nov 26, 2018
Jkt 247001
review and verification process and
associated findings are detailed in the
2018 report, Review and Validation of
the Analytic Syntax Used to Produce
CFSR Statewide Data Indicators
(https://www.acf.hhs.gov/cb/resource/
cfsr-round3-sdi-syntax-validation).
Supporting Documents
The documents identified below are
provided to assist states and other
interested parties with the independent
syntax review process:
A. List of CFSR Round 3 Statewide
Data Indicator Syntax Revisions:
Provides a summary of errors,
corrections, and changes made to the
syntax used to calculate performance on
the statewide data indicator(s) and data
quality checks. (https://
www.acf.hhs.gov/cb/resource/cfsrround3-sdi-syntax).
B. CFSR Round 3 Statewide Data
Indicator Data Dictionary: Provides a
description of each statewide data
indicator and data quality check,
including the numerators,
denominators, risk adjustments,
exclusions, and corresponding data
notes. (https://www.acf.hhs.gov/cb/
resource/cfsr-round3-sdi-datadictionary).
C. Statewide Data Indicators—PDF
Version of Syntax: Provides the syntax
used to perform data quality checks,
calculate observed performance, and
calculate risk-standardized performance
on the seven statewide data indicators.
This document may help individuals
develop a better understanding of the
statistical calculations or provide
assistance for individuals who do not
have SPSS and/or STATA statistical
software to run the syntax files. (https://
www.acf.hhs.gov/cb/resource/cfsrround3-sdi-syntax-pdf).
D. CFSR Round 3 Statewide Data
Indicator Syntax Zip File: Provides
syntax and supporting files in SPSS
format that are used to calculate data
quality and observed performance. Note:
Syntax to calculate risk-standardized
performance (RSP) for each state uses a
national, risk-adjusted model that
requires child-level data from all states
(i.e., national datasets) and thus cannot
be replicated by states and interested
parties. (https://www.acf.hhs.gov/cb/
resource/cfsr-round3-sdi-syntax-zip).
E. Instructions to Run Data Quality
(DQ) Checks and Observed Performance
Syntax for Statewide Data Indicators:
Provides an overview and brief
description of the folders, files, and
syntax included in the CFSR round 3
zip file; considerations and steps for
running the syntax; and software
requirements. (https://www.acf.hhs.gov/
PO 00000
Frm 00056
Fmt 4703
Sfmt 4703
cb/resource/cfsr-round3-sdi-dqinstructions).
Invitation To Comment
We are inviting state child welfare
agencies, partner organizations, and the
public to review, test, and provide
comments on the revised syntax to
perform data quality checks and
calculate observed performance on the
statewide data indicators. The purpose
of this invitation is to provide an
opportunity for additional independent
review prior to finalizing the revised
syntax. Interested parties may choose to
review a written version of the
statistical syntax and supporting
documents, and/or follow the
instructions to replicate the syntax files
to generate CFSR data indicator
performance information using stategenerated AFCARS and NCANDS files.
We are interested in receiving
comments on the revised syntax and
related considerations and
recommendations to strengthen our
statistical calculations for generating
CFSR statewide data indicator
performance information. We encourage
states and interested parties to share the
outcome of their review and/or results
from running the revised syntax files.
Please be specific in identifying the
corresponding statewide data indicator
or data quality check being referenced
in the comment; the syntax revision,
question, or recommendation; and the
rationale for considerations or
recommendation(s), as applicable.
Comments containing suggested
revisions should be limited to
operationalizing the measures and
calculation methods contained in the
statistical syntax used to generate
performance data. We are not making
changes or requesting comments at this
time pertaining to the statewide data
indicator measures as defined in prior
Federal Register notices and the Data
Dictionary available with this invitation.
Lynn A. Johnson,
Assistant Secretary for Children and Families.
[FR Doc. 2018–25835 Filed 11–26–18; 8:45 am]
BILLING CODE 4184–25–P
DEPARTMENT OF HEALTH AND
HUMAN SERVICES
Solicitation for Written Comments on
Proposed Objectives for Healthy
People 2030
Office of Disease Prevention
and Health Promotion, Office of the
Assistant Secretary for Health, Office of
the Secretary, Department of Health and
Human Services.
AGENCY:
E:\FR\FM\27NON1.SGM
27NON1
]]>Agencies
[Federal Register Volume 83, Number 228 (Tuesday, November 27, 2018)]
[Notices]
[Pages 60874-60876]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2018-25835]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Statewide Data Indicators for Child and Family Services Reviews:
Request for Public Comment
AGENCY: Children's Bureau (CB), Administration for Children and
Families (ACF), Administration on Children, Youth and Families (ACYF),
Department of Health and Human Services (HHS).
ACTION: Request for public comment on revised syntax used to calculate
Statewide Data Indicators for Child and Family Services Reviews
(CFSRs).
-----------------------------------------------------------------------
SUMMARY: In October 2014, the Administration for Children and Families
(ACF) published a notice in the Federal Register (79 FR 61241) with the
final plan to replace the statewide data indicators used to determine a
state's substantial conformity with titles IV-B and IV-E of the Social
Security Act through the Child and Family Services Reviews (CFSRs). In
May 2015, ACF published a notice in the Federal Register (80 FR 27263)
to correct some of the calculations and language used in the October
2014 notice. After the May 2015 notice, additional technical errors in
the syntax and formulation of the statewide data indicators were
identified. Some of those errors were discovered by states and other
interested parties that operationalized the indicators for continuous
quality improvement. Based on the amount of time required to complete a
comprehensive review of the syntax, make revisions and validate the
accuracy of the calculations; the Children's Bureau (CB) decided to
suspend use of the indicators in determinations of substantial
conformity during the third round of CFSRs. In October 2016, CB
published CFSR Technical Bulletin #9 to inform states of the decision
to limit use of the CFSR statewide data indicators and national
performance to context information for round three of CFSRs. Since
then, CB has thoroughly reviewed, revised, tested, and obtained
independent review and validation of the syntax used to perform data
quality checks and calculate state performance on the statewide data
indicators. Before CB finalizes the revised syntax, and publishes
revised performance results for all states and revised national
standards, we invite state child welfare agencies, partner
organizations, and the public to review, test, and provide comments on
the revised syntax used to perform data quality checks and calculate
observed performance on the statewide data indicators.
DATES: Comments are due by February 25, 2019.
ADDRESSES: You may send comments, identified by Docket Number ACF-2018-
0007, by one of the following methods:
[[Page 60875]]
Federal eRulemaking Portal: https://www.regulations.gov.
Follow the instructions for submitting comments.
Email: [email protected]. Include Docket Number ACF-
2018-0007 in subject line of the message.
FOR FURTHER INFORMATION CONTACT: James Gregory, (202) 260-4668,
[email protected].
SUPPLEMENTARY INFORMATION: This Federal Register announcement has four
sections: Background describes the purpose, history, and authority for
the CFSRs and the use of statewide data indicators; Overview of Review
and Validation of Syntax to Calculate CFSR 3 Statewide Data Indicator
Performance describes the internal and independent review and
validation process and findings; Supporting Documents provides a brief
description of five resources available to aid interested parties in
the syntax review process; and Invitation to Comment wherein we solicit
public comment on the revised syntax that is pending finalization.
Background
CB implemented the CFSRs in 2001 in response to a mandate in the
Social Security Amendments of 1994. The reviews are required for CB to
determine if such programs are in substantial conformity with title IV-
B and IV-E plan requirements. The review process, as regulated at 45
CFR 1355.31-37, grew out of extensive consultation with interested
groups, individuals, and experts in the field of child welfare and
related areas.
The CFSRs enable CB to: (1) Ensure conformity with federal child
welfare requirements; (2) determine what is actually happening to
children and families as they are engaged in child welfare services;
and (3) assist states to enhance their capacity to help children and
families achieve positive outcomes. CB conducts the reviews in
partnership with state child welfare agency staff and other partners
and stakeholders involved in the provision of child welfare services.
We have structured the reviews to help states identify strengths as
well as areas needing improvement within their agencies and programs.
We use the CFSR to assess state performance on seven outcomes and
seven systemic factors. The seven outcomes focus on key items measuring
safety, permanency, and well-being. The seven systemic factors focus on
key state plan requirements of titles IV-B and IV-E that provide a
foundation for child outcomes. If we determine a state has not achieved
substantial conformity in one or more of the areas assessed in the
review, the state is required to develop and implement a program
improvement plan addressing the areas of nonconformity within 2 years.
CB supports the states with technical assistance and monitors
implementation of their program improvement plans. If the state is
unable to complete its program improvement plan successfully, a portion
of the state's federal title IV-B and IV-E funds is withheld.
In the April 23, 2014, Federal Register (79 FR 22604), we provided
a detailed review of the consultation with the field and information
considered in developing the third round of CFSRs, and proposed a set
of statewide data indicators for public comment. We considered all
public comments and issued a final plan in the October 10, 2014,
Federal Register (79 FR 61241) that included national standards for
state performance on statewide data indicators and criteria CB would
use to determine whether a state is in substantial conformity with
certain child welfare outcomes. Simultaneously, CB released CFSR
Technical Bulletin #8 which provided more details on calculation
methods, and a workbook that showed individual state performance on the
indicators and preliminary findings of whether the state met national
standards. In responding to state and other stakeholder questions after
the release of those publications, we discovered errors in our
descriptions and calculations. As a result, on May 13, 2015, we
released a notice in the Federal Register (80 FR 27263) to correct some
calculations and language used in the October 2014 final notice. We
also amended Technical Bulletin #8 that provided technical detail on
calculations, how CB determined whether a state met national standards,
and how program improvement goals relative to the statewide data
indicators would be established for states not meeting national
standards.
After the May 2015 notice, we discovered additional technical
errors in the syntax and formulation of the statewide data indicators.
Some of the changes were identified by states and other interested
parties that operationalized the indicators for continuous quality
improvement. CB determined additional time was necessary to review,
correct, and thoroughly test revised syntax used to calculate national
and state performance on the data indicators. Recognizing challenges
associated with the delay and implementation of revised indicators,
especially for states already involved in the CFSR and program
improvement planning process, CB decided to suspend use of the
indicators in determinations of substantial conformity and as a basis
for potentially imposing financial penalties during the third round of
CFSRs. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and
(5) to add, amend, or suspend any of the statewide data indicators and
to adjust the national standards when appropriate. CB issued Technical
Bulletin #9 on October 11, 2016, to notify states of the change in the
use of the statewide data indicators. Technical Bulletin #9, which
rescinded Technical Bulletin #8A, provided guidance on how state
performance on the CFSR 3 statewide data indicators would be used as
context, revised related program improvement plan requirements, and re-
issued the description of program improvement measurement methods.
We recently completed the comprehensive syntax review, revision,
and validation process for calculating statewide data indicator
performance information. The revisions do not change the statewide data
indicator measures as defined in prior Federal Register notices.
Revisions are limited to operationalizing the measures and calculation
methods contained in the statistical syntax used to generate
performance information.
Following the commencement of this Federal Register Announcement,
we will review public comments to determine readiness to finalize the
revised syntax or the potential need for additional revisions. We will
issue a Federal Register to provide notice when the revised syntax is
finalized. The notification will address applicable revisions and
clarifications necessary to the Federal Register document from May
2015, and include a summary of, and our responses to, public comments
we received. We will also publish revised performance results for all
states, revised national standards, and the final revised syntax and
supporting files to calculate data quality, observed performance, and
risk-standardized performance for the CFSR 3 statewide data indicators.
Overview of Review and Validation of Syntax To Calculate CFSR 3
Statewide Data Indicator Performance
After the discovery of technical errors in the syntax and
formulation of the statewide data indicators, federal staff completed a
comprehensive internal review to identify and make syntax revisions.
The internal in-depth line-level review of all syntax related to
computing performance on the statewide indicators was completed in May
2017. Staff verified each step of
[[Page 60876]]
indicator computation to determine whether the syntax was functioning
accurately and as designed. Additional data quality tests were
performed to confirm the integrity of the population of child records
eligible for analysis, and syntax functioned correctly across a variety
of scenarios, including outlier cases with high error risk. The
comprehensive review resulted in the identification of additional
syntax revisions, changes to how measurement methods are
operationalized, and new methods to screen and address data quality
problems in AFCARS and NCANDS submissions.
In addition to our internal review, we contracted for an
independent review and validation of the syntax used to complete data
quality checks and calculate observed and risk-standardized performance
on the seven statewide data indicators. The independent reviewers used
the following multi-step process to systematically review and verify
all of the syntax:
1. Reviewed syntax for completeness, functionality on systems external
to the Children's Bureau
2. Performed reliability checks against performance indicators and
state data profiles
3. Performed line-level validity checks of SPSS and STATA syntax files
for accurate task execution, data quality verification
a. Reviewed the syntax and assessed its function/purpose
b. Selected and applied the appropriate validity check(s)
c. Ran all syntax and inspected dataset(s) and output
d. Debriefed on similar code segments across syntax files
The independent reviewers determined the syntax can validly and
reliably develop source data files, perform data quality checks, and
calculate state observed performance and risk-standardized performance
on the seven statewide data indicators according to the definitions set
forth in the Federal Register Notices and associated CB guidance
documents. The reviewers also verified there are no errors in the
revised syntax for observed performance and data quality checks, or the
original syntax for risk-standardized performance that would cause
inaccuracies in the calculations. The independent review also helped
demonstrate the feasibility for states and partner agencies to
replicate the syntax with several additional steps to calculate data
quality and observed performance on their own. Additional information
about the independent review and verification process and associated
findings are detailed in the 2018 report, Review and Validation of the
Analytic Syntax Used to Produce CFSR Statewide Data Indicators (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-validation).
Supporting Documents
The documents identified below are provided to assist states and
other interested parties with the independent syntax review process:
A. List of CFSR Round 3 Statewide Data Indicator Syntax Revisions:
Provides a summary of errors, corrections, and changes made to the
syntax used to calculate performance on the statewide data indicator(s)
and data quality checks. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax).
B. CFSR Round 3 Statewide Data Indicator Data Dictionary: Provides
a description of each statewide data indicator and data quality check,
including the numerators, denominators, risk adjustments, exclusions,
and corresponding data notes. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-data-dictionary).
C. Statewide Data Indicators--PDF Version of Syntax: Provides the
syntax used to perform data quality checks, calculate observed
performance, and calculate risk-standardized performance on the seven
statewide data indicators. This document may help individuals develop a
better understanding of the statistical calculations or provide
assistance for individuals who do not have SPSS and/or STATA
statistical software to run the syntax files. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-pdf).
D. CFSR Round 3 Statewide Data Indicator Syntax Zip File: Provides
syntax and supporting files in SPSS format that are used to calculate
data quality and observed performance. Note: Syntax to calculate risk-
standardized performance (RSP) for each state uses a national, risk-
adjusted model that requires child-level data from all states (i.e.,
national datasets) and thus cannot be replicated by states and
interested parties. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-zip).
E. Instructions to Run Data Quality (DQ) Checks and Observed
Performance Syntax for Statewide Data Indicators: Provides an overview
and brief description of the folders, files, and syntax included in the
CFSR round 3 zip file; considerations and steps for running the syntax;
and software requirements. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-dq-instructions).
Invitation To Comment
We are inviting state child welfare agencies, partner
organizations, and the public to review, test, and provide comments on
the revised syntax to perform data quality checks and calculate
observed performance on the statewide data indicators. The purpose of
this invitation is to provide an opportunity for additional independent
review prior to finalizing the revised syntax. Interested parties may
choose to review a written version of the statistical syntax and
supporting documents, and/or follow the instructions to replicate the
syntax files to generate CFSR data indicator performance information
using state-generated AFCARS and NCANDS files.
We are interested in receiving comments on the revised syntax and
related considerations and recommendations to strengthen our
statistical calculations for generating CFSR statewide data indicator
performance information. We encourage states and interested parties to
share the outcome of their review and/or results from running the
revised syntax files. Please be specific in identifying the
corresponding statewide data indicator or data quality check being
referenced in the comment; the syntax revision, question, or
recommendation; and the rationale for considerations or
recommendation(s), as applicable.
Comments containing suggested revisions should be limited to
operationalizing the measures and calculation methods contained in the
statistical syntax used to generate performance data. We are not making
changes or requesting comments at this time pertaining to the statewide
data indicator measures as defined in prior Federal Register notices
and the Data Dictionary available with this invitation.
Lynn A. Johnson,
Assistant Secretary for Children and Families.
[FR Doc. 2018-25835 Filed 11-26-18; 8:45 am]
BILLING CODE 4184-25-P
