Statewide Data Indicators for Child and Family Services Reviews: Request for Public Comment, 60874-60876 [2018-25835]

Download as PDF 60874 Federal Register / Vol. 83, No. 228 / Tuesday, November 27, 2018 / Notices amozie on DSK3GDR082PROD with NOTICES1 Form Number: CMS–179 (OMB control number 0938–0193); Frequency: Occasionally; Affected Public: State, Local, and Tribal Governments; Number of Respondents: 56; Total Annual Responses: 1,120; Total Annual Hours: 22,400. (For policy questions regarding this collection contact Annette Pearson at 410–786–6958.) 2. Type of Information Collection Request: Extension; Title of Information Collection: Home Health Change of Care Notice; Use: The purpose of the Home Health Change of Care Notice (HHCCN) is to notify original Medicare beneficiaries receiving home health care benefits of plan of care changes. Home health agencies (HHAs) are required to provide written notice to Original Medicare beneficiaries under various circumstances involving the reduction or termination of items and/or services consistent with Home Health Agencies Conditions of Participation (COPs). The home health COP requirements are set forth in § 1891[42 U.S.C. 1395bbb] of the Social Security Act (the Act). The implementing regulations under 42 CFR 484.10(c) specify that Medicare patients receiving HHA services have rights. The patient has the right to be informed, in advance about the care to be furnished, and of any changes in the care to be furnished. The HHA must advise the patient in advance of the disciplines that will furnish care, and the frequency of visits proposed to be furnished. The HHA must advise the patient in advance of any change in the plan of care before the change is made.’’ Notification is required for covered and non-covered services listed in the plan of care (POC). The beneficiary will use the information provided to decide whether or not to pursue alternative options to continue receiving the care noted on the HHCCN. Form Number: CMS–10280 (OMB control number: 0938–1196); Frequency: Yearly; Affected Public: Private Sector (Business or other for-profits, Not-for-Profit Institutions); Number of Respondents: 12,149; Total Annual Responses: 13,640,524; Total Annual Hours: 908,459. (For policy questions regarding this collection contact Jennifer McCormick at 410–786– 2852.) Dated: November 21, 2018. William N. Parham, III, Director, Paperwork Reduction Staff, Office of Strategic Operations and Regulatory Affairs. [FR Doc. 2018–25858 Filed 11–26–18; 8:45 am] BILLING CODE 4120–01–P VerDate Sep<11>2014 17:45 Nov 26, 2018 Jkt 247001 DEPARTMENT OF HEALTH AND HUMAN SERVICES Centers for Medicare & Medicaid Services [Document Identifier: CMS–179] Agency Information Collection Activities: Proposed Collection; Comment Request ACTION: Notice; partial withdrawal. On October 1, 2018 (83 FR 49389), the Centers for Medicare & Medicaid Services (CMS) published a notice entitled, ‘‘Agency Information Collection Activities: Proposed Collection; Comment Request.’’ That notice invited public comments on three separate information collection requests specific to document identifiers: CMS– 10142, CMS–R–262, and CMS–179. Due to technical difficulties associated with CMS–179 (OMB control number: 0938– 0193) titled ‘‘Medicaid State Plan Base Plan Pages,’’ through the publication of this document we are withdrawing CMS–179 from our October 1, 2018, information collection request. While the technical issues associated with CMS–179 have recently been resolved, because of the delay we are publishing a new 60-day information collection request elsewhere in today’s Federal Register. The November 30, 2018, comment due date for the remaining two collections [CMS–10142 (OMB control number: 0938–0944) titled, ‘‘Bid Pricing Tool (BPT) for Medicare Advantage (MA) Plans and Prescription Drug Plans (PDP)’’ and CMS–R–262 (OMB control number: 0938–0763), titled ‘‘Contract Year 2020 Plan Benefit Package (PBP) Software and Formulary Submission’’] remains in effect without change. SUMMARY: Dated: November 21, 2018. William N. Parham, III, Director, Paperwork Reduction Staff, Office of Strategic Operations and Regulatory Affairs. [FR Doc. 2018–25840 Filed 11–26–18; 8:45 am] BILLING CODE 4120–01–P DEPARTMENT OF HEALTH AND HUMAN SERVICES Administration for Children and Families Statewide Data Indicators for Child and Family Services Reviews: Request for Public Comment Children’s Bureau (CB), Administration for Children and Families (ACF), Administration on AGENCY: PO 00000 Frm 00054 Fmt 4703 Sfmt 4703 Children, Youth and Families (ACYF), Department of Health and Human Services (HHS). Request for public comment on revised syntax used to calculate Statewide Data Indicators for Child and Family Services Reviews (CFSRs). ACTION: In October 2014, the Administration for Children and Families (ACF) published a notice in the Federal Register (79 FR 61241) with the final plan to replace the statewide data indicators used to determine a state’s substantial conformity with titles IV–B and IV–E of the Social Security Act through the Child and Family Services Reviews (CFSRs). In May 2015, ACF published a notice in the Federal Register (80 FR 27263) to correct some of the calculations and language used in the October 2014 notice. After the May 2015 notice, additional technical errors in the syntax and formulation of the statewide data indicators were identified. Some of those errors were discovered by states and other interested parties that operationalized the indicators for continuous quality improvement. Based on the amount of time required to complete a comprehensive review of the syntax, make revisions and validate the accuracy of the calculations; the Children’s Bureau (CB) decided to suspend use of the indicators in determinations of substantial conformity during the third round of CFSRs. In October 2016, CB published CFSR Technical Bulletin #9 to inform states of the decision to limit use of the CFSR statewide data indicators and national performance to context information for round three of CFSRs. Since then, CB has thoroughly reviewed, revised, tested, and obtained independent review and validation of the syntax used to perform data quality checks and calculate state performance on the statewide data indicators. Before CB finalizes the revised syntax, and publishes revised performance results for all states and revised national standards, we invite state child welfare agencies, partner organizations, and the public to review, test, and provide comments on the revised syntax used to perform data quality checks and calculate observed performance on the statewide data indicators. SUMMARY: Comments are due by February 25, 2019. DATES: You may send comments, identified by Docket Number ACF– 2018–0007, by one of the following methods: ADDRESSES: E:\FR\FM\27NON1.SGM 27NON1 Federal Register / Vol. 83, No. 228 / Tuesday, November 27, 2018 / Notices • Federal eRulemaking Portal: https://www.regulations.gov. Follow the instructions for submitting comments. • Email: CBComments@acf.hhs.gov. Include Docket Number ACF–2018– 0007 in subject line of the message. FOR FURTHER INFORMATION CONTACT: James Gregory, (202) 260–4668, James.Gregory@acf.hhs.gov. SUPPLEMENTARY INFORMATION: This Federal Register announcement has four sections: Background describes the purpose, history, and authority for the CFSRs and the use of statewide data indicators; Overview of Review and Validation of Syntax to Calculate CFSR 3 Statewide Data Indicator Performance describes the internal and independent review and validation process and findings; Supporting Documents provides a brief description of five resources available to aid interested parties in the syntax review process; and Invitation to Comment wherein we solicit public comment on the revised syntax that is pending finalization. amozie on DSK3GDR082PROD with NOTICES1 Background CB implemented the CFSRs in 2001 in response to a mandate in the Social Security Amendments of 1994. The reviews are required for CB to determine if such programs are in substantial conformity with title IV–B and IV–E plan requirements. The review process, as regulated at 45 CFR 1355.31–37, grew out of extensive consultation with interested groups, individuals, and experts in the field of child welfare and related areas. The CFSRs enable CB to: (1) Ensure conformity with federal child welfare requirements; (2) determine what is actually happening to children and families as they are engaged in child welfare services; and (3) assist states to enhance their capacity to help children and families achieve positive outcomes. CB conducts the reviews in partnership with state child welfare agency staff and other partners and stakeholders involved in the provision of child welfare services. We have structured the reviews to help states identify strengths as well as areas needing improvement within their agencies and programs. We use the CFSR to assess state performance on seven outcomes and seven systemic factors. The seven outcomes focus on key items measuring safety, permanency, and well-being. The seven systemic factors focus on key state plan requirements of titles IV–B and IV– E that provide a foundation for child outcomes. If we determine a state has not achieved substantial conformity in one or more of the areas assessed in the review, the state is required to develop VerDate Sep<11>2014 17:45 Nov 26, 2018 Jkt 247001 and implement a program improvement plan addressing the areas of nonconformity within 2 years. CB supports the states with technical assistance and monitors implementation of their program improvement plans. If the state is unable to complete its program improvement plan successfully, a portion of the state’s federal title IV–B and IV–E funds is withheld. In the April 23, 2014, Federal Register (79 FR 22604), we provided a detailed review of the consultation with the field and information considered in developing the third round of CFSRs, and proposed a set of statewide data indicators for public comment. We considered all public comments and issued a final plan in the October 10, 2014, Federal Register (79 FR 61241) that included national standards for state performance on statewide data indicators and criteria CB would use to determine whether a state is in substantial conformity with certain child welfare outcomes. Simultaneously, CB released CFSR Technical Bulletin #8 which provided more details on calculation methods, and a workbook that showed individual state performance on the indicators and preliminary findings of whether the state met national standards. In responding to state and other stakeholder questions after the release of those publications, we discovered errors in our descriptions and calculations. As a result, on May 13, 2015, we released a notice in the Federal Register (80 FR 27263) to correct some calculations and language used in the October 2014 final notice. We also amended Technical Bulletin #8 that provided technical detail on calculations, how CB determined whether a state met national standards, and how program improvement goals relative to the statewide data indicators would be established for states not meeting national standards. After the May 2015 notice, we discovered additional technical errors in the syntax and formulation of the statewide data indicators. Some of the changes were identified by states and other interested parties that operationalized the indicators for continuous quality improvement. CB determined additional time was necessary to review, correct, and thoroughly test revised syntax used to calculate national and state performance on the data indicators. Recognizing challenges associated with the delay and implementation of revised indicators, especially for states already involved in the CFSR and program improvement planning process, CB PO 00000 Frm 00055 Fmt 4703 Sfmt 4703 60875 decided to suspend use of the indicators in determinations of substantial conformity and as a basis for potentially imposing financial penalties during the third round of CFSRs. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and (5) to add, amend, or suspend any of the statewide data indicators and to adjust the national standards when appropriate. CB issued Technical Bulletin #9 on October 11, 2016, to notify states of the change in the use of the statewide data indicators. Technical Bulletin #9, which rescinded Technical Bulletin #8A, provided guidance on how state performance on the CFSR 3 statewide data indicators would be used as context, revised related program improvement plan requirements, and re-issued the description of program improvement measurement methods. We recently completed the comprehensive syntax review, revision, and validation process for calculating statewide data indicator performance information. The revisions do not change the statewide data indicator measures as defined in prior Federal Register notices. Revisions are limited to operationalizing the measures and calculation methods contained in the statistical syntax used to generate performance information. Following the commencement of this Federal Register Announcement, we will review public comments to determine readiness to finalize the revised syntax or the potential need for additional revisions. We will issue a Federal Register to provide notice when the revised syntax is finalized. The notification will address applicable revisions and clarifications necessary to the Federal Register document from May 2015, and include a summary of, and our responses to, public comments we received. We will also publish revised performance results for all states, revised national standards, and the final revised syntax and supporting files to calculate data quality, observed performance, and risk-standardized performance for the CFSR 3 statewide data indicators. Overview of Review and Validation of Syntax To Calculate CFSR 3 Statewide Data Indicator Performance After the discovery of technical errors in the syntax and formulation of the statewide data indicators, federal staff completed a comprehensive internal review to identify and make syntax revisions. The internal in-depth linelevel review of all syntax related to computing performance on the statewide indicators was completed in May 2017. Staff verified each step of E:\FR\FM\27NON1.SGM 27NON1 amozie on DSK3GDR082PROD with NOTICES1 60876 Federal Register / Vol. 83, No. 228 / Tuesday, November 27, 2018 / Notices indicator computation to determine whether the syntax was functioning accurately and as designed. Additional data quality tests were performed to confirm the integrity of the population of child records eligible for analysis, and syntax functioned correctly across a variety of scenarios, including outlier cases with high error risk. The comprehensive review resulted in the identification of additional syntax revisions, changes to how measurement methods are operationalized, and new methods to screen and address data quality problems in AFCARS and NCANDS submissions. In addition to our internal review, we contracted for an independent review and validation of the syntax used to complete data quality checks and calculate observed and riskstandardized performance on the seven statewide data indicators. The independent reviewers used the following multi-step process to systematically review and verify all of the syntax: 1. Reviewed syntax for completeness, functionality on systems external to the Children’s Bureau 2. Performed reliability checks against performance indicators and state data profiles 3. Performed line-level validity checks of SPSS and STATA syntax files for accurate task execution, data quality verification a. Reviewed the syntax and assessed its function/purpose b. Selected and applied the appropriate validity check(s) c. Ran all syntax and inspected dataset(s) and output d. Debriefed on similar code segments across syntax files The independent reviewers determined the syntax can validly and reliably develop source data files, perform data quality checks, and calculate state observed performance and risk-standardized performance on the seven statewide data indicators according to the definitions set forth in the Federal Register Notices and associated CB guidance documents. The reviewers also verified there are no errors in the revised syntax for observed performance and data quality checks, or the original syntax for risk-standardized performance that would cause inaccuracies in the calculations. The independent review also helped demonstrate the feasibility for states and partner agencies to replicate the syntax with several additional steps to calculate data quality and observed performance on their own. Additional information about the independent VerDate Sep<11>2014 17:45 Nov 26, 2018 Jkt 247001 review and verification process and associated findings are detailed in the 2018 report, Review and Validation of the Analytic Syntax Used to Produce CFSR Statewide Data Indicators (https://www.acf.hhs.gov/cb/resource/ cfsr-round3-sdi-syntax-validation). Supporting Documents The documents identified below are provided to assist states and other interested parties with the independent syntax review process: A. List of CFSR Round 3 Statewide Data Indicator Syntax Revisions: Provides a summary of errors, corrections, and changes made to the syntax used to calculate performance on the statewide data indicator(s) and data quality checks. (https:// www.acf.hhs.gov/cb/resource/cfsrround3-sdi-syntax). B. CFSR Round 3 Statewide Data Indicator Data Dictionary: Provides a description of each statewide data indicator and data quality check, including the numerators, denominators, risk adjustments, exclusions, and corresponding data notes. (https://www.acf.hhs.gov/cb/ resource/cfsr-round3-sdi-datadictionary). C. Statewide Data Indicators—PDF Version of Syntax: Provides the syntax used to perform data quality checks, calculate observed performance, and calculate risk-standardized performance on the seven statewide data indicators. This document may help individuals develop a better understanding of the statistical calculations or provide assistance for individuals who do not have SPSS and/or STATA statistical software to run the syntax files. (https:// www.acf.hhs.gov/cb/resource/cfsrround3-sdi-syntax-pdf). D. CFSR Round 3 Statewide Data Indicator Syntax Zip File: Provides syntax and supporting files in SPSS format that are used to calculate data quality and observed performance. Note: Syntax to calculate risk-standardized performance (RSP) for each state uses a national, risk-adjusted model that requires child-level data from all states (i.e., national datasets) and thus cannot be replicated by states and interested parties. (https://www.acf.hhs.gov/cb/ resource/cfsr-round3-sdi-syntax-zip). E. Instructions to Run Data Quality (DQ) Checks and Observed Performance Syntax for Statewide Data Indicators: Provides an overview and brief description of the folders, files, and syntax included in the CFSR round 3 zip file; considerations and steps for running the syntax; and software requirements. (https://www.acf.hhs.gov/ PO 00000 Frm 00056 Fmt 4703 Sfmt 4703 cb/resource/cfsr-round3-sdi-dqinstructions). Invitation To Comment We are inviting state child welfare agencies, partner organizations, and the public to review, test, and provide comments on the revised syntax to perform data quality checks and calculate observed performance on the statewide data indicators. The purpose of this invitation is to provide an opportunity for additional independent review prior to finalizing the revised syntax. Interested parties may choose to review a written version of the statistical syntax and supporting documents, and/or follow the instructions to replicate the syntax files to generate CFSR data indicator performance information using stategenerated AFCARS and NCANDS files. We are interested in receiving comments on the revised syntax and related considerations and recommendations to strengthen our statistical calculations for generating CFSR statewide data indicator performance information. We encourage states and interested parties to share the outcome of their review and/or results from running the revised syntax files. Please be specific in identifying the corresponding statewide data indicator or data quality check being referenced in the comment; the syntax revision, question, or recommendation; and the rationale for considerations or recommendation(s), as applicable. Comments containing suggested revisions should be limited to operationalizing the measures and calculation methods contained in the statistical syntax used to generate performance data. We are not making changes or requesting comments at this time pertaining to the statewide data indicator measures as defined in prior Federal Register notices and the Data Dictionary available with this invitation. Lynn A. Johnson, Assistant Secretary for Children and Families. [FR Doc. 2018–25835 Filed 11–26–18; 8:45 am] BILLING CODE 4184–25–P DEPARTMENT OF HEALTH AND HUMAN SERVICES Solicitation for Written Comments on Proposed Objectives for Healthy People 2030 Office of Disease Prevention and Health Promotion, Office of the Assistant Secretary for Health, Office of the Secretary, Department of Health and Human Services. AGENCY: E:\FR\FM\27NON1.SGM 27NON1

Agencies

[Federal Register Volume 83, Number 228 (Tuesday, November 27, 2018)]
[Notices]
[Pages 60874-60876]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2018-25835]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Statewide Data Indicators for Child and Family Services Reviews: 
Request for Public Comment

AGENCY: Children's Bureau (CB), Administration for Children and 
Families (ACF), Administration on Children, Youth and Families (ACYF), 
Department of Health and Human Services (HHS).

ACTION: Request for public comment on revised syntax used to calculate 
Statewide Data Indicators for Child and Family Services Reviews 
(CFSRs).

-----------------------------------------------------------------------

SUMMARY: In October 2014, the Administration for Children and Families 
(ACF) published a notice in the Federal Register (79 FR 61241) with the 
final plan to replace the statewide data indicators used to determine a 
state's substantial conformity with titles IV-B and IV-E of the Social 
Security Act through the Child and Family Services Reviews (CFSRs). In 
May 2015, ACF published a notice in the Federal Register (80 FR 27263) 
to correct some of the calculations and language used in the October 
2014 notice. After the May 2015 notice, additional technical errors in 
the syntax and formulation of the statewide data indicators were 
identified. Some of those errors were discovered by states and other 
interested parties that operationalized the indicators for continuous 
quality improvement. Based on the amount of time required to complete a 
comprehensive review of the syntax, make revisions and validate the 
accuracy of the calculations; the Children's Bureau (CB) decided to 
suspend use of the indicators in determinations of substantial 
conformity during the third round of CFSRs. In October 2016, CB 
published CFSR Technical Bulletin #9 to inform states of the decision 
to limit use of the CFSR statewide data indicators and national 
performance to context information for round three of CFSRs. Since 
then, CB has thoroughly reviewed, revised, tested, and obtained 
independent review and validation of the syntax used to perform data 
quality checks and calculate state performance on the statewide data 
indicators. Before CB finalizes the revised syntax, and publishes 
revised performance results for all states and revised national 
standards, we invite state child welfare agencies, partner 
organizations, and the public to review, test, and provide comments on 
the revised syntax used to perform data quality checks and calculate 
observed performance on the statewide data indicators.

DATES: Comments are due by February 25, 2019.

ADDRESSES: You may send comments, identified by Docket Number ACF-2018-
0007, by one of the following methods:

[[Page 60875]]

     Federal eRulemaking Portal: https://www.regulations.gov. 
Follow the instructions for submitting comments.
     Email: [email protected]. Include Docket Number ACF-
2018-0007 in subject line of the message.

FOR FURTHER INFORMATION CONTACT: James Gregory, (202) 260-4668, 
[email protected].

SUPPLEMENTARY INFORMATION: This Federal Register announcement has four 
sections: Background describes the purpose, history, and authority for 
the CFSRs and the use of statewide data indicators; Overview of Review 
and Validation of Syntax to Calculate CFSR 3 Statewide Data Indicator 
Performance describes the internal and independent review and 
validation process and findings; Supporting Documents provides a brief 
description of five resources available to aid interested parties in 
the syntax review process; and Invitation to Comment wherein we solicit 
public comment on the revised syntax that is pending finalization.

Background

    CB implemented the CFSRs in 2001 in response to a mandate in the 
Social Security Amendments of 1994. The reviews are required for CB to 
determine if such programs are in substantial conformity with title IV-
B and IV-E plan requirements. The review process, as regulated at 45 
CFR 1355.31-37, grew out of extensive consultation with interested 
groups, individuals, and experts in the field of child welfare and 
related areas.
    The CFSRs enable CB to: (1) Ensure conformity with federal child 
welfare requirements; (2) determine what is actually happening to 
children and families as they are engaged in child welfare services; 
and (3) assist states to enhance their capacity to help children and 
families achieve positive outcomes. CB conducts the reviews in 
partnership with state child welfare agency staff and other partners 
and stakeholders involved in the provision of child welfare services. 
We have structured the reviews to help states identify strengths as 
well as areas needing improvement within their agencies and programs.
    We use the CFSR to assess state performance on seven outcomes and 
seven systemic factors. The seven outcomes focus on key items measuring 
safety, permanency, and well-being. The seven systemic factors focus on 
key state plan requirements of titles IV-B and IV-E that provide a 
foundation for child outcomes. If we determine a state has not achieved 
substantial conformity in one or more of the areas assessed in the 
review, the state is required to develop and implement a program 
improvement plan addressing the areas of nonconformity within 2 years. 
CB supports the states with technical assistance and monitors 
implementation of their program improvement plans. If the state is 
unable to complete its program improvement plan successfully, a portion 
of the state's federal title IV-B and IV-E funds is withheld.
    In the April 23, 2014, Federal Register (79 FR 22604), we provided 
a detailed review of the consultation with the field and information 
considered in developing the third round of CFSRs, and proposed a set 
of statewide data indicators for public comment. We considered all 
public comments and issued a final plan in the October 10, 2014, 
Federal Register (79 FR 61241) that included national standards for 
state performance on statewide data indicators and criteria CB would 
use to determine whether a state is in substantial conformity with 
certain child welfare outcomes. Simultaneously, CB released CFSR 
Technical Bulletin #8 which provided more details on calculation 
methods, and a workbook that showed individual state performance on the 
indicators and preliminary findings of whether the state met national 
standards. In responding to state and other stakeholder questions after 
the release of those publications, we discovered errors in our 
descriptions and calculations. As a result, on May 13, 2015, we 
released a notice in the Federal Register (80 FR 27263) to correct some 
calculations and language used in the October 2014 final notice. We 
also amended Technical Bulletin #8 that provided technical detail on 
calculations, how CB determined whether a state met national standards, 
and how program improvement goals relative to the statewide data 
indicators would be established for states not meeting national 
standards.
    After the May 2015 notice, we discovered additional technical 
errors in the syntax and formulation of the statewide data indicators. 
Some of the changes were identified by states and other interested 
parties that operationalized the indicators for continuous quality 
improvement. CB determined additional time was necessary to review, 
correct, and thoroughly test revised syntax used to calculate national 
and state performance on the data indicators. Recognizing challenges 
associated with the delay and implementation of revised indicators, 
especially for states already involved in the CFSR and program 
improvement planning process, CB decided to suspend use of the 
indicators in determinations of substantial conformity and as a basis 
for potentially imposing financial penalties during the third round of 
CFSRs. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and 
(5) to add, amend, or suspend any of the statewide data indicators and 
to adjust the national standards when appropriate. CB issued Technical 
Bulletin #9 on October 11, 2016, to notify states of the change in the 
use of the statewide data indicators. Technical Bulletin #9, which 
rescinded Technical Bulletin #8A, provided guidance on how state 
performance on the CFSR 3 statewide data indicators would be used as 
context, revised related program improvement plan requirements, and re-
issued the description of program improvement measurement methods.
    We recently completed the comprehensive syntax review, revision, 
and validation process for calculating statewide data indicator 
performance information. The revisions do not change the statewide data 
indicator measures as defined in prior Federal Register notices. 
Revisions are limited to operationalizing the measures and calculation 
methods contained in the statistical syntax used to generate 
performance information.
    Following the commencement of this Federal Register Announcement, 
we will review public comments to determine readiness to finalize the 
revised syntax or the potential need for additional revisions. We will 
issue a Federal Register to provide notice when the revised syntax is 
finalized. The notification will address applicable revisions and 
clarifications necessary to the Federal Register document from May 
2015, and include a summary of, and our responses to, public comments 
we received. We will also publish revised performance results for all 
states, revised national standards, and the final revised syntax and 
supporting files to calculate data quality, observed performance, and 
risk-standardized performance for the CFSR 3 statewide data indicators.

Overview of Review and Validation of Syntax To Calculate CFSR 3 
Statewide Data Indicator Performance

    After the discovery of technical errors in the syntax and 
formulation of the statewide data indicators, federal staff completed a 
comprehensive internal review to identify and make syntax revisions. 
The internal in-depth line-level review of all syntax related to 
computing performance on the statewide indicators was completed in May 
2017. Staff verified each step of

[[Page 60876]]

indicator computation to determine whether the syntax was functioning 
accurately and as designed. Additional data quality tests were 
performed to confirm the integrity of the population of child records 
eligible for analysis, and syntax functioned correctly across a variety 
of scenarios, including outlier cases with high error risk. The 
comprehensive review resulted in the identification of additional 
syntax revisions, changes to how measurement methods are 
operationalized, and new methods to screen and address data quality 
problems in AFCARS and NCANDS submissions.
    In addition to our internal review, we contracted for an 
independent review and validation of the syntax used to complete data 
quality checks and calculate observed and risk-standardized performance 
on the seven statewide data indicators. The independent reviewers used 
the following multi-step process to systematically review and verify 
all of the syntax:

1. Reviewed syntax for completeness, functionality on systems external 
to the Children's Bureau
2. Performed reliability checks against performance indicators and 
state data profiles
3. Performed line-level validity checks of SPSS and STATA syntax files 
for accurate task execution, data quality verification
    a. Reviewed the syntax and assessed its function/purpose
    b. Selected and applied the appropriate validity check(s)
    c. Ran all syntax and inspected dataset(s) and output
    d. Debriefed on similar code segments across syntax files

    The independent reviewers determined the syntax can validly and 
reliably develop source data files, perform data quality checks, and 
calculate state observed performance and risk-standardized performance 
on the seven statewide data indicators according to the definitions set 
forth in the Federal Register Notices and associated CB guidance 
documents. The reviewers also verified there are no errors in the 
revised syntax for observed performance and data quality checks, or the 
original syntax for risk-standardized performance that would cause 
inaccuracies in the calculations. The independent review also helped 
demonstrate the feasibility for states and partner agencies to 
replicate the syntax with several additional steps to calculate data 
quality and observed performance on their own. Additional information 
about the independent review and verification process and associated 
findings are detailed in the 2018 report, Review and Validation of the 
Analytic Syntax Used to Produce CFSR Statewide Data Indicators (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-validation).

Supporting Documents

    The documents identified below are provided to assist states and 
other interested parties with the independent syntax review process:
    A. List of CFSR Round 3 Statewide Data Indicator Syntax Revisions: 
Provides a summary of errors, corrections, and changes made to the 
syntax used to calculate performance on the statewide data indicator(s) 
and data quality checks. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax).
    B. CFSR Round 3 Statewide Data Indicator Data Dictionary: Provides 
a description of each statewide data indicator and data quality check, 
including the numerators, denominators, risk adjustments, exclusions, 
and corresponding data notes. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-data-dictionary).
    C. Statewide Data Indicators--PDF Version of Syntax: Provides the 
syntax used to perform data quality checks, calculate observed 
performance, and calculate risk-standardized performance on the seven 
statewide data indicators. This document may help individuals develop a 
better understanding of the statistical calculations or provide 
assistance for individuals who do not have SPSS and/or STATA 
statistical software to run the syntax files. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-pdf).
    D. CFSR Round 3 Statewide Data Indicator Syntax Zip File: Provides 
syntax and supporting files in SPSS format that are used to calculate 
data quality and observed performance. Note: Syntax to calculate risk-
standardized performance (RSP) for each state uses a national, risk-
adjusted model that requires child-level data from all states (i.e., 
national datasets) and thus cannot be replicated by states and 
interested parties. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-zip).
    E. Instructions to Run Data Quality (DQ) Checks and Observed 
Performance Syntax for Statewide Data Indicators: Provides an overview 
and brief description of the folders, files, and syntax included in the 
CFSR round 3 zip file; considerations and steps for running the syntax; 
and software requirements. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-dq-instructions).

Invitation To Comment

    We are inviting state child welfare agencies, partner 
organizations, and the public to review, test, and provide comments on 
the revised syntax to perform data quality checks and calculate 
observed performance on the statewide data indicators. The purpose of 
this invitation is to provide an opportunity for additional independent 
review prior to finalizing the revised syntax. Interested parties may 
choose to review a written version of the statistical syntax and 
supporting documents, and/or follow the instructions to replicate the 
syntax files to generate CFSR data indicator performance information 
using state-generated AFCARS and NCANDS files.
    We are interested in receiving comments on the revised syntax and 
related considerations and recommendations to strengthen our 
statistical calculations for generating CFSR statewide data indicator 
performance information. We encourage states and interested parties to 
share the outcome of their review and/or results from running the 
revised syntax files. Please be specific in identifying the 
corresponding statewide data indicator or data quality check being 
referenced in the comment; the syntax revision, question, or 
recommendation; and the rationale for considerations or 
recommendation(s), as applicable.
    Comments containing suggested revisions should be limited to 
operationalizing the measures and calculation methods contained in the 
statistical syntax used to generate performance data. We are not making 
changes or requesting comments at this time pertaining to the statewide 
data indicator measures as defined in prior Federal Register notices 
and the Data Dictionary available with this invitation.

Lynn A. Johnson,
Assistant Secretary for Children and Families.
[FR Doc. 2018-25835 Filed 11-26-18; 8:45 am]
 BILLING CODE 4184-25-P