Submission for OMB Review; Comment Request, 41029-41031 [2013-16489]

Agencies

• DEPARTMENT OF COMMERCE

[Federal Register Volume 78, Number 131 (Tuesday, July 9, 2013)]
[Notices]
[Pages 41029-41031]
From the Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 2013-16489]


=======================================================================
-----------------------------------------------------------------------

DEPARTMENT OF COMMERCE


Submission for OMB Review; Comment Request

    The Department of Commerce will submit to the Office of Management 
and Budget (OMB) for clearance the following proposal for collection of 
information under the provisions of the Paperwork Reduction Act (44 
U.S.C. chapter 35).
    Agency: U.S. Census Bureau.
    Title: Generic Clearance for Questionnaire Pretesting Research.
    OMB Control Number: 0607-0725.
    Form Number(s): Various.
    Type of Request: Extension of a currently approved collection.
    Burden Hours: 16,500 over the next three years.
    Number of Respondents: 5,500 annually.
    Average Hours per Response: 1 hour.
    Needs and Uses: In recent years, there has been an increased 
interest among federal agencies and others in the importance of testing 
questionnaires. This interest has been spurred by a recognition that 
the traditional methods of pretesting are weak tools for evaluating 
questionnaires and procedures. These methods consist of a small 
``hothouse'' field test accompanied by interviewer debriefing, and the 
information collected through their use is quite limited in its ability 
to detect and diagnose problems with the instruments and the procedures 
being tested.
    In response to this recognition, new methods have come into popular 
use, which are useful for identifying questionnaire and procedural 
problems, suggesting solutions, and measuring the relative 
effectiveness of alternative solutions. Through the use of these kinds 
of techniques, employed routinely in the testing phase of Census Bureau 
surveys, questionnaires can be simplified for respondents, respondent 
burden can be reduced, and the quality of the questionnaires used in 
continuing and one-time surveys can be improved. Thus an increase in 
the quality of the data collected through these surveys can be achieved 
as well.
    In September 1991, the Census Bureau requested and received a 
generic clearance (Number 0607-0725) on an experimental basis, which 
relaxed some of the time constraints and enabled the Census Bureau to 
begin conducting extended cognitive and questionnaire design research 
as part of testing for its censuses and surveys. The clearance covered 
data collections in the demographic, economic, and decennial areas of 
the Bureau, and specifically applied to research that is focused on 
questionnaire design and procedures aimed at reducing measurement 
errors in surveys. Research on paying respondents was specifically 
excluded from the clearance. As part of the experimental clearance, the 
Census Bureau submitted to the Office of Management and Budget (OMB) a 
report that contained indicators of the work that was conducted under 
the clearance. At the end of the experimental period, the Census Bureau 
requested and received a three-year renewal of the clearance (through 
December 1995), covering the same kinds of research activities. As part 
of the clearance, the Census Bureau has submitted to OMB a report of 
pretesting activities at the end of each year of the clearance.
    Subsequently, the Census Bureau has received six more three-year 
renewals of the generic clearance for pretesting (through August 2013). 
The current clearance contains approval for three additional types of 
activities: Research about incentives, expanded field tests conducted 
to include split sample questionnaire experiments in multiple panels, 
and usability testing of electronic instruments.
    At this time, the Census Bureau is seeking another three-year 
renewal of the generic clearance for pretesting, with the same 
conditions as the previous clearance. This will enable the Census 
Bureau to continue providing support for pretesting activities, which 
is important given the length of time required to plan the activities.
    The specific methods proposed for coverage by this clearance are 
described below. Also outlined are the procedures in place for keeping 
the Economics and Statistics Administration and OMB informed about the 
identity of the surveys and the nature of the research activities being 
conducted.
    The methods proposed for use in questionnaire development are as 
follows:
    Field test. For the purposes of this clearance, we are defining 
field tests as small data collection efforts of 500 cases or less, 
conducted among either purposive or statistically representative 
samples, for which evaluation of the questionnaire and/or procedures is 
the main objective and no plans to publish the data other than for 
purely methodological purposes are envisioned.
    Field tests are an essential component of this clearance package 
because they serve as the vehicle for conducting standardized behavior 
coding of the interaction between the respondent and the interviewer. 
This methodology does not require any additional data collection above 
and beyond the field test--it involves applying a standardized coding 
scheme to the completion of a field interview, either by a coder using 
a tape-recording of the interview or by a ``live'' observer at the time 
of the interview. The coding scheme is designed to identify situations 
that occur during the interview that reflect problems with the 
questionnaire. For example, if respondents frequently interrupt the 
interviewer before the question is completed, the question may be too 
long. If respondents frequently give inadequate answers, this suggests 
there are some other problems with the question. Quantitative data 
derived from this type of standardized coding scheme can provide 
valuable information to identify problem areas in a questionnaire, and 
research (``New Techniques for Pretesting Survey Questions'' by 
Cannell, Kalton, Oksenberg, Bischoping, and Fowler, 1989) has 
demonstrated that this is a more objective and reliable method of 
identifying problems than the traditional interviewer debriefing, which 
is typically the sole tool used to evaluate the results of a 
traditional field test.
    Interviewer debriefing has advantages as well, since it utilizes 
the knowledge of the employees who have the closest contact with our 
respondents. In conjunction with other methods, we plan to use this 
method in our field tests to collect information about how interviewers 
react to the survey instruments.
    Field tests conducted under this clearance will involve either 
purposive or statistically representative samples. Under this clearance 
a variety of surveys will be pretested, and the exact nature

[[Page 41030]]

of the surveys and the samples is undetermined at present. However, due 
to the small nature of the tests, we expect that some will not involve 
representative samples. In these cases, samples will basically be 
convenience samples, which will be limited to specific geographic 
locations and may involve expired rotation groups of a current survey 
or census blocks that are known to have specific aggregate demographic 
characteristics. The needs of the particular sample will vary based on 
the content of the survey being tested, but the selection of sample 
cases will not be completely arbitrary in any instance.
    Respondent debriefing questionnaire. In this method, standardized 
debriefing questionnaires are administered to respondents who have 
participated in a field test. The debriefing form is administered at 
the end of the questionnaire being tested, and contains questions that 
probe to determine how respondents interpret the questions and whether 
they have problems in completing the survey/questionnaire. This 
structured approach to debriefing enables quantitative analysis of data 
from a representative sample of respondents, to learn whether 
respondents can answer the questions, and whether they interpret them 
in the manner intended by the questionnaire designers.
    Split sample experiments. This involves testing alternative 
versions of questionnaires, at least some of which have been designed 
to address problems identified in draft questionnaires or 
questionnaires from previous survey waves. The use of multiple 
questionnaires, randomly assigned to permit statistical comparisons, is 
the critical component here; data collection can include mail, 
telephone, or personal visit interviews or group sessions at which 
self-administered questionnaires are completed. Comparison of revised 
questionnaires against a control version, preferably, or against each 
other facilitates statistical evaluation of the performance of 
alternative versions of the questionnaire.
    In any split sample experiments conducted under this clearance, 
alternative questionnaire versions will be tested. The number of 
versions tested and the number of cases per version will depend on the 
objectives of the test. We cannot specify with certainty a minimum 
panel size, although we would expect that no questionnaire versions 
would be administered to less than fifty persons in a split sample 
test.
    Split sample tests that incorporate methodological questionnaire 
design experiments will have a larger maximum sample size (up to 
several hundred cases per panel) than field tests using other pretest 
methods. This will enable the detection of statistically significant 
differences, and facilitate methodological experiments that can extend 
questionnaire design knowledge more generally for use in a variety of 
Census Bureau data collection instruments. The Census Bureau will 
consult with OMB prior to submission regarding split sample tests with 
sample sizes over 1000.
    Cognitive interviews. This method involves intensive, one-on-one 
interviews in which the respondent is typically asked to ``think 
aloud'' as he or she answers survey questions. A number of different 
techniques may be involved, including asking respondents to paraphrase 
questions, probing questions asked to determine how respondents came up 
with their answers, and so on. The objective is to identify problems of 
ambiguity or misunderstanding, or other difficulties respondents may 
have answering questions. This is frequently the first stage of 
revising a questionnaire.
    Usability Interviews. This method involves getting respondent input 
to aid in the development of automated questionnaires and Web sites and 
associated materials. A number of different techniques may be involved, 
such as one-on-one usability interviews with think aloud, probing, and 
paraphrasing tasks, card-sorting techniques, and disability 
accommodation testing. The objective is to identify problems that keep 
respondents from completing automated questionnaires accurately and 
efficiently, with minimal burden or that prevent respondents from 
successfully navigating Web sites and finding the information they 
seek.
    Focus groups. This method involves group sessions guided by a 
moderator, who follows a topical outline containing questions or topics 
focused on a particular issue, rather than adhering to a standardized 
questionnaire. Focus groups are useful for surfacing and exploring 
issues (e.g., confidentiality concerns) which people may feel some 
hesitation about discussing.
    This clearance will only cover pretests that involve more extensive 
testing than the traditional field test with interviewer debriefing as 
the only evaluative component. Since the types of surveys included 
under the umbrella of the clearance are so varied, it is impossible to 
specify at this point what kinds of activities would be involved in any 
particular test. But at a minimum, one of the types of testing 
described above or some other form of cognitive pretesting would be 
incorporated into the testing program for each survey.
    We will provide OMB with a copy of questionnaires, protocols, and 
debriefing materials in advance of any testing activity. Depending on 
the stage of questionnaire development, this may be the printed 
questionnaire from the last round of a survey or a revised draft based 
on analysis of other evaluation data. When the time schedule for a 
single survey permits multiple rounds of testing, the questionnaire(s) 
for each round will be provided separately. When split sample 
experiments are conducted, either in small group sessions or as part of 
a field test, all the questionnaires to be used will be provided. For a 
test of alternative procedures, the description and rationale for the 
procedures would be submitted. A brief description of the planned field 
activity will also be provided. OMB will endeavor to provide comments 
on substantive issues within 10 working days of receipt.
    Any large field tests or dress rehearsals that follow from the 
initial questionnaire development activity included here are not 
covered by this generic clearance. Separate submissions for any such 
data collection efforts will be made.
    The Census Bureau will consult with the Economics and Statistics 
Administration (ESA) and OMB prior to submission on the appropriateness 
of submissions under this clearance that may raise policy or 
substantive issues. With respect to ESA, this will include all research 
and testing related to the American Community Survey (ACS) and any 
testing of any activities directly related to the 2020 decennial. In 
addition, the Census Bureau will consult with ESA on any research and 
testing proposals that are presented to the Data Stewardship Executive 
Policy (DSEP) Committee. Consultation with ESA includes the Census 
Bureau providing copies of questionnaires, protocols, and debriefing 
materials in advance of any of the above-mentioned activities.
    The Census Bureau will send ESA and OMB an annual report at the end 
of each year summarizing the number of hours used, as well as the 
nature and results of the activities completed under this clearance.
    The information collected in this program of developing and testing 
questionnaires will be used by staff from the Census Bureau and 
sponsoring agencies to evaluate and improve the quality of the data in 
the surveys and censuses that are ultimately conducted. None of the 
data collected under this

[[Page 41031]]

clearance will be published for its own sake.
    Because the questionnaires being tested under this clearance are 
still in the process of development, the data that result from these 
collections are not considered official statistics of the Census Bureau 
or other Federal agencies. Data will be included in research reports 
prepared for sponsors inside and outside of the Census Bureau. The 
results may also be prepared for presentations related to survey 
methodology at professional meetings or publications in professional 
journals.
    Affected Public: Individuals or households, business or other for-
profit.
    Frequency: On occasion.
    Respondent's Obligation: Voluntary.
    Legal Authority: Data collection for this project is authorized 
under the authorizing legislation for the questionnaire being tested. 
This may be Title 13, Sections 131, 141, 161, 181, 182, 193, and 301 
for Census Bureau-sponsored surveys, and Title 13 and 15 for surveys 
sponsored by other Federal agencies. We do not now know what other 
titles will be referenced, since we do not know what survey 
questionnaires will be pretested during the course of the clearance.
    OMB Desk Officer: Brian Harris-Kojetin, (202) 395-7314.
    Copies of the above information collection proposal can be obtained 
by calling or writing Jennifer Jessup, Departmental Paperwork Clearance 
Officer, (202) 482-0336, Department of Commerce, Room 6616, 14th and 
Constitution Avenue NW., Washington, DC 20230 (or via the Internet at 
jjessup@doc.gov).
    Written comments and recommendations for the proposed information 
collection should be sent within 30 days of publication of this notice 
to Brian Harris-Kojetin, OMB Desk Officer either by fax (202-395-7245) 
or email (bharrisk@omb.eop.gov).

    Dated: July 3, 2013.
Glenna Mickelson,
Management Analyst, Office of the Chief Information Officer.
[FR Doc. 2013-16489 Filed 7-8-13; 8:45 am]
BILLING CODE 3510-07-P