Announcement of Requirements and Registration for a Prize Competition Titled: Sub-Seasonal Climate Forecast Rodeo, 92848-92851 [2016-30593]

Agencies

• DEPARTMENT OF THE INTERIOR
• Bureau of Reclamation

[Federal Register Volume 81, Number 244 (Tuesday, December 20, 2016)]
[Notices]
[Pages 92848-92851]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2016-30593]


-----------------------------------------------------------------------

DEPARTMENT OF THE INTERIOR

Bureau of Reclamation

[RR08100000, 17XR0680A1, RY.1541CH20.WA01701]


Announcement of Requirements and Registration for a Prize 
Competition Titled: Sub-Seasonal Climate Forecast Rodeo

AGENCY: Bureau of Reclamation, Interior.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: The Bureau of Reclamation is announcing the following prize 
competition, Sub-Seasonal Climate Forecast Rodeo. This Challenge seeks 
to improve on existing sub-seasonal forecasts and asks Solvers (i.e., 
competitors in the Challenge) to develop systems that perform 
demonstratively better than existing baseline forecasts for temperature 
and precipitation over a 15-42-day time frame. Solvers will have 
approximately 3 months to develop their system, at which point they are 
asked to provide forecasts every 2 weeks over a 13-month period, with 
the first month being a ``pre-season'' to become familiar with the 
submission and evaluation processes.

DATES: Listed below are the specific dates pertaining to this prize 
competition. Please note that times in meteorology are based upon a 
world-wide 24-hour clock called Zulu time (Z). Additional detail on 
Zulu time is available in the SUPPLEMENTARY INFORMATION section of this 
notice.
    1. First forecast due on or before 0Z (Zulu) March 21, 2017 (pre-
season), and 0Z April 18, 2017 (actual competition).
    2. Final forecast and hind-cast due on or before 0Z April 3, 2018.
    3. Final submission due on or before 11:59 p.m. (U.S. Eastern Time) 
May 3, 2018.
    4. Judging period ends on August 3, 2018.
    5. Winners announced on or before September 5, 2018.

ADDRESSES: The Sub-Seasonal Climate Forecast Rodeo Prize Competition 
will be posted on the following crowd-sourcing platforms where Solvers 
can register for this prize competition:
    1. The Water Pavilion located at the InnoCentive Challenge Center: 
www.innocentive.com/water-pavilion/.
    2. U.S. Federal Government Challenge Platform: www.Challenge.gov. 
InnoCentive, Inc. is administering this challenge under a challenge 
support services contract with the Bureau of Reclamation. Challenge.gov 
will re-direct the Solver community to the InnoCentive Challenge Center 
as the administrator for this prize competition. Additional details for 
this prize competition, including background information, templates, 
and the Challenge Agreement specific for this prize competition, can be 
accessed through either of these prize competition web addresses. The 
Challenge Agreement contains more details of the prize competition 
rules and terms that Solvers must agree with to be eligible to compete.
    Information pertaining to this competition will be posted to the 
Bureau of Reclamation's current prize competitions Web page at 
www.usbr.gov/research/challenges/current/.

FOR FURTHER INFORMATION CONTACT: Challenge Manager: Dr. David Raff, 
Science Advisor, Bureau of Reclamation, (202) 440-1284, draff@usbr.gov; 
Ken Nowak (303) 445-2197, knowak@usbr.gov.

SUPPLEMENTARY INFORMATION: The Bureau of Reclamation (Reclamation) is 
announcing the following prize competition in compliance with 15 U.S.C. 
3719, Prize Competitions. The intent is to spur innovation toward 
improved forecasts of temperature and precipitation using a real-time 
competition and cash prizes as incentives.
    Prize Competition Summary: Improved sub-seasonal forecasts for 
weather and climate conditions (lead-times ranging from 15 to 45 days 
and beyond) would allow water managers to better prepare for shifts in 
hydrologic regimes such as the onset of drought or occurrence of wet 
weather extremes. The challenge with sub-seasonal weather and climate 
forecasting is that it encompasses the time frame where initial state 
or condition information, such as coupled land-atmosphere processes 
becomes less important, and

[[Page 92849]]

slowly varying long term states, such as sea surface temperature, soil 
moisture, and snowpack, become more important to predictions. In 
addition, the relative importance of the initial state or condition, 
versus the longer term state, depends on the lead time, region of 
interest, and time of year.
    Accurate sub-seasonal weather forecasting has proven to be 
particularly difficult to accomplish but is of great interest to water 
managers tasked with predicting sub-seasonal streamflow and water 
supply. Sub-seasonal forecasting, which spans approximately 15 to 45 
days in the future, is difficult because it bridges short-term 
forecasting, where initial conditions primarily determine upcoming 
weather, and long-term forecasting in which slowly varying factors 
become more important.
    This Challenge is asking Solvers to develop systems that improve 
upon existing sub-seasonal temperature and precipitation forecasts. 
Solvers are not required to develop entire systems from scratch. 
Methods could include, but are not limited to, approaches for improving 
the accuracy of existing sub-seasonal forecasts, techniques that 
leverage climate teleconnections, or statistical models. This Challenge 
will be active for approximately 17 months, starting with a 3-month 
development period followed by a 1-month ``pre-season'', and a 12-month 
competition period. Following the competition period, Solvers will have 
1 month to prepare final submissions. During the competition period, 
Solvers will be required to upload sub-seasonal forecasts every 2 
weeks. An online leaderboard hosted by the National Integrated Drought 
Information System at drought.gov will track and display Solvers' 
performance for the duration of the competition period. Please note 
that InnoCentive usernames will be shared with the National Oceanic and 
Atmospheric Administration (NOAA) as part of the forecast evaluation 
process and leaderboard tracking. At the conclusion of the competition 
period, a final submission is required.
    Prizes total $800,000. Four categories are defined by two forecast 
outlook periods and two forecast variables (temperature and 
precipitation). In each category, prizes for eligible solvers are as 
follows:

1st place--$100,000
2nd place--$50,000
3rd place--$25,000

    In addition, one $25,000 prize per category may be awarded to an 
eligible solver based solely on hind-cast performance, submission of 
which is a requirement to be eligible for the above listed prizes.
    Final submissions to the Challenge should include the following:
    1. The detailed description of the proposed Solution addressing the 
specific Technical Requirements that are presented in the Detailed 
Description of the Challenge. This description should be accompanied by 
a well-articulated rationale for the proposed Solution.
    2. Locally executable application and corresponding source code to 
implement the proposed Solution that are documented.
    3. Proof-of-concept data obtained as outlined in the Detailed 
Description of the Challenge.
    The Challenge award is contingent upon theoretical evaluation and 
operational validation of the submitted Solutions by the Seeker 
(Reclamation).
    To receive an award, the Solvers will grant to the Seeker a non-
exclusive license to practice their solutions and make the solution 
available as open source software licensable under either Berkeley 
Software Distribution or General Public License Open Source license.
    Technical Requirements. Water managers and other users have 
indicated that they want the existing forecasts to be better rather 
than wanting the development of new forecast systems. Therefore, this 
competition seeks new and novel forecast method(s) to improve the 
forecasting of precipitation and temperature that can be incorporated 
into existing frameworks relied upon for water management. Solvers may 
leverage existing forecasts or ensembles in their solution, but must be 
able to demonstrate appreciable value added by the solution relative to 
any input or foundational framework. Specifically, the competition 
desires solutions that can outperform current operational forecasts 
(including forecasts from operational centers outside the U.S.) and a 
damped persistence forecast at a 1x1 degree gridded resolution for the 
western United States at two forecast outlooks: 15-28 days (weeks 3-4) 
and 29-42 days (weeks 5-6) for temperature and precipitation. Overall 
skill- spatially and across a range of weather/climate conditions- is 
most important. The ability to skillfully forecast extreme conditions 
is also very desirable. Any system that meets these criteria is sought.
    Evaluation criteria. Forecast skill will be evaluated for 
temperature and precipitation separately since the drivers responsible 
for prediction of these variables are different and the subsequent 
skill level is also expected to be different. Moreover, the 15-28-day 
and 29-42-day periods will be evaluated individually for similar 
reasons. Winning forecasts must outperform NOAA's sub-seasonal modeling 
system, the Climate Forecast System Version 2 (CFSv2) and damped 
persistence forecasts (see definitions below). Specifically, skill will 
be evaluated individually for temperature and precipitation for weeks 
3-4 and weeks 5-6 as the highest skill over the competition's 
identified geographic area, averaged over the entire competition time 
period. To be prize eligible, Solvers must also demonstrate historical 
skill of statistical significance that is equal to or greater than that 
of the CFSv2 through submission of a hind-cast analysis described 
below.
    Definitions.
     Anomaly is defined as the difference between a given value 
and climatology for a specific location (grid cell by grid cell) and 
time.
     Climatology is defined as average temperature and 
precipitation for a specific 2-week period as computed from the Climate 
Prediction Center's daily unified gauge data set for precipitation at 
ftp://ftp.cpc.ncep.noaa.gov/precip/CPC_UNI_PRCP/GAUGE_GLB/, and the 
Climate Prediction Center's global gridded temperature data set over 
the 30 year period of 1981-2010 at ftp://ftp.cpc.ncep.noaa.gov/precip/wd52ws/global_temp/.
     A damped persistence forecast will be represented using 
seasonally developed regression coefficients based on the historical 
climatology period of 1981-2010 that relate observations of the past 2 
weeks to the forecast outlook periods on a grid cell by grid cell 
basis.
     Skill is defined as spatial anomaly correlation, averaged 
over time. Final competition standings will be the average spatial 
anomaly correlation of all 26 forecasts issued for the western U.S. 
geographic area, as defined by the forecast submission template.
    The CFSv2 forecasts at https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2 will be 
combined into a forecast baseline over the same time period as Solver 
submissions using an ensemble mean of the 32 ensemble-member forecasts 
leading up to each competition submission. The CFSv2 is run out to 45 
days, 4 times daily, with four initial conditions per run for a total 
of 16 forecasts per day. For example, for a forecast due March 1st, the 
16 CFSv2 ensemble-member forecasts issued on February 27 and the 16 
CFSv2 ensemble-member forecasts issued on

[[Page 92850]]

February 28 will be averaged together for each grid and over the two 
forecast-periods to become the forecast baseline for the March 1st due 
date. Further, the ensemble mean will be bias corrected using the 
method employed by NOAA's Climate Prediction Center in developing their 
operational forecasts. Specifically, this process involves comparing 
the CFSv2 reforecasts for the period of 1999-2010 with observed data to 
establish and correct daily bias on a grid cell by grid cell basis. 
This is done for temperature and precipitation at both forecast outlook 
periods.
    Forecast submission requirements and instructions. Over the course 
of this Challenge, Solvers will be required to submit 2 pre-season 
forecasts and 26 forecasts during the year-long competition, for each 
of the four categories detailed above and must meet the following 
requirements:
    1. Beginning at the start of the competition period and every 2 
weeks thereafter (see below for required submission deadlines) for the 
duration of the competition, Solvers will submit 1x1 degree gridded 
forecasts corresponding to western U.S. competition geographic area as 
defined by the forecast template available on Reclamation's current 
competitions Web page listed in the ADDRESSES section of this notice 
for:
    a. Average temperature (degrees C) for days 15-28.
    b. Cumulative precipitation (millimeters) for days 15-28.
    c. Average temperature (degrees C) for days 29-42.
    d. Cumulative precipitation (millimeters) for days 29-42.
    2. To be eligible for an award, Solvers must submit all 26 
forecasts on time. A grace of two missed forecast per category will be 
allowed. For the purpose of computing skill, climatology will be 
substituted for a missed forecast. More than two missed forecast in any 
category will result in disqualification.
    3. Forecast submissions will be made through a portal hosted by 
NOAA and must utilize the supplied template, available on Reclamation's 
current prize competitions Web page.
    4. The first month of the 13-month competition period (i.e. the 
first two forecasts) will be considered a `pre-season' for Solvers to 
become familiar with the submission process and will not count toward 
their standing in the competition or against the two missed forecast 
grace described above.
    Forecast submission deadlines. The forecast competition will run 
for 1 year following the `pre-season' with new forecast submissions 
required every 2 weeks. All forecasts need to be received by 0Z on the 
day of the required forecast submission. All aspects of meteorology are 
based upon a world-wide 24-hour clock called Zulu time (Z), more 
commonly called Coordinated Universal Time (UTC). More information on 
this topic is available on the National Weather Service Web site at 
https://www.srh.noaa.gov/jetstream/synoptic/time.html, including 
conversion to common U.S. time zones. Specific forecast submission 
deadlines, including dates and times, will be available on 
Reclamation's current prize competitions Web page listed in the 
ADDRESSES section of this notice, the first of which is due 
approximately 3 months following the announcement of this challenge.
    In addition to requirements listed above for the forecast 
submissions, proposed solutions must meet the following Technical 
Requirements:
    1. For first, second or third place, systems must outperform, based 
on the formula described above, cumulatively over the competition time 
period, the CFSv2 baseline and damped persistence forecasts for at 
least one category: Week 3-4 temperature, week 5-6 temperature, week 3-
4 precipitation, or week 5-6 precipitation.
    2. Must demonstrate historical skill of statistical significance 
that is equal to or greater than that of the CFSv2 through submission 
of a hind-cast analysis. The CFSv2 reforecasts at https://nomads.ncdc.noaa.gov/thredds/catalog/cfsr-hpr-ts45/catalog.html will 
provide the baseline for the hind-cast, in the same way that the CFSv2 
forecasts provide a baseline for the real-time competition. Two notable 
distinctions between the CFSv2 forecasts and reforecasts are (1) the 
reforecast baseline will be based on a mean of 8 ensemble members (only 
one set of initial conditions were used, thereby producing 4 
reforecasts per day) and (2) bias correction to the reforecast will be 
developed and applied for each reforecast year independently, so as to 
not leverage knowledge of that year's reforecast performance toward 
bias correcting itself. Note that skill and prize eligibility will be 
evaluated for each category individually--i.e. to be prize eligible in 
a particular category, Solvers need only outperform the CFSv2 in that 
category. To do this, Solvers will submit, hind-casts for the four 
categories, issued every 2 weeks, for the period of 1999-2010, no later 
than the Challenge final forecast due date. The hind-cast should be 
performed as a ``leave one out cross validation.'' This is accomplished 
by removing one year of observed historical data from the period of 
1999-2010, calibrating the model based on the remaining years of data, 
and forecasting that year. Hind-casts are to be issued at the month/day 
combinations specified in the forecast submission deadlines referenced 
above. Given the dates of the competition, each hind-cast will span 2 
calendar years, thereby resulting in 11 1-year hind-cast periods for 
the ``leave one out cross validation.'' This process should be repeated 
until forecasts have been issued for all 11 years. Solvers will submit 
one historical hind-cast per category for the 11 years. For more 
information on this technique, see the discussion on cross-validation 
by Clarke et al. in Principles and Theory for Data Mining and Machine 
Learning (2009).
    3. Solvers may utilize any available sub-seasonal forecast as the 
starting point for their system, but must demonstratively improve upon 
that forecast.
    4. Must be written in C++, R, python, Fortran, or other widely 
recognized programming languages and be licensable under either 
Berkeley Software Distribution or General Public License Open Source 
license.
    5. Solvers with winning solutions are required to provide all code, 
data, and other components of their forecast system necessary to run 
the system and reproduce the forecasts issued in the competition. 
Failure of code to reproduce performance during the competition or for 
the hind-cast may result in disqualification. Furthermore, the Solvers 
may be required to iterate with competition judges to ensure 
documentation is sufficient.
    Project Deliverables: In addition to the hind-casts, 2 pre-season 
forecast submissions, and 26 forecast submissions during the year-long 
competition, Solvers must submit a final proposed Solution by the 
Challenge deadline. The submitted proposal should include the 
following:
    1. Detailed description of a sub-seasonal forecasting system that 
meets the Technical Requirements listed in the Detailed Description and 
Requirements section of the Challenge.
    2. A well-reasoned rationale supporting the methodology of the 
proposed system and addressing each of the Technical Requirements 
described in the Detailed Description.
    3. Locally executable application and corresponding documented 
source code implementing the proposed Solution.
    The Seeker may wish to partner with the Solver at the conclusion of 
the Challenge. Solver should describe their expertise and include a 
statement indicating their interest in this opportunity.

[[Page 92851]]

    The proposal should not include any personal identifying 
information (name, username, company, address, phone, email, personal 
Web site, resume, etc.) or any information the Solvers may consider as 
their Intellectual Property they do not want to share.
    Judging: An online leaderboard hosted by the National Integrated 
Drought Information System will track and display Solvers' performance 
for the duration of the competition period. The Challenge award is 
contingent upon theoretical evaluation and operational validation of 
the submitted Solutions by the Seeker. If multiple proposals meet all 
the Solution Requirements, the Seeker reserves the right to award only 
the top three solutions per category which they believe are of sound 
technical foundation. After the Challenge final submission deadline, 
submissions will be identified as potentially prize eligible, as 
determined by the quantitative forecast evaluation performed by NOAA 
and described above. Of those potentially prize eligible solutions, the 
Judging Panel will evaluate each with respect to the Solution 
Requirements and make a decision on winning solution(s). The Judging 
Panel may be composed of Federal and/or Non Federal scientists, 
engineers, and other technical experts, including subject matter 
experts from the listed collaborators for this Challenge. All Solvers 
that submit a proposal will be notified on the status of their 
submissions. Decisions by the Seeker cannot be contested.
    Eligibility Rules: To be able to win a prize under this 
competition, an individual or entity must:
    1. Agree to the rules of the competition (15 U.S.C. 3719(g)(1));
    2. Be an entity that is incorporated in and maintains a primary 
place of business in the United States, or (b) in the case of an 
individual, a citizen or permanent resident of the United States (15 
U.S.C. 3719(g)(3)).
    However, submissions can be entertained from all Solvers regardless 
of whether they are U.S. citizens/entities. Meritorious submissions 
from non-eligible persons and entities, if any, will be recognized in 
publications issued by the Seeker announcing the results of the 
competition, such as press releases. Non-U.S citizens/permanent 
residents or non-U.S entities can also be included on U.S. teams. 
However, prizes--whether monetary or otherwise--will only be awarded to 
eligible persons and entities under the authority of the America 
COMPETES Reauthorization Act of 2010 (15 U.S.C. 3719).
    3. Not be a Federal entity or Federal employee acting within the 
scope of their employment (15 U.S.C. 3719(g)(4)). A Federal entity is 
defined by 5 U.S.C. Appendix 8G with a list of current Federal entities 
periodically posted on the Federal Register.
    4. Assume risks and waive claims against the Federal Government and 
its related entities (15 U.S.C. 3719(i)(1)(B)); and,
    5. Not use Federal facilities, or consult with Federal employees 
during the competition unless the facilities and employees are made 
available to all individuals and entities participating in the 
competition on an equitable basis.
    The following individuals or entities are not eligible regardless 
of whether they meet the criteria set forth above:
    1. Any individual or organization who employs an evaluator on the 
Judging Panel or otherwise has a material business relationship or 
affiliation with any Judge.
    2. Any individual who is a member of any Judge's immediate family 
or household.
    3. The Seeker, participating organizations, and any advertising 
agency, contractor or other individual or organization involved with 
the design, production, promotion, execution, or distribution of the 
prize competition; and all employees, and all members of the immediate 
family or household of any such individual or organization.
    4. Any individual or entity that uses Federal funds to develop the 
proposed solution now or any time in the past, unless such use is 
consistent with the grant award, or other applicable Federal funds 
awarding document. Note: Individuals or entities that have been funded 
by the Federal Government in the past to work within the technical 
domain of the competition are eligible provided their specific 
submission was not developed by them with Federal funds. Submissions 
that propose to improve or adapt existing federally funded technologies 
for the solution sought in this prize competition are also eligible. 
Individuals are also encouraged to consult with their employer Ethics 
Officer for additional guidance and considerations.
    Consultation: Reclamation and collaborator scientists, engineers, 
and technical specialists were consulted in identifying and selecting 
the topic of this prize competition. Direct and indirect input from 
various stakeholders and the broader water resources community of 
practice were also considered.
    Public Disclosure: InnoCentive, Inc. is administering this 
challenge under a challenge support services contract with Reclamation. 
Participation is conditioned on providing the data required on 
InnoCentive's online registration form. Personal data will be processed 
in accordance with InnoCentive's Privacy Policy which can be located at 
https://www.innocentive.com/privacy.php. Before including your address, 
phone number, email address, or other personal identifying information 
in your proposal, you should be aware that the Seeker is under no 
obligation to withhold such information from public disclosure, and it 
may be made publicly available at any time. Neither InnoCentive nor the 
Seeker is responsible for human error, theft, destruction, or damage to 
proposed solutions, or other factors beyond its reasonable control.
    Liability and Indemnification: By participating in this Challenge, 
each Solver agrees to assume any and all risks and waive claims against 
the federal government and its related entities, except in the case of 
willful misconduct, for any injury, death, damage, or loss of property, 
revenue, or profits, whether direct, indirect, or consequential, 
arising from participation in this Challenge, whether the injury, 
death, damage, or loss arises through negligence or otherwise. By 
participating in this Challenge, each Solver agrees to indemnify the 
federal government against third party claims for damages arising from 
or related to Challenge activities
    No Insurance Required: Based on the subject matter of the 
Challenge, the type of work that it will possibly require, as well as 
an analysis of the likelihood of any claims for death, bodily injury, 
or property damage, or loss potentially resulting from competition 
participation, Solvers are not required to obtain liability insurance 
or demonstrate financial responsibility in order to participate in this 
Challenge.

    Dated: December 7, 2016.
Levi Brekke,
Acting Science Advisor.
[FR Doc. 2016-30593 Filed 12-19-16; 8:45 am]
 BILLING CODE 4332-90-P