CPC Verification Summary

[+] Heidke

[+] RPSS

The Rank Probability Skill Score (RPSS) measures the improvement of multi-category forecasts relative to a reference forecast and takes into account the probability for each category.

Ranked Probability Score (RPS) is a squared error score with respect to the cumulative probabilities for multi-category forecasts and whether or not the event occurred. The Ranked Probability Skill Score (RPSS) measures the improvement of the multi-category forecast relative to a reference forecast (the sample climatology). RPSS values range from -Infinity to 1. Forecasts made with higher probabilities are penalized heavily if they are wrong, and forecasts made with lower probabilities are penalized less severely if they are wrong, since they aren't expected to be correct as often. Conversely, forecasts made with higher probabilities are rewarded more heavily if they are correct, and forecasts made with lower probabilities are rewarded less heavily if they are correct.

EC forecasts for monthly and seasonal forecasts are included in scores.

The equation for the score is:

RPSS = 1 - RPSforecast/RPSreference

The RPS score representing a collection of forecasts in space or time is:

where n is the total number of valid forecast-observation pairs. For the chart, each forecast data point in space is evaluated and represents each k_th value. probB_k, probN_k, probA_k are the probabilities of the below normal, near normal, and above normal forecast categories respectively at each point k, and obsB_k, obsN_k, obsA_k are either 0 if that category was observed, or 1 if that category was not observed at each point k.

The score is the sum of the squares of the difference between the forecast probability of each category and either "0" or "1" for the category observed at that point over all forecast points, divided by the total number of forecast points used. To get RPS_forecast, the forecast probabilities are used. To get RPS_reference, cumulative probabilities starting at 1/3 are used for each category.

To learn about what each of the options, you can click on the question mark next to each option label to pop-out more information about each.

In general, the difference between the "Chart" and "Map" tabs is that the chart displays time series of scores. These scores represent the Continental US (CONUS) at each time step. The map represents scores for each location across the period of time displayed in the titles of the graphics.

The labels on the x-axis of the charts represent the center dates of the valid periods. The term 'valid period' refers to the period that the forecast was made to represent. The best way to think of this is to think of the chart results in terms of "how well the forecast of a range of time (which is represented by the center date) compared to what was actually observed".

Even though the date in the time series chart results may be labeled as a single day, it does not mean that it represents a daily forecast. The 6-10 day and 8-14 day forecasts represent a range of time across 5 days and 7 days, respectively. Each day, the Climate Prediction Center releases updated forecast maps that represent these time periods. In the verification results, 6-10 day and 8-14 day forecasts are labeled as "YYYYMMDD", which is the center day in the time range the forecast is for.

For example, a score value on the resulting time series chart with the date 7/30/2011 for a 6-10 day forecast (which spans across 5 days), represents the skill of the forecast that was made for the period July 28, 2011 - August 1, 2011. If this date is shown for an 8-14 day forecast (which spans across 7 days), the skill score represents the skill of the forecast made for July 27, 2011 - August 2, 2011. In both cases, July 30, 2011 is the middle date of those periods.

Additionally, a chart date can have the format "YYYYMM". If this applies to a monthly forecast skill score, the value is for that month and year. However, if it is a seasonal forecast score, the date on the results represents the middle month and year of the 3-month period that the forecast is valid for. For example, a score value for 7/2011 represents a seasonal forecast valid for June-July-August 2011.

The y-axis represents the score values.

All the forecast and observation datasets used to produce the scores are in a 2 x 2 degree gridded format originally.

The categories of observations are used in the skill scores along with the forecast categories. The observation categories are determined by using the climatology data from 1981-2010 as the reference period.

Sometimes text such as "Image not available" may be displayed instead of some graphics on the page. To get more details on why a graphic is not displayed, you can click the "Why?" link where the graphic should be, which will jump to the warnings and errors section at the bottom of the page.

You can download or view data in ASCII text format by using the 'Download data' links under the associated graphics. The header of the file contains information about the options used in the score calculations and uses a pound '#' symbol at the beginning of each header line. The percent and number of valid scores are included in the header. This represents the percent and number of scores in the file that are considered valid (non-NaN values).

The data in the ASCII files is space delimited. Missing values are denoted as 'NaN'.

The content and header names (described below in single quotes and bold font) of the columns are described below in the order they are listed in the file from the left to right:

• Reference point - The type of data in this column depends on whether it represents a chart or map :
• chart ('Date_Valid') : Valid date of forecast. The date is expressed as the middle date of the valid period that the forecast represents. This is formatted as either MM/dd/yyyy (medium-range) or MM/yyyy (long-range).
• map ('id') : Assigned gridpoint value formatted as XXYY, where XX and YY are zero-padded integers representing a gridpoint in the X and Y location, respectively. The bottom left gridpoint is 0101 and the top right gridpoint is 3619.
• Latitude and longitude ('lat','lon') - This is for maps only. Position of the associated gridpoint location. The lat and lon represents the center point of the associated gridpoint.
• Forecast source ('Forecast_Source') - Forecast type (ie. 'manual' which is the official CPC forecast).
• Scores ('all_cats')- Skill scores. The header is labeled 'all_cats' because the scores take into account all forecast categories (above-normal, normal, and below-normal) in the score calculation.
• Percent Valid Dates ('Percent_Valid_Locations','Percent_Valid_Dates')- Percent valid (non-NaN) locations (dates) that went into the calculation of each score for charts (maps).

Sometimes text such as "Image not available" may be displayed instead of some graphics on the page. To get more details on why a graphic is not displayed, you can click the "Why?" link where the graphic should be, which will jump to the warnings and errors section at the bottom of the page.

6-10 Day and 8-14 Day Forecasts: These forecasts represent the period of 6 - 10 days and 8 - 14 days after the date issued. The dates that you select to verify will calculate scores for the forecasts where the date is the center of those periods.

The "CPC Official (Manual)" forecast refers to forecasts that were manually created by forecasters. These are updated daily by CPC. The forecasts from models (which forecasters often use as guidance tools to create their forecasts) are available to verify as well. The "CPC Official (Auto)" forecast refers to forecasts that are not created manually by forecasters. These forecasts are created by an automated process, which combines other model forecasts. During the weekend, these forecasts are quality controlled by a forecaster and issued as the official forecast.

Monthly Forecasts: The CPC official forecasts are manually created by a forecaster and updated the middle of every month and are valid the next month. The 0.5 month lead time forecast refers to what is considered the "official" forecast, and the 0 month lead time forecast refers to the "revised" forecast, which is a manually created forecast issued the last day of the month for the following month.

Seasonal Forecasts: The CPC official forecasts are manually created by a forecaster and updated the middle of every month and are valid for various 3-month periods after the month issued. The below models are considered forecast tools used by forecasters as guidance to create the official forecast. The definitions can be found at:

CPC Tools Definition