Table of Contents
1.0 Models 2.0 Observations 3.0 Stratification 4.0 Time Increment 5.0 Output
The DTC Winter Forecast Experiment (DWFE) was motivated by the needs of the National Weather Service to improve model guidance in support of their winter weather forecast and warning mission. The DWFE experiment uses high-resolution (5 km) NWP models with improved physics, in an effort to offer a solution. The experiment will run from 15 January 2005 through 31 March 2005 over a CONUS domain, with a special emphasis on the Eastern United States. More information can be found on the DTC's website.
The Real-Time Verification System (RTVS) is being utilized for the DWFE project to perform verification on numerous model forecast fields, including precipitation. There are two types of precipitation verification: grid-to-grid and grid-to-point (station-based verification). These two methods are utilized in an effort to allow the user an option in the verification technique applied to this problem. The literature has shown that there are scale representativeness issues in both techniques.
This grid-to-point web page (RTVS Station Based Precipitation Verification) provides verification statistics for model forecasts bi-linearly interpolated to more than 4,500 Hydrometeorological Automated Data System (HADS) gauge observation sites.
1.0 Models
The model refers to the forecast model being verified. There are three possible selections available to the user for DWFE verification, which are the Advanced Research WRF (WRF/ARW), the Non-Hydrostatic Mesoscale Model (WRF/NMM), and the operational Eta Model. Statistics can be generated for any combination of the models by checking the boxes next to the model names.
1.1 Advanced Research WRF (WRF/ARW)
The ARW is run at NCAR at a 5 km resolution. The ARW features a Eulerian mass (hydrostatic pressure) vertical coordinate and an Eulerian solver for the fully compressible nonhydrostatic equations. Detailed information about the WRF/ARW can be found at the following website.
1.2 Non-Hydrostatic Mesoscale Model (WRF/NMM)
The NMM was developed at NCEP and is currently being run at the Earth System Research Laboratory (ESRL) at a 5 km resolution. The NMM offers a non-hydrostatic dynamic core as a second option to the Eulerian mass core model. More information about the NMM can be found in the following PDF file.
1.3 NCEP Operational Eta Model
The Eta model is run operationally at NCEP on a 12 km grid. More information about this model can be found at the following website.
2.0 Observations
2.1 Precipitation analysis
The DWFE Station-Based precipitation field is verified against the hourly precipitation gauge data from the Hydrometeorological Automated Data System (HADS). HADS is a real-time data acquisition and data distribution system operated by the Office of Hydrologic Development of the National Weather Service. More information on HADS gauge data can be found at http://www.nws.noaa.gov/oh/hads/.
3.0 Stratification
3.1 Region
The models are run on a full U.S. domain. Verification is performed on the full National domain (covers the CONUS), as well as the West, Central, and Eastern regions (Figure 3.1).
Figure 3.1. Map of the U.S. showing the West, Central, and Eastern domains.
3.2 Threshold
Forecast values are bi-linearly interpolated to observation points and compared at particular thresholds. Forecast/observation pairs are then computed for each of the following thresholds: 0.01, 0.10, 0.25, 0.50, 0.75, 1.00, 1.50, and 2.00 inches.
4.0 Time Increment
4.1 Beginning Date
The Beginning Date will default to either the previous date chosen by the user or to the earliest date for which data are available. The dates are used to allow access to statistics for any user-defined period of time. All dates in the RTVS precipitation verification system refer to the model run times.
4.2 Ending Date
The Ending Date will default to either the previous date chosen by the user or to the latest date for which data are available. All dates in the RTVS precipitation verification system refer to the model run times.
4.3 Forecast Length & Accumulation Period
The Forecast Length is the period of the model forecast, which ranges from 3-48 hours in 3 hour increments. Data for verification are accumulated over this selected period. For example, when choosing a forecast length and accumulation period of 48 hours, precipitation is accumulated from the beginning of the model run throughout the entire 48-hour period. Observations are similarly accumulated for the specified period (see Figure 4.1).
Figure 4.1. Time periods for precipitation verification in the RTVS grid-to-point system.
4.4 Issuance Time
The issuance time is the time (UTC) that the model is initialized. For the DWFE experiment, only 0000Z forecasts are verified because this is the only time that the WRF models are run.
5.0 Output
5.1 Plot Type
5.1.1 Time Period Aggregation
Time period aggregation plots are automatically produced for the DWFE Station-Based Precipitation experiment. The threshold is plotted on the x-axis and the time-averaged skill score is plotted on the y-axis. Currently, two statistics are plotted on the y-axis: Equitable Threat Score (ETS) and Frequency Bias. More about these skill scores can be found in Section 5.2.
Figure 5.1. Example of the Time Period Aggregation plot.
5.2 Statistics
The forecast/observation pairs used to create the skill scores are summarized in Table 5.1. The rows in the table represent the forecasts, the columns in the table represent the observations, and the elements in the cells represent the counts of forecast/observation pairs.
Forecast Observation Total Yes No Yes YY YN YY+YN No NY NN NY+NN Total YY+NY YN+NN YY+YN+NY+NN Table 5.1 Contingency table for evaluation of dichotomous (Yes/No) forecasts. Elements in the cells are the counts of forecast/observation pairs (YY, YN, NY, NN).
5.2.1 Frequency Bias
Frequency Bias is the ratio of the number of Yes forecasts to the number of Yes observations. A bias score greater than one indicates over-forecasting, while a bias less than one indicates under-forecasting.
Bias = (YY + YN) / (YY + NY)5.2.2 Equitable Threat Score (ETS)
The Equitable Threat Score (ETS) measures the fraction of all events forecast and/or observed that were correctly diagnosed, accounting for the hits that would occur purely due to random chance. A score of one is considered perfect for this statistic.
ETS = (YY - C2) / [(YY + YN + NY) - C2]
where C2 = [(YY + YN)*(YY + NY)] / (YY + YN + NY + NN)
Back to top
