1.0 Product

The product refers to the advisories and algorithms being verified. There are five possible selections available to the user for operational ceiling and visibility, including Airman's Meteorological Advisories (AIRMETs), AIRMETS without Amendments, National Ceiling and Visibility (NCV) Ceiling, NCV Ceiling and Visibility, and NCV Visibility.

1.1 AIRMETs (Airman's Meteorological Advisories)

AIRMETs are advisories issued by the Aviation Weather Center (AWC) to alert the aviation community to weather conditions that may be hazardous. AIRMETs are amended as necessary due to changing weather conditions or the issuance or cancellation of a SIGMET. When AIRMETs are chosen, amendments are included in the statistical results. More information on ceiling and visibility AIRMETs can be found at the following RTVS document. RTVS began its evaluation of AIRMETs in 1999.

1.2 AIRMETs without Amendments

When AIRMETs without amendments are chosen, no adjustments are made to the valid time of the scheduled AIRMET. For this selection, only the scheduled AIRMETs are used in the verification process.

1.3 NCV Ceiling and Visibility (National Ceiling and Visibility)

NCV Ceiling and Visibility is a National Center for Atmospheric Research (NCAR) algorithm that forecasts the current ceiling, surface visibility and flight category information every fifteen minutes. Detailed information on the NCV Ceiling and Visibility products can be found at the NCV website. RTVS began verification of the NCV Ceiling and NCV Visibility in 2002, and began evaluation of the NCV Ceiling and Visibility product in 2003.

2.0 Observation

2.1 METARs (Meteorological Aviation Report)

METARs describe the weather conditions at stations throughout the U.S. every hour. These data are collected centrally by the U.S. National Weather Service (NWS) and distributed. More information on METARs can be found at the following website.

3.0 Stratification

3.1 Region

3.1.1 National

The national region surrounds the U.S., as shown in Figure 3.1. The coastal waters are included in the area computation, but are not included when producing the skill scores.

Figure 3.1 Map of U.S. showing the regions used for verification of the ceiling and visibility products. The National region is the entire domain represented by the solid black surrounding the U.S.

4.0 Time Increment

The Beginning and Ending Dates are used to allow access to statistics for any user-defined period of time (e.g. day, week, month, year). Users can change any portion of the date boxes. The months that are provided to the user through the interface are dependent upon the year that the user chooses.

4.1 Beginning Date

The Beginning Date will default to either the previous date chosen by the user or to the earliest date for which data are available.

4.2 Ending Date

The Ending Date will default to either the previous date chosen by the user or to the latest date for which data are available.

4.3 Time Window

The Time Window drop-down menu allows the user to specify a time window around the forecast valid time. Any forecast/observation pairs that are collected in that time window are used in the statistical analysis. A larger time window benefits regions in which observations are sparse because it allows for the collection of more data.

4.4 Valid Time

Valid Time is an option for advisories, such as AIRMETs (with or without amendments).

When valid time is chosen, users can choose to access statistics for: i) one particular valid time between 0000 and 2300 UTC where the forecast/observation pairs generated for the advisory are computed using METARs valid over the 2-h window that surrounds the valid time, or ii) for ALL hours, where the forecast/observation pairs are combined for a day (0000Z-2300Z).

4.5 Issue Time

Issue time is an option for AIRMETs (with or without amendments).

For issue time, users can choose: i) one scheduled AIRMET period (e.g. 0145), where the AIRMETs are verified using 6 h of METARs, or ii) for All hours, where the forecast/observation pairs are accumulated for all issue times for a day.

4.6 Run Time

Algorithms are applied to model output files to create forecasts. The time the model is run to produce the output files for the algorithm is called the run time. The time the algorithm creates the forecast is called the issue time. For algorithms, the sum of the run time and the forecast length determines the valid time. This is the time used to verify the algorithms using the 2-h time window that surrounds the valid time. The "ALL" option includes forecast/observation pairs combined for all run times and specified forecast lengths.

4.7 Forecast Length

The Forecast Length is the lead-time of the forecast. For example, NCV Ceiling has forecast length options of 1, 3, 6, 9, and 12 hours. The 1 hour forecast will be valid 1 hours after the issue time, the 3 hour forecast will be valid 3 hours after the issue time, and so on. The "ALL" option includes forecast/observation pairs combined for the specified run times and all forecast lengths.

4.8 Period

With the Period option, users can determine a time period for the x-axis for the time series output option. Choices include daily, weekly, monthly, or quarterly. When daily is chosen, forecast/observation pairs are summed over a day and one point is plotted per day. When weekly is chosen, the forecast/observation pairs are summed over a 7-day period. If data is missing within the 7-day period, only the available data within that period are used to compute the weekly statistics. When monthly is chosen, statistics are computed for any pairs collected during the span of the month. The quarterly period is computed by accumulating pairs over one quarter of the year (3 months). Yearly statistics are computed from data collected from 1 January - 31 December.

5.0 Output

When producing a plot, users make a statistical comparison by indicating the x-axis and y-axis information. Statistics that may be selected for the x- or y-axes are discussed further in Section 5.2.

5.1 Plot Type

5.1.1 Scatter Plot

An example of a scatterplot is shown in Figure 5.1. Each dot on the scatterplot represents one specific forecast period (i.e. run time/forecast length for the chosen period). The number of forecast periods displayed on the plots is determined by the time period chosen by the user (in this case the time period chosen was from 1 May 2003 - 30 September 2004).

Figure 5.1 Scatterplot of % Area vs PODn for the NCV Ceiling product using METARs from 1 May 2003 – 30 September 2004. PODs range from 0 – 1.0.

5.1.2 Summary Table

The summary table allows users to view the tabular statistical results (described in Section 5.2) for the selected date range and period.

5.1.3 Time Series

An example of a time series plot is shown in Figure 5.2. The time period for display on the x-axis can be daily, weekly, monthly, quarterly or yearly. Choosing the weekly period when verifying METARs ensures that enough forecast/observation pairs are used in the computation.

Caution: When comparing a forecast versus an algorithm, please note that the verification dates for the forecast and algorithm may be different, depending on the run times and forecast length that are selected by the user. I.e., if comparing AIRMETs with a selected valid time of 00Z to the NCV Ceiling algorithm, with a run time of 18Z and forecast length of 6 hours, the verification dates will not correspond because the valid time for the algorithm is equal to the run time plus the forecast length (18Z + 6 h), which makes the valid time 00Z on the next day.

Figure 5.2 Time series plot of daily PODn for the ceiling and visibility AIRMETs using METARs from 1 - 30 September 2004. PODs range from 0 – 1.0.

5.1.4 Distribution Tables

Joint forecast/observation distribution tables are an output option for ceiling and visibility algorithms. Tables are available for ceiling and visibility individually as well as ceiling and visibility combined. These tables provide the user with a more detailed view of algorithm performance as it relates to flight rules.

Flight rule categories include:

• Low Instrument Flight Rule (LIFR) - Ceiling less than 500 ft. above ground level and/or visibility less than 1 mile
• Instrument Flight Rule (IFR) - Ceiling 500 to (but not equalling) 1000 ft. above ground level and/or visibility 1 to less than 3 miles
• Marginal Visual Flight Rule (MVFR) - Ceiling 1000-3000 ft. above ground level and/or visibility 3-5 miles
• Visual Flight Rule (VFR) - Celings greater than 3000 ft. above ground level and visibility greater than 5 miles

Four tables are automatically provided for this output type:

• Distribution of Forecasts and Observations (shown in Table 5.1)
• Conditional Probability of Observation Given Forecast
• Conditional Probability of Forecast Given Observations
• Contingency Table (using IFR conditions as threshold)

Table 5.1. Joint forecast/observation distribution table for the NCV Ceiling and Visibility algorithm from 1 November 2003 - 10 February 2005. Other probability and contingency tables are based on data in this table.

5.2 Statistic

The forecast/observation pairs used to create the skill scores are summarized in Table 5.1. The rows in the table represent the forecasts, the columns in the table represent the observations, and the elements in the cells represent the counts of forecast/observation pairs. Note that the counts in the verification table are observation-based (i.e., the sum of the counts is the total number of Yes and No METARs that were included in the analysis) and not all forecasts may be verified if there are no observations to verify them against.

Table 5.2 Contingency table for evaluation of dichotomous (Yes/No) forecasts. Elements in the cells are the counts of forecast/observation pairs.

Forecast	Observation		Total
	Yes	No
Yes	YY	YN	YY+YN
No	NY	NN	NY+NN
Total	YY+NY	YN+NN	YY+YN+NY+NN

See the Verification Techniques section for additional details.

The PODy, PODn and TSS statistics are available for the ceiling and visibility products.

5.2.1 PODn

The PODn is defined as the probability of detecting a NO event. It is the proportion of NO events that were correctly forecast.

5.2.2 PODy

The PODy is defined as the probability of detecting a YES event. It is the proportion of YES events that were correctly forecast.

5.2.3 TSS

The True Skill Statistic (Doswell et al 1990) is a measure of the ability of the forecasts to discriminate between "Yes" and "No" observations. It is also known as the Hanssen-Kuipers discrimination statistic (Wilks 1995).

6.0 References

Brown, Barbara G., Jennifer L. Mahoney, Tressa L. Fowler, and Judy Henderson, 2001: Approaches for Verification of Ceiling and Visibility Diagnoses and Forecasts. Submitted to FAA Aviation Weather Research Program (available from B. Brown, Research Applications Program, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307-3000).

Doswell, C.A., R. Davies-Jones, and David L. Keller, 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. and Forec., 5, 576-585.

National Weather Service, 1991: National Weather Service Operations Manual, D-22. National Weather Service. (Available at this web site: http://www.nws.noaa.gov)

Wilks, D.S., 1995: Statistical Methods in the Atmospheric Science. Academic Press, 467 pp.