Experimental Ceiling and Visibility On-line User Manual

Introduction

Numerous ceiling and visibility forecasts are verified within the RTVS. The forecasts can be broadly classified into two types. The first, Airmen's Meteorological Advisories (AIRMETs), are human-generated and take the form of polygons drawn over the continental U.S. and adjacent coastal waters that are issued on an as-needed basis when hazardous ceiling and visibility conditions are present. The other broad class of forecasts that RTVS currently handles are automated, gridded algorithms that are either output or derived from numerical models such as the RUC and Eta.

The differing types of forecasts require differing approaches to verification and it is important to understand these differences. The AIRMET is a scheduled product that is produced four times per day. Within this product may be any number of polygons that describe hazardous conditions to aviation. We will continue to focus solely on the ceiling and visibility AIRMET in this document. Other potential information from AIRMETs exists for icing and turbulence. In addition to the polygons being produced every six hours, the AIRMETs may be amended to account for necessary changes in the forecasts, reducing their valid times to a period less than six hours. The verifying data used for all ceiling and visibility forecasts in RTVS are hourly Meteorological Aviation Report (METAR) observations. In order to utilize the maximum amount of observational data, the AIRMETs are evaluated in hourly intervals to utilize the METAR data. For instance, if an AIRMET was valid from 0200 UTC to 0800 UTC, the polygon would be evaluated at 0200, 0300, 0400, ..., 0800 UTC. In order to increase the number of observations to evaluate the AIRMET polygons, a temporal window is applied around each intermediate "valid hour". For AIRMETs this window is currently set at 60 minutes either side of this "valid time".

The actual verification is done in the following way: each METAR observation is tested to determine if it falls within the boundaries for each AIRMET polygon that is in effect at a given valid time. This station-based approach has the limitation that AIRMETs without any observations falling within it will not be included in the results. This is an extremely rare situation. Once a forecast and observation pair has been created the next step is determine if the forecast is correct or incorrect. AIRMETs are issued when so-called 'IFR conditions' are expected. No numerical values are specified that provide the expected ceiling and visibility values; it is simply a categorical statement that IFR conditions will occur within this polygon. IFR is an acronym for Instrument Flight Rules and are especially important to aviation because certain restrictions occur when pilots must switch from VFR, or Visual Flight Rules, to IFR. IFR conditions represent ceilings less than or equal to 1000' and horizontal visibilities are less than 3 miles. More information flight rules are discussed below in section X.

Once the determination is made for the station as to whether or not IFR conditions are present a count may be increased into one of the following categories: IFR forecast and IFR observed (or yes forecast, yes observed; hereafter YY), IFR forecast and no IFR observed (yes forecast, no observed; hereafter YN), IFR not forecast and IFR observed (NY) and IFR not forecast and IFR not observed (NN). This is repeated for all METAR stations. This data is then used to produce a variety of summary statistics to assess the forecast performance.

The model algorithms are verified using a slightly different approach. The model data come in gridded form to the RTVS and must again be matched with observations at certain valid times. The current method uses bi-linear interpolation to take the gridded data to the station locations. A 30 minute temporal interval is chosen around each valid time to determine a 'worst case' observation at each METAR location. This observation is then compared to the forecast values in an identical manner as described above for AIRMETs and counts of the various summary categories are created for analysis.

The final forecast type currently implemented in RTVS is persistence. Persistence simply means that the weather observed at a given time will continue to be observed at some time in the future. Persistence forecasts are created at all METAR locations used to verify the AIRMET and model algorithm forecasts.

The following sections describe the various choices and values that the user may alter when using the Ceiling and Visibility statistical tool.

1.0 Product

The product refers to the forecast being verified. The current selections available to the user include Airmen's Meteorological Advisories (AIRMETs), AIRMETS without Amendments, National Ceiling and Visibility Algorithm (NCV) Ceiling, NCV Ceiling and Visibility, NCV Visibility, Eta Visibility, RUC Visibility, RUC Cloud Base Height (a close proxy to ceiling), Persistence Ceiling, Persistence Visibility, and the combined Persistence Ceiling and Visibility.

A brief description of each product follows.

1.1 AIRMETs

AIRMETs (Airmen's Meteorological Information) are advisories issued by the Aviation Weather Center (AWC) to alert the aviation community to weather conditions that may be hazardous. AIRMETs are amended as necessary due to changing weather conditions or the issuance or cancellation of a SIGMET. When AIRMETs are chosen, amendments are included in the statistical results. More information on ceiling and visibility AIRMETs can be found at the following RTVS document. RTVS began its evaluation of AIRMETs in 1999.

1.2 AIRMETs without Amendments

When AIRMETs without amendments are chosen, no adjustments are made to the valid time of the scheduled AIRMET. For this selection, only the scheduled AIRMETs are used in the verification process. RTVS began its evaluation of AIRMETs without Amendments in 1999.

1.3 Eta Visibility

The Eta Visibility forecast is produced by NCEP and is based on the Eta model. More information can be found at the following website. RTVS has been evaluating Eta Visibility since 2003.

1.4 NCV Ceiling and Visibility

The NCV Ceiling and Visibility product, developed by the FAA's ceiling and visibility Product Development Team, produces forecasts for the current ceiling, surface visibility and flight category information every hour. Detailed information on the NCV Ceiling and Visibility products can be found at the NCV website. RTVS began verification of the NCV Ceiling and NCV Visibility in 2002, and began evaluation of the NCV Ceiling and Visibility product in 2003.

1.5 RUC Visibility

The RUC visibility forecast is obtained from the operational RUC model being run at NCEP. More information can be found at the following website. RUC Visibility forecasts have been evaluated through RTVS since 2003.

2.0 Observation

2.1 METARs (Meteorological Aviation Report)

METARs describe the weather conditions at stations, primarily colocated with airports, throughout the U.S. every hour. These data are collected centrally by the U.S. National Weather Service (NWS) and distributed. More information on METARs can be found at the following website.

3.0 Stratification

3.1 Region

3.1.1 National

The National region includes the continental U.S. and adjacent coastal waters. The coastal waters are included in the area computation, but are not included when producing other statistics which require METAR observations.

Figure 3.1 Map of U.S. showing the regions used for verification of the ceiling and visibility products. The National region is the entire domain represented by the heavy black line surrounding the U.S.

4.0 Time Increment

The Beginning and Ending Dates are used to allow access to statistics for any user-defined period of time (e.g. day, week, month, year). Users can change any portion of the date boxes. The months that are provided to the user through the interface are dependent upon the year that the user chooses.

4.1 Beginning Date

The Beginning Date will default to either the previous date chosen by the user or to the earliest date for which data are available.

4.2 Ending Date

The Ending Date will default to either the previous date chosen by the user or to the latest date for which data are available.

4.3 Valid Time

When valid time is chosen, users can choose to access statistics for: i) one particular valid time between 0000 and 2300 UTC where the forecast/observation pairs generated for the AIRMETs (with or without amendments) are computed using METARs valid over the 2-h window that surrounds the valid time, or ii) for all hours, where the forecast/observation pairs (with or without amendments in the case of AIRMETs) are aggregated over the time period chosen.

4.4 Issue Time

Options for issue time (or algorithm run time) include: i) one scheduled AIRMET period (e.g. 0145/0245), where the AIRMETs without amendments are verified or ii) for all hours, where the forecast/observation pairs generated for the AIRMETs without amendments are combined for all hours over the time period chosen. The issue time option is only available for AIRMETs without amendments.

4.5 Time Window

The Time Window drop-down menu allows the user to specify a time window around the forecast valid time for which the forecast/observations are collected and used to compute the statistical scores. Currently this menu is used solely for information purposes and cannot be manipulated.

4.6 Forecast Length

The Forecast Length (or forecast hour) refers to the lead-time of the forecast. For example, NCV Ceiling is verified at forecast lengths 1, 3, 6, 9, and 12 hours. The 1 hour forecast will be valid 1 hour after the issue time, the 3 hour forecast will be valid 3 hours after the issue time, and so on. This field will change depending upon the availability of the requested product for each data source.

4.7 Period

Users can determine a time period that is used to aggregate the forecast/observation pairs. Choices include daily, weekly, monthly, quarterly, or yearly. When daily is chosen, the statistics displayed on the time series plot are generated from an accumulation of the forecast/observation pairs for that day. When weekly is chosen, the forecast/observation pairs used to generate the statistics are summed over 7-day periods starting from the chosen starting date. If data is missing within the 7-day period, only the available data within that period are used to compute the weekly statistics. When monthly is chosen, statistics are computed for any pairs collected during the span of each calendar month within the chosen time period. The quarterly period is computed by accumulating pairs over one quarter of the year (3 months). Yearly statistics are computed from data from 1 January to 31 December.

5.0 Output

5.0.1 Axis information

When producing a plot, users make a statistical comparison by indicating the X-axis and Y-axis information. Statistics that may be selected for the X- or Y-axes are discussed further in Section 5.2.

5.1 Plot Type

5.1.1 Scatter Plot

An example of a scatter plot is shown in Figure 5.1. Each dot on the scatter plot represents one specific forecast period (i.e. issue/lead time for the chosen period). The number of forecast periods displayed on the plots is determined by the time period chosen by the user (in this case the time period chosen was from 1 May 2003 - 30 September 2004).

Figure 5.1 Scatter plot of % Area vs PODn for the NCV Ceiling product using METARs from 1 May 2003 to September 2004. PODs range from 0.0 to 1.0.

5.1.2 Time Series

An example of a time series plot is shown in Figure 5.2. The time period for display on the X-axis can be daily, weekly, monthly, quarterly or yearly. Choosing the weekly period when verifying AIRMETs ensures that enough forecast/observation pairs are used in the computation.

Figure 5.2 Time series plot of daily PODn for the ceiling and visibility AIRMETs using METARs from 1 - 30 September 2004. PODs range from 0 – 1.0.

5.1.3 Summary Table

The summary table allows users to view the tabular statistical results for a number of different summary statistics (described in Section 5.2) for the selected date range and period.

5.1.4 Distribution Tables

For all datasets except for AIRMETs an informative set of tables can be derived that provide more information than the aforementioned 'was it forecast and was it observed' information. In these tables the information is further broken down into the complete set of aviation flight rules. Previously, IFR conditions have been mentioned as these represent the most important conditions to aviation. The complete set of conditions are (from least restrictive to most):

Flight rule categories include:

• Low Instrument Flight Rule (LIFR) - Ceiling less than 500 ft. above ground level and/or visibility less than 1 mile
• Instrument Flight Rule (IFR) - Ceiling 500 to (but not equalling) 1000 ft. above ground level and/or visibility 1 to less than 3 miles
• Marginal Visual Flight Rule (MVFR) - Ceiling 1000-3000 ft. above ground level and/or visibility 3-5 miles
• Visual Flight Rule (VFR) - Celings greater than 3000 ft. above ground level and visibility greater than 5 miles

This configuration allows the forecast to be placed into one of the four categories listed above (VFR, MVFR, IFR, LIFR). Each observation can be categorized in a similar manner. Once the categorization has taken place a series of tables can be created that allow the user to view not only the raw counts of how often each element in the 16-cell table occur but two important factorizations of that information. The first factorization is concerned with what is the probability of a given observation occurring when a certain forecast is issued (e.g., how frequently do I get an MVFR observation when I forecast LIFR conditions). The other factorization is complementary and is concerned with how often is a certain forecast category issued given that an observation in that category is found. It is important to understand the difference between these two factorizations.

Four tables are automatically provided for this output type:

• Distribution of Forecasts and Observations (shown in Table 5.1)
• Joint distribution of forecasts and observations
• Conditional Probability of Observation Given Forecast
• Conditional Probability of Forecast Given Observations
• Contingency Table (using IFR conditions as threshold)

Table 5.1. Joint forecast/observation distribution table for the NCV Ceiling and Visibility algorithm from 1 November 2003 - 10 February 2005. Other probability and contingency tables are based on data in this table.

5.2 Statistic

The forecast/observation pairs used to create the summary statistics are summarized in Table 5.1. The rows in the table represent the forecasts, the columns in the table represent the observations, and the elements in the cells represent the counts of forecast/observation pairs. Note that the counts in the verification table are observation-based (i.e., the sum of the counts is the total number of Yes and No METARs that were included in the analysis) and not all forecasts may be verified if there are no observations to verify them against.

Table 5.1 Contingency table for evaluation of dichotomous (Yes/No) forecasts. Elements in the cells are the counts of forecast-observation pairs.

Forecast	Observation		Total
	Yes	No
Yes	YY	YN	YY+YN
No	NY	NN	NY+NN
Total	YY+NY	YN+NN	YY+YN+NY+NN

The available statistics include:

5.2.1 PODn

The PODn is defined as the probability of detecting a non-event (in this case non-IFR conditions). It is the proportion of "no" events that were correctly forecast.

5.2.2 PODy

The PODy is defined as the probability of detecting a "yes" (or IFR) event. It is the proportion of "yes" events that were correctly forecast.

5.2.3 TSS

The True Skill Statistic (Doswell et al. 1990) is a measure of the ability of the forecasts to discriminate between "Yes" and "No" observations. It is also known as the Hanssen-Kuipers discrimination statistic (Wilks 1995).

5.2.4 CSI

The Critical Success Index, also known as the Threat Score, measures the fraction of observed or forecast events that were correctly forecast. Correct forecasts of non-events (the NN element of the contigency table) are not considered.

5.2.5 Bias

Bias is a simple measure that says, on average, is the forecast over- or underforecasting the event in question. A Bias of 1 means the forecast shows no bias. Values greater than 1 indicate overforecasting while values less than 1 indicate underforecasting. Bias = (YY + YN) / (YY + NY)

5.2.6 HSS

The Heidke Skill Score is a skill score that uses the hit rate as its basic measure of accuracy. The maximum possible value is 1. Forecasts that perform worse than the reference forecasts will receive a score less than zero.

5.2.7 FAR

The False Alarm Rate is simply the fraction of forecasts for an event to occur that do not actually occur. Given this situation, smaller values of the score are preferred over larger ones.

6.0 References

Brown , Barbara G., Jennifer L. Mahoney, Tressa L. Fowler, and Judy Henderson, 2001: Approaches for Verification of Ceiling and Visibility Diagnoses and Forecasts. Submitted to FAA Aviation Weather Research Program (available from B. Brown, Research Applications Program, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307-3000).

Doswell, C.A., R. Davies-Jones, and David L. Keller, 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. and Forec., 5, 576-585.

National Weather Service, 1991: National Weather Service Operations Manual, D-22. National Weather Service. (Available at this web site: http://www.nws.noaa.gov)

Wilks, D.S., 1995: Statistical Methods in the Atmospheric Science. Academic Press, 467 pp.