Parallel Programming

There are several obstacles to the operational use of current distributed memory HPC systems: portability, programmability, and performance.   Historically NWP models have been large codes that did not run efficiently on HPC systems without some restructuring.   Typically a message passing library is used to handle inter-process communication, process synchronization, global operations, data decomposition, and I/O required to run in distributed memory environments.   The development of an efficient Message Passing Interface (MPI) library, supported by most vendors has improved the portability of models on distributed memory computers.   However, MPI is sufficiently low level in nature that it can be difficult to use.  To speed code parallelization, the ACS has developed a high level tool called the Scalable Modeling System (SMS) that simplifies the task required to port and run NWS models on HPC's while offering good scalable performance.

The Scalable Modeling System

The SMS is a directive-based parallelization tool that translates Fortran code into a parallel version that runs efficiently on both shared and distributed memory systems.  This software has been used sucessfully since 1993 to parallelize and run many oceanic and atmospheric models.

SMS was significantly updated in 2008 to support icosahedral grids used in the FIM and NIM models.  During this upgrade significant portions of legacy code were removed and our list of supported codes was reduced.  Codes listed in italics are legacy codes that may not work under SMS version 3.0.0 or higher.  Historically, we have used SMS on most HPC systems, however, the upgraded version has only been tested on linux clusters (intel).  We anticipate porting SMS version 3 to IBM SP systems at NCEP in late 2009 or early 2010 and to other systems as the need arises.

Models Parallelized using SMS

Atmospheric Models
  • Flow-following Finite volume Icosahedral Model (FIM)
    • Earth System Research Laboratory (ESRL)
  • Non-hydrostatic Icosahedral Model (NIM)
    • Earth System Research Laboratory (ESRL)
  • Rapid Update Cycle Model (RUC)
    • Earth System Research Laboratory (ESRL)
  • ETA
    • National Centers for Environmental Prediction (NCEP)
  • Global Forecast System (GFS)
    • Central Weather Bureau (CWB)    - Taiwan
  • Typhoon Forecast System (TFS)
    • Central Weather Bureau (CWB)    - Taiwan
  • Non-Hydrostatic Forecast System (NFS)
    • Central Weather Bureau (CWB)    - Taiwan
  • Atmospheric Chemistry Model (NALROM)
    • NOAA's Aeronomy Laboratory (NOAA / AL)
Oceanic Models
  • Princeton Ocean Model (POM)
    • NASA Goddard

  • Regional Ocean Modeling System (ROMS)
    • Rutgers University
    • National Institute of Water and Atmospheric Research (NIWA)
    • University of Alaska at Fairbanks (UAF)
    • NOAA / Pacific Marine Environmental Laboratory (NOAA/PMEL)
  • Hybrid Coordinate Ocean Model (HYCOM)
    • Los Alamos National Laboratory

These models contain structured regular grids that are resolved using a finite difference approximation.  As SMS has matured, the time and effort required to parallelize codes for MPPs has been reduced significantly.  Code parallelization has become simpler because SMS provides support for advanced operations including incremental parallelization and parallel debugging.

High Performance: SMS provide a number of performance optimizations.  The SMS run-time libraries have been optimized to speed inter-processor communications using techniques such as aggregation.  Array aggregation permits multiple model variables to be combined into a single communications call to reduce message-passing latency.  SMS also allows the user to perform computations in the halo region to reduce communications.  High performance I/O is also provided by SMS.  Since  models typically output forecasts several times during a model run, SMS can output these data asynchronous to model execution.   These optimization can lead to significantly faster execution times.

Documentation

Publications
 
The Scalable Modeling System: Directive-Based Code Parallelization for Distributed and Shared Memory Computers
             -   August 2003:  Journal of Parallel Computing volume/issue: 29/8 pp995-1020.
The Parallel Pre-Processor: a Compiler for Shared and Distributed Memory Computers
            -   December 2002:  Accepted for the 8th International Workshop on High-Level Parallel Programming Models and Supportive Environments
Performance Analysis of the Scalable Modeling System
             -   November 2002: Tenth ECMWF Workshop on the use of Parallel Processors in Meteorology.
SMS: A high level alternative to MPI
             -   November 2000: Ninth ECMWF Workshop on the use of Parallel Processors in Meteorology.

Software Downloads

SMS software is freely available but there are some restrictions.  To install, gunzip the downloaded file, un-tar it, cd to the unloaded SMS directory and then follow the instructions in file INSTALL.

Once you begin using SMS to parallelize code, the following information may be useful. Any additional questions should be directed to sms-info.fsl@noaa.gov

Prepared by Mark Govett, Mark.W.Govett@noaa.gov
Date of last update:    September-2009