4. Developer Guide to install and run CMAQv5.4 on Single VM or Parallel Cluster

• 4.1. Install CMAQv5.4+ on Single Virtual Machine Advanced (optional)
  • 4.1.1. Install Software and run CMAQv5.4 on c6a.xlarge for the 12km Listos Training Domain
    • Build CMAQv5.4+ on c6a.xlarge EC2 instance
      • Create a c6a.xlarge Virtual Machine
      • Login to the Virtual Machine
      • Make the /shared directory
      • Change the group and ownership of the shared directory
      • Create subdirectories on /shared
      • Check operating system version
      • Install Environment Modules
      • Logout and then log back in to activate modules command
      • Verify module command works
      • Set up build environment
      • Install Compilers and OpenMPI
      • Change shell to use tcsh
      • Logout and log back in, then check the shell
      • Check available versions of compiler
      • Choose gcc-9 and gfortran-9 as default compilers
      • Check version of gcc
      • Check version of gfortran
      • Check version of OpenMPI
      • Install Parallel Cluster CMAQ Repo
      • Install and build netcdf C, netcdf Fortran, I/O API, and CMAQ
      • Install netcdf-C and netcdf-Fortran
      • Copy a file to set paths
      • Exit cluster and log back in to activate the update shell, or use csh
      • Create Custom Environment Module for Libraries
      • Find path for openmpi libraries
      • Find path for include files for openmpi
      • Edit the config_cmaq_singlevm.csh script to specify the paths for OpenMPI
      • Install Python
      • Install jupyter notebook.
      • Install and Build CMAQ
        • Add compile option to makefile to get beyond a type mismatch error (note, this is only needed if you were using the gcc-11 compiler.
      • Run make again
      • Copy the run scripts from the repo to the run directory
    • Download the Input data from the S3 Bucket
      • Install aws command line
      • Install unzip and unzip file
      • Edit .cshrc
      • Install the input data using the s3 script
      • Link the input data directory to the default location
    • Run CMAQ interactively using the following command:
      • First check to see how many cpus you have available on the machine.
      • If you upgrade this VM from a c6.xlarge to a c6.8xlarge, then you could run CMAQ interactively on 16 pes using the following command:
  • 4.1.2. Install Software and run CMAQv5.4 on c7g-hpc7g for the 12km Listos Training Domain
    • Build CMAQv5.4+ on c7g.xlarge EC2 instance
      • Create a c7g.xlarge Virtual Machine
      • Login to the Virtual Machine
      • Make the /shared directory
      • Change the group and ownership of the shared directory
      • Create subdirectories on /shared
      • Check operating system version
      • Install Environment Modules
      • Logout and then log back in to activate modules command
      • Verify module command works
      • Set up build environment
      • Install Compilers and OpenMPI
      • Compiler versions
      • Change shell to use tcsh
      • Logout and log back in, then check the shell
      • Check available versions of compiler
      • Choose gcc-9 and gfortran-9 as default compilers
      • Check version of gcc
      • Check version of gfortran
      • Check version of OpenMPI
      • Install Parallel Cluster CMAQ Repo
      • Install and build netcdf C, netcdf Fortran, I/O API, and CMAQ
      • Install netcdf-C and netcdf-Fortran
      • Copy a file to set paths
      • Exit cluster and log back in to activate the update shell, or use csh
      • Create Custom Environment Module for Libraries
      • Find path for openmpi libraries
      • Find path for include files for openmpi
      • Edit the config_cmaq_singlevm.csh script to specify the paths for OpenMPI
      • Install Python
      • Install jupyter notebook.
      • Install and Build CMAQ
        • Add compile option to makefile to get beyond a type mismatch error (note, this is only needed if you were using the gcc-11 compiler.
      • Run make again
      • Copy the run scripts from the repo to the run directory
    • Download the Input data from the S3 Bucket
      • Install aws command line
      • Install unzip and unzip file
      • Edit .cshrc
      • Install the input data using the s3 script
      • Link the input data directory to the default location
    • Run CMAQ interactively using the following command:
      • First check to see how many cpus you have available on the machine.
      • If you upgrade this VM from a c6.xlarge to a c6.8xlarge, then you could run CMAQ interactively on 16 pes using the following command:
  • 4.1.3. Install I/O API libraries that support HDF5
  • 4.1.4. Upgrade to run CMAQ on larger EC2 Instance
    • Save the AMI and create a new VM using a larger c6a.8xlarge (with 32 processors)
      • Use the AWS Console to Stop the Image
      • Use the AWS Console to Create a new AMI
      • Use the newly created AMI to launch a new Single VM using a larger EC2 instance.
      • Load the modules
      • Test running the listos domain on 32 processors
    • Run CMAQv5.4 for the full 12US1 Domain on c6a.48xlarge
      • Run utility to uncompress hdf5 *.nc4 files and save as classic *.nc files
      • Increased disk space on /shared to 500 GB
• 4.2. Install CMAQv5.4 on ParallelCluster (optional)
  • 4.2.1. Configure Parallel Cluster
  • 4.2.2. Create the hpc7g.16xlarge pcluster
    • Check on status of cluster
    • Login to cluster
    • Check to make sure elastic network adapter (ENA) is enabled
    • Check what modules are available on the cluster
    • Load the openmpi module
    • Load the Libfabric module
    • Verify the gcc compiler version is greater than 8.0
  • 4.2.3. Install CMAQ sofware and libraries on ParallelCluster version 3.6
    • Login to updated cluster
    • Change shell to use .tcsh
    • Check to see the tcsh shell is default
    • Reload the environment modules
    • Check to make sure elastic network adapter (ENA) is enabled
    • Verify the gcc compiler version is greater than 8.0
    • Change directories to install and build the libraries and CMAQ
    • Build netcdf C and netcdf F libraries - these scripts work for the gcc 8+ compiler
    • A .cshrc script with LD_LIBRARY_PATH was copied to your home directory, enter the shell again and check environment variables that were set using
    • If the .cshrc was not created use the following command to create it
    • Execute the shell to activate it
    • Verify that you see the following setting
    • Build I/O API library
    • Build CMAQ
  • 4.2.4. Install netCDF libraries that use HDF5 and support nc4 compressed files
    • Create Custom Environment Module for Libraries
  • 4.2.5. Install gh following these instructions
  • 4.2.6. Use gh authentication
  • 4.2.7. Run CMAQ using hpc7g.16xlarge compute nodes
    • Verify that you have an updated set of run scripts from the pcluster-cmaq repo
    • Verify that the input data is imported to /fsx from the S3 Bucket
    • Preloading the files
    • Create a /fsx/data and /fsx/data/output directory
    • Link the data to what is being used in the run scriptso
    • Run the 12US1 Domain on 32 pes
    • Check the status in the queue
    • check on the status of the cluster using CloudWatch
    • check the timings while the job is still running using the following command
    • When the job has completed, use tail to view the timing from the log file.
    • Submit a request for a 64 pe job ( 2 x 32 pe) using 2 nodes
    • Check on the status in the queue
    • Check the status of the run
    • Check whether the scheduler thinks there are cpus or vcpus
    • When multiple jobs are submitted to the queue they will be dispatched to different compute nodes.
    • When the job has completed, use tail to view the timing from the log file.
    • Submit a job to run on 96 cores, 3x32 nodes
    • Verify that it is running on 3 nodes
    • Check the log for how quickly the job is running
    • Submit a job to run on 128 cores, 4x32 nodes
    • Verify that it is running on 4 nodes
    • Check the log for how quickly the job is running
  • 4.2.8. Run CMAQ using hpc7g.8xlarge compute nodes
    • Verify that you have an updated set of run scripts from the pcluster-cmaq repo
    • Run the 12US1 Domain on 32 pes on hpc7g.8xlarge
    • When the job has completed, use tail to view the timing from the log file.
    • Submit a request for a 64 pe job ( 2 x 32 pe) using 2 nodes on hpc7g.8xlarge
    • Check on the status in the queue
    • Check the status of the run
    • Check whether the scheduler thinks there are cpus or vcpus
    • When multiple jobs are submitted to the queue they will be dispatched to different compute nodes.
    • When the job has completed, use tail to view the timing from the log file.
    • Submit a job to run on 96 cores, 3x32 nodes on hpc7g.8xlarge
    • Verify that it is running on 3 nodes
    • Check the log for how quickly the job is running
  • 4.2.9. Install Input Data on ParallelCluster
    • Verify AWS CLI is available obtain data from AWS S3 Bucket
    • Verify you can run the aws command
    • Copy Input Data from S3 Bucket to lustre filesystem
    • Use the S3 script to copy the CONUS input data from the CMAS s3 bucket
    • For ParallelCluster: Import the Input data from a public S3 Bucket (optional)
    • Convert the *.nc4 compressed netCDF4 files to netCDF classic (nc3) files