5. Developer Guide to install and run CMAQv5.33 on Single VM or Parallel Cluster

• 5.1. CMAQv5.3.3 on Single Virtual Machine Advanced (optional)
  • 5.1.1. Install Software and run CMAQv5.3.3 on c6a.2xlarge for the 2016_12US3 Benchmark
    • Build CMAQv5.3.3 on c6a.2xlarge EC2 instance
      • Create a c6a.xlarge Virtual Machine
      • Login to the Virtual Machine
      • Make the /shared directory
      • Change the group and ownership of the shared directory
      • Create subdirectories on /shared
      • Check operating system version
      • Install Environment Modules
      • Logout and then log back in to activate modules command
      • Verify module command works
      • Set up build environment
      • Install Compilers and OpenMPI
      • Change shell to use tcsh
      • Logout and log back in, then check the shell
      • Check available versions of compiler
      • Choose gcc-9 and gfortran-9 as default compilers
      • Check version of gcc
      • Check version of gfortran
      • Check version of OpenMPI
      • Install Parallel Cluster CMAQ Repo
      • Install and build netcdf C, netcdf Fortran, I/O API, and CMAQ
      • Install netcdf-C and netcdf-Fortran
      • Copy a file to set paths
      • Exit cluster and log back in to activate the update shell, or use csh
      • Create Custom Environment Module for Libraries
      • Find path for openmpi libraries
      • Find path for include files for openmpi
      • Edit the config_cmaq_singlevm.csh script to specify the paths for OpenMPI
      • Install Python
      • Install jupyter notebook.
      • Install and Build CMAQ
        • Add compile option to makefile to get beyond a type mismatch error (note, this is only needed if you were using the gcc-11 compiler.
      • Run make again
      • Check to see what scripts are available
    • Download the Input data from the S3 Bucket
      • Install aws command line
      • Install unzip and unzip file
      • Edit .cshrc
      • Install the input data using the s3 script
    • Run CMAQ interactively using the following command:
      • First check to see how many cpus you have available on the machine.
  • 5.1.2. Install I/O API libraries that support HDF5
  • 5.1.3. Upgrade to run CMAQ on larger EC2 Instance
    • Save the AMI and create a new VM using a larger c6a.8xlarge (with 32 processors)
      • Use the AWS Console to Stop the Image
      • Use the AWS Console to Create a new AMI
      • Use the newly created AMI to launch a new Single VM using a larger EC2 instance.
      • Load the modules
      • Test running the listos domain on 32 processors
    • Run CMAQv5.4 for the full 12US1 Domain on c6a.48xlarge
      • Run utility to uncompress hdf5 *.nc4 files and save as classic *.nc files
      • Increased disk space on /shared to 500 GB
• 5.2. CMAQv5.3.3 Advanced Tutorial (optional)
  • 5.2.1. Use ParallelCluster without Software and Data pre-installed
    • Create CMAQ Cluster using SPOT pricing
      • Use an existing yaml file from the git repo to create a ParallelCluster
      • Use a configuration file from the github repo that was cloned to your local machine
      • Edit the c5n-4xlarge.yaml
      • Replace the key pair and subnet ID in the c5n-4xlarge.yaml file with the values created when you configured the demo cluster
      • The Yaml file for the c5n-4xlarge contains the settings as shown in the following diagram.
  • 5.2.2. Create the c5n-4xlarge pcluster
    • Check on status of cluster
    • Start the compute nodes
    • Login to cluster
    • Show compute nodes
  • 5.2.3. Update the compute nodes
    • Exit the cluster
    • Stop the compute nodes
    • Verify that the compute nodes are stopped
    • Examine the differences between YAML files
    • Edit the YAML file for c5n.n18xlarge
  • 5.2.4. Create the c5n.18xlarge cluster
    • Verify that the compute nodes have been updated
    • Wait 5 to 10 minutes for the update to be completed
    • Re-start the compute nodes
    • Verify status of cluster
  • 5.2.5. Login to c5n.18xlarge cluster
    • Check to make sure elastic network adapter (ENA) is enabled
    • Check what modules are available on the cluster
    • Load the openmpi module
    • Load the Libfabric module
    • Verify the gcc compiler version is greater than 8.0
  • 5.2.6. Install Input Data on ParallelCluster
    • Verify AWS CLI is available obtain data from AWS S3 Bucket
    • Verify you can run the aws command
    • Copy Input Data from S3 Bucket to lustre filesystem
    • Use the S3 script to copy the CONUS input data from the CMAS s3 bucket
    • For ParallelCluster: Import the Input data from a public S3 Bucket
  • 5.2.7. Install CMAQ sofware and libraries on ParallelCluster
    • Login to updated cluster
    • Change shell to use .tcsh
    • Check to see the tcsh shell is default
    • Use a configuration file from the github repo that was cloned to your local machine
    • Check to make sure elastic network adapter (ENA) is enabled
    • Check what modules are available on the cluster
    • Load the openmpi module
    • If you don’t see the openmpi module
      • Install OpenMPI
    • Load the Libfabric module
    • Verify the gcc compiler version is greater than 8.0
    • Change directories to install and build the libraries and CMAQ
    • Build netcdf C and netcdf F libraries - these scripts work for the gcc 8+ compiler
    • A .cshrc script with LD_LIBRARY_PATH was copied to your home directory, enter the shell again and check environment variables that were set using
    • If the .cshrc was not created use the following command to create it
    • Execute the shell to activate it
    • Verify that you see the following setting
    • Build I/O API library
    • Build CMAQ
  • 5.2.8. Run CMAQ
    • Verify that you have an updated set of run scripts from the pcluster-cmaq repo
    • Verify that the input data is imported to /fsx from the S3 Bucket
    • Create the output directory
    • Run the CONUS Domain on 180 pes
    • Check the status in the queue
    • check on the status of the cluster using CloudWatch
    • check the timings while the job is still running using the following command
    • When the job has completed, use tail to view the timing from the log file.
    • Submit a request for a 288 pe job ( 8 x 36 pe) or 8 nodes instead of 5 nodes
    • Check on the status in the queue
    • Check the status of the run
    • Check whether the scheduler thinks there are cpus or vcpus
    • edit run script to use
    • Edit the yaml file to use DisableSimultaneousMultithreading: true
    • Confirm that there are only 36 cpus available to the slurm scheduler
    • Re-run the CMAQ CONUS Case
    • Submit a request for a 288 pe job ( 8 x 36 pe) or 8 nodes instead of 10 nodes with full output
    • Check the status of the run