5.2. CMAQv5.3.3 Advanced Tutorial (optional)

• Learn how to upgrade the ParallelCluster, by first creating a cluster that uses c5n.4xlarge as the compute nodes, and then upgrading the cluster to use c5n.18xlarge as the compute nodes.

• Learn how to install CMAQ software and underlying libraries, copy input data, and run CMAQ.

Notice

Skip this tutorial if you successfully completed the Intermediate Tutorial and wish to proceed to the post-processing and QA instructions. Note, you may wish to build the underlying libraries and CMAQ and code if you wish to create a ParallelCluster using a different family of compute nodes, such as the c6gn.16xlarge compute nodes AMD Graviton.

• 5.2.1. Use ParallelCluster without Software and Data pre-installed
  • Create CMAQ Cluster using SPOT pricing
    • Use an existing yaml file from the git repo to create a ParallelCluster
    • Use a configuration file from the github repo that was cloned to your local machine
    • Edit the c5n-4xlarge.yaml
    • Replace the key pair and subnet ID in the c5n-4xlarge.yaml file with the values created when you configured the demo cluster
    • The Yaml file for the c5n-4xlarge contains the settings as shown in the following diagram.
• 5.2.2. Create the c5n-4xlarge pcluster
  • Check on status of cluster
  • Start the compute nodes
  • Login to cluster
  • Show compute nodes
• 5.2.3. Update the compute nodes
  • Exit the cluster
  • Stop the compute nodes
  • Verify that the compute nodes are stopped
  • Examine the differences between YAML files
  • Edit the YAML file for c5n.n18xlarge
• 5.2.4. Create the c5n.18xlarge cluster
  • Verify that the compute nodes have been updated
  • Wait 5 to 10 minutes for the update to be completed
  • Re-start the compute nodes
  • Verify status of cluster
• 5.2.5. Login to c5n.18xlarge cluster
  • Check to make sure elastic network adapter (ENA) is enabled
  • Check what modules are available on the cluster
  • Load the openmpi module
  • Load the Libfabric module
  • Verify the gcc compiler version is greater than 8.0
• 5.2.6. Install Input Data on ParallelCluster
  • Verify AWS CLI is available obtain data from AWS S3 Bucket
  • Verify you can run the aws command
  • Copy Input Data from S3 Bucket to lustre filesystem
  • Use the S3 script to copy the CONUS input data from the CMAS s3 bucket
  • For ParallelCluster: Import the Input data from a public S3 Bucket
• 5.2.7. Install CMAQ sofware and libraries on ParallelCluster
  • Login to updated cluster
  • Change shell to use .tcsh
  • Check to see the tcsh shell is default
  • Use a configuration file from the github repo that was cloned to your local machine
  • Check to make sure elastic network adapter (ENA) is enabled
  • Check what modules are available on the cluster
  • Load the openmpi module
  • Load the Libfabric module
  • Verify the gcc compiler version is greater than 8.0
  • Change directories to install and build the libraries and CMAQ
  • Build netcdf C and netcdf F libraries - these scripts work for the gcc 8+ compiler
  • A .cshrc script with LD_LIBRARY_PATH was copied to your home directory, enter the shell again and check environment variables that were set using
  • If the .cshrc was not created use the following command to create it
  • Execute the shell to activate it
  • Verify that you see the following setting
  • Build I/O API library
  • Build CMAQ
• 5.2.8. Run CMAQ
  • Verify that you have an updated set of run scripts from the pcluster-cmaq repo
  • Verify that the input data is imported to /fsx from the S3 Bucket
  • Create the output directory
  • Run the CONUS Domain on 180 pes
  • Check the status in the queue
  • check on the status of the cluster using CloudWatch
  • check the timings while the job is still running using the following command
  • When the job has completed, use tail to view the timing from the log file.
  • Submit a request for a 288 pe job ( 8 x 36 pe) or 8 nodes instead of 5 nodes
  • Check on the status in the queue
  • Check the status of the run
  • Check whether the scheduler thinks there are cpus or vcpus
  • edit run script to use
  • Edit the yaml file to use DisableSimultaneousMultithreading: true
  • Confirm that there are only 36 cpus available to the slurm scheduler
  • Re-run the CMAQ CONUS Case
  • Submit a request for a 288 pe job ( 8 x 36 pe) or 8 nodes instead of 10 nodes with full output
  • Check the status of the run