Contents Menu Expand Light mode Dark mode Auto light/dark mode
aws-cmaq documentation
Logo
aws-cmaq documentation

Contents:

  • 1. System Requirements
    • 1.1. Configurations for running CMAQv5.3.3 on AWS ParallelCluster
    • 1.2. Software Requirements for CMAQ on AWS ParallelCluster
    • 1.3. Single VM Configuration for CMAQv5.3.2_Benchmark_2Day_Input.tar.gz Benchmark
    • 1.4. ParallelCluster Configuration for 12US2 Benchmark Domain
  • 2. Create Single VM and run CMAQv5.3.3 (software pre-installed)
    • 2.1. Use AWS Management Console to Create Single VM and run CMAQv5.3.3 (software pre-installed)
      • 2.1.1. Learn how to Use the AWS Management Console to launch EC2 instance using Public AMI
      • 2.1.2. Run CMAQv5.3.3 on c6a.2xlarge
      • 2.1.3. Load the environment modules
      • 2.1.4. Update the pcluster-cmaq repo using git
      • 2.1.5. Run CMAQv5.3.3 for 2016_12SE1 1 Day benchmark Case on 4 pe
      • 2.1.6. Save output data and run script logs
      • 2.1.7. Save the full input data, run scripts, output data and logs to an AMI that is owned by your account.
      • 2.1.8. Stop Instance
    • 2.2. CMAQv5.3.3 on Single Virtual Machine Intermediate (software pre-installed)
      • 2.2.1. Learn how to Use AWS CLI to launch c6a.2xlarge EC2 instance using Public AMI
      • 2.2.2. Learn how to Use AWS CLI to launch c6a.8xlarge EC2 instance using Public AMI
      • 2.2.3. Learn how to Use AWS CLI to launch c6a.48xlarge EC2 instance using Public AMI
      • 2.2.4. Documentation of Troubleshooting effort for CMAQv5.4+ on 12US1
      • 2.2.5. Use new ami instance with faster storage (io1) to create c6a.48xlarge ec2 instance
  • 3. Create a Parallel Cluster and run CMAQv5.3.3
    • 3.1.1. Introductory Tutorial
      • Step by Step Instructions to Build a Demo ParallelCluster.
      • Establish Identity and Permissions
      • Configure a Demo Cluster
      • Create a Demo Cluster
      • Login and Examine Cluster
      • Delete the Demo Cluster
    • 3.1.2. CMAQv5.3.3 Intermediate Tutorial
      • Use ParallelCluster pre-installed with software and data.
      • Create CMAQ ParallelCluster with software/data pre-installed
      • Log into the new cluster
      • Change shell to use tcsh
      • Verify Software
      • Verify Input Data
      • Examine CMAQ Run Scripts
      • Submit Job to Slurm Queue
      • Submit a minimum of 2 benchmark runs
    • 3.1.3. CMAQv5.3.3 Parallel Cluster Benchmark on HPC6a-48xlarge with EBS and Lustre (optional)
      • Use ParallelCluster pre-installed with CMAQv5.3.3 software and 12US2 Benchmark
      • Create CMAQ ParallelCluster with software/data pre-installed
      • Log into the new cluster
      • Resize the EBS Volume
      • Change shell to use tcsh
      • Verify Software
      • Verify Input Data
      • Examine CMAQ Run Scripts
      • To run on the EBS Volume a code modification is required.
      • Build the code by running the makefile
      • Submit Job to Slurm Queue to run CMAQ on Lustre
      • Submit a run script to run on the EBS volume
      • Modify YAML and then Update Parallel Cluster.
      • Submit a minimum of 2 benchmark runs
      • upgrade pcluster version to try Persistent 2 Lustre Filesystem
      • Query the stack formation log messages
  • 4. Performance and Cost Optimization
    • 4.1. Right-sizing Compute Nodes for the ParallelCluster Configuration
    • 4.2. An explanation of why a scaling analysis is required for Multinode or Parallel MPI Codes
    • 4.3. Slurm Compute Node Provisioning
    • 4.4. Spot versus On-Demand Pricing
    • 4.5. Benchmark Timings for CMAQv5.3.3 12US2 Benchmark
    • 4.6. Benchmark Scaling Plots for CMAQv5.3.3 12US2 Benchmark
    • 4.7. Cost Information
    • 4.8. Recommended Workflow for extending to annual run
    • 4.9. Side by Side Comparison of the information in the log files for 12x9 pe run compared to 9x12 pe run.
  • 5. Developer Guide to install and run CMAQv5.33 on Single VM or Parallel Cluster
    • 5.1. CMAQv5.3.3 on Single Virtual Machine Advanced (optional)
      • 5.1.1. Install Software and run CMAQv5.3.3 on c6a.2xlarge for the 2016_12US3 Benchmark
      • 5.1.2. Install I/O API libraries that support HDF5
      • 5.1.3. Upgrade to run CMAQ on larger EC2 Instance
    • 5.2. CMAQv5.3.3 Advanced Tutorial (optional)
      • 5.2.1. Use ParallelCluster without Software and Data pre-installed
      • 5.2.2. Create the c5n-4xlarge pcluster
      • 5.2.3. Update the compute nodes
      • 5.2.4. Create the c5n.18xlarge cluster
      • 5.2.5. Login to c5n.18xlarge cluster
      • 5.2.6. Install Input Data on ParallelCluster
      • 5.2.7. Install CMAQ sofware and libraries on ParallelCluster
      • 5.2.8. Run CMAQ
  • 6. Post-process and qa
    • 6.1. Post-process CMAQ and Install R
      • 6.1.1. Scripts to run combine and post processing
      • 6.1.2. Install R, Rscripts and Packages
    • 6.2. QA CMAQ
      • 6.2.1. Quality Assurance
      • 6.2.2. Run m3diff to compare the output data for two runs that have different values for NPCOL
      • 6.2.3. Use m3diff to compare two runs that have the same NPCOL
      • 6.2.4. Run an R script to create the box plots and spatial plots comparing the output of two runs
    • 6.3. Compare Timing of CMAQ Routines
      • 6.3.1. Parse timings from the log file
    • 6.4. Copy Output to S3 Bucket
      • 6.4.1. Copy Output Data and Run script logs to S3 Bucket
      • 6.4.2. Copy scripts and logs to /fsx
      • 6.4.3. Examine the output files
      • 6.4.4. Copy the output to an S3 Bucket
  • 7. Logout and Delete ParallelCluster
    • 7.1. Logout of cluster when you are done
    • 7.2. Delete Cluster
    • 7.3. Verify that the cluster was deleted
  • 8. Additional Resources
    • 8.1. FAQ
    • 8.2. Free Training
    • 8.3. Another workshop to learn the AWS CLI 3.0
    • 8.4. Youtube video
    • 8.5. Intro to AWS for HPC People - HPC Tech Shorts
    • 8.6. Benchmarking
    • 8.7. Help Resources for CMAQ
    • 8.8. Computing on the Cloud References
    • 8.9. Resources from AWS for diagnosing issues with running the Parallel Cluster
    • 8.10. Instructions on how to create Parallel Cluster Amazon Machine Image (AMI) from the command line
    • 8.11. ParallelCluster Update
    • 8.12. Use Elastic Fabric Adapter/Elastic Network Adapter for better performance
    • 8.13. VPC Management
    • 8.14. Using Cost Allocation Tags with ParallelCluster
  • 9. Future Work
    • 9.1. Future Work
  • 10. Contribute to this Tutorial
    • 10.1. Contribute to Pcluster-cmaq Documentation
Back to top
Edit this page

8.2. Free Training#

Numerical Weather Prediction HPC Workship

AWS Free Training Courses

8.3. Another workshop to learn the AWS CLI 3.0#

Workshop on learning AWS CLI 3.0

8.4. Youtube video#

Youtube video on AWS CLI 3.0

8.5. Intro to AWS for HPC People - HPC Tech Shorts#

Intro to AWS for HPC People - Tech Short Foundations Level 1

8.6. Benchmarking#

Benchmarks optimized for HPC high memory

AWS Graviton WRF Performance Comparison

Deep Dive hpc7g

AWS EPYC and WRF Performance Comparison

Next
8.7. Help Resources for CMAQ
Previous
8.1. FAQ
Copyright © 2022, CMAS Center
Made with Sphinx and @pradyunsg's Furo