Contents Menu Expand Light mode Dark mode Auto light/dark mode
aws-cmaq documentation
Logo
aws-cmaq documentation

Contents:

  • 1. Create Single VM
    • 1.1. Create a VM from the AWS Web Console
    • 1.2. Create a VM using the AWS Command Line
    • 1.3. Run CMAQv5.4 on c6a.2xlarge
  • 2. Create a Parallel Cluster and run CMAQv5.4
    • 2.1. Build a Demo ParallelCluster
    • 2.2. Use ParallelCluster with Software and Data pre-installed on hpc7g.16xlarge
    • 2.3. Run CMAQ on hpc7g.16xlarge
    • 2.4. Run DESID CMAQ on hpc7g.16xlarge
    • 2.5. Modify the ParallelCluster to remove the lustre filesystem
    • 2.6. Create Cost Allocation Tags for Analysis using AWS Cost Explorer.
  • 3. Performance and Cost Optimization
    • 3.1. ParallelCluster Configuration
    • 3.2. CMAQv5.4 Benchmarks
    • 3.3. Slurm Compute Node Provisioning
    • 3.4. Benchmark Timings for CMAQv5.4 12US1 Benchmark
    • 3.5. Benchmark Scaling Plots for CMAQv5.4 12US1 Benchmark
    • 3.6. Cost Information
    • 3.7. Recommended Workflow for extending to annual run
  • 4. Developer Guide to install and run CMAQv5.4 on Single VM or Parallel Cluster
    • 4.1. Install CMAQv5.4+ on Single Virtual Machine Advanced (optional)
      • 4.1.1. Install Software and run CMAQv5.4 on c6a.xlarge for the 12km Listos Training Domain
      • 4.1.2. Install Software and run CMAQv5.4 on c7g-hpc7g for the 12km Listos Training Domain
      • 4.1.3. Install I/O API libraries that support HDF5
      • 4.1.4. Upgrade to run CMAQ on larger EC2 Instance
    • 4.2. Install CMAQv5.4 on ParallelCluster (optional)
      • 4.2.1. Configure Parallel Cluster
      • 4.2.2. Create the hpc7g.16xlarge pcluster
      • 4.2.3. Install CMAQ sofware and libraries on ParallelCluster version 3.6
      • 4.2.4. Install netCDF libraries that use HDF5 and support nc4 compressed files
      • 4.2.5. Install gh following these instructions
      • 4.2.6. Use gh authentication
      • 4.2.7. Run CMAQ using hpc7g.16xlarge compute nodes
      • 4.2.8. Run CMAQ using hpc7g.8xlarge compute nodes
      • 4.2.9. Install Input Data on ParallelCluster
  • 5. Post-process and QA
    • 5.1. Post-process CMAQ and Install R and Anacoda
      • 5.1.1. Scripts to run combine and post processing
      • 5.1.2. Install R, Rscripts and Packages
      • 5.1.3. Install Anaconda on the /shared/build directory
    • 5.2. QA CMAQ
      • 5.2.1. Quality Assurance
      • 5.2.2. Run m3diff to compare the output data for two runs that have different values for NPCOL
      • 5.2.3. Run an R script to create the box plots and spatial plots comparing the output of two runs
      • 5.2.4. Run Jupyter Notebook to analyze difference between with DESID Emissions and the base case (no emission reduction)
    • 5.3. Compare Timing of CMAQ Routines
      • 5.3.1. Parse timings from the log file
    • 5.4. Copy Output to S3 Bucket
      • 5.4.1. Copy Output Data and Run script logs to S3 Bucket
      • 5.4.2. Copy scripts and logs to /fsx
      • 5.4.3. Examine the output files
      • 5.4.4. Copy the output to an S3 Bucket
  • 6. Logout and Delete ParallelCluster
    • 6.1. Logout of cluster when you are done
    • 6.2. Delete Cluster
    • 6.3. Verify that the cluster was deleted
  • 7. Run AMET on a VM
    • 7.1. AMET on AWS
    • 7.2. Spin up a server using pre-installed AMET AMI
    • 7.3. Create Air Quality Plots using the AMET AQ Website
    • 7.4. Programs to create plots (74)
    • 7.5. Air Quality Observation Networks (48)
    • 7.6. Air Quality and Met Species (172)
    • 7.7. Method used to create plots
    • 7.8. Example plots using the aqExample database
    • 7.9. Load your own AQ model data to MariaDB
    • 7.10. Loading EPA’s EQUATES Database
    • 7.11. Example plots using the EQUATES 2002-2019 Projects in the amad_EQUATES database
    • 7.12. Create Met Plots using the AMET Met Website
    • 7.13. Met Observation Networks
    • 7.14. Load your own MET data to MariaDB
    • 7.15. Types of Errors Creating Plots and how to avoid them.
    • 7.16. How AMETv1.6 was installed (developers only)
  • 8. Additional Resources
    • 8.1. FAQ
    • 8.2. Free Training
    • 8.3. Another workshop to learn the AWS CLI 3.0
    • 8.4. Youtube video
    • 8.5. Intro to AWS for HPC People - HPC Tech Shorts
    • 8.6. Benchmarking
    • 8.7. Help Resources for CMAQ
    • 8.8. Computing on the Cloud References
    • 8.9. AWS Resources for the aws cli method to launch ec2 instances.
    • 8.10. Resources from AWS for diagnosing issues with running the Parallel Cluster
    • 8.11. Instructions on how to create Parallel Cluster Amazon Machine Image (AMI) from the command line
    • 8.12. ParallelCluster Update
    • 8.13. Use Elastic Fabric Adapter/Elastic Network Adapter for better performance
    • 8.14. VPC Management
    • 8.15. Using Cost Allocation Tags with ParallelCluster
  • 9. Future Work
    • 9.1. Future Work
  • 10. Contribute to this Tutorial
    • 10.1. Contribute to Pcluster-cmaq Documentation
Back to top

5.3. Compare Timing of CMAQ Routines#

Compare the timing of CMAQ Routines for two different run configurations.

  • 5.3.1. Parse timings from the log file
    • Compare the timings for the CONUS ParallelCluster Runs
    • Edit the R script
      • Run parse_timing.pes.lustre.cmaqv5.4.r script to examine timings of each science process in CMAQ
Next
5.3.1. Parse timings from the log file
Previous
5.2.4. Run Jupyter Notebook to analyze difference between with DESID Emissions and the base case (no emission reduction)
Copyright © 2022, CMAS Center
Made with Sphinx and @pradyunsg's Furo
Last updated on 2026-06-18 16:23:10 +0000