Contents Menu Expand Light mode Dark mode Auto light/dark mode
aws-cmaq documentation
Logo
aws-cmaq documentation

Contents:

  • 1. System Requirements
    • 1.1. Configurations for running CMAQv5.3.3 on AWS ParallelCluster
    • 1.2. Software Requirements for CMAQ on AWS ParallelCluster
    • 1.3. Single VM Configuration for CMAQv5.3.2_Benchmark_2Day_Input.tar.gz Benchmark
    • 1.4. ParallelCluster Configuration for 12US2 Benchmark Domain
  • 2. Create Single VM and run CMAQv5.3.3 (software pre-installed)
    • 2.1. Use AWS Management Console to Create Single VM and run CMAQv5.3.3 (software pre-installed)
      • 2.1.1. Learn how to Use the AWS Management Console to launch EC2 instance using Public AMI
      • 2.1.2. Run CMAQv5.3.3 on c6a.2xlarge
      • 2.1.3. Load the environment modules
      • 2.1.4. Update the pcluster-cmaq repo using git
      • 2.1.5. Run CMAQv5.3.3 for 2016_12SE1 1 Day benchmark Case on 4 pe
      • 2.1.6. Save output data and run script logs
      • 2.1.7. Save the full input data, run scripts, output data and logs to an AMI that is owned by your account.
      • 2.1.8. Stop Instance
    • 2.2. CMAQv5.3.3 on Single Virtual Machine Intermediate (software pre-installed)
      • 2.2.1. Learn how to Use AWS CLI to launch c6a.2xlarge EC2 instance using Public AMI
      • 2.2.2. Learn how to Use AWS CLI to launch c6a.8xlarge EC2 instance using Public AMI
      • 2.2.3. Learn how to Use AWS CLI to launch c6a.48xlarge EC2 instance using Public AMI
      • 2.2.4. Documentation of Troubleshooting effort for CMAQv5.4+ on 12US1
      • 2.2.5. Use new ami instance with faster storage (io1) to create c6a.48xlarge ec2 instance
  • 3. Create a Parallel Cluster and run CMAQv5.3.3
    • 3.1.1. Introductory Tutorial
      • Step by Step Instructions to Build a Demo ParallelCluster.
      • Establish Identity and Permissions
      • Configure a Demo Cluster
      • Create a Demo Cluster
      • Login and Examine Cluster
      • Delete the Demo Cluster
    • 3.1.2. CMAQv5.3.3 Intermediate Tutorial
      • Use ParallelCluster pre-installed with software and data.
      • Create CMAQ ParallelCluster with software/data pre-installed
      • Log into the new cluster
      • Change shell to use tcsh
      • Verify Software
      • Verify Input Data
      • Examine CMAQ Run Scripts
      • Submit Job to Slurm Queue
      • Submit a minimum of 2 benchmark runs
    • 3.1.3. CMAQv5.3.3 Parallel Cluster Benchmark on HPC6a-48xlarge with EBS and Lustre (optional)
      • Use ParallelCluster pre-installed with CMAQv5.3.3 software and 12US2 Benchmark
      • Create CMAQ ParallelCluster with software/data pre-installed
      • Log into the new cluster
      • Resize the EBS Volume
      • Change shell to use tcsh
      • Verify Software
      • Verify Input Data
      • Examine CMAQ Run Scripts
      • To run on the EBS Volume a code modification is required.
      • Build the code by running the makefile
      • Submit Job to Slurm Queue to run CMAQ on Lustre
      • Submit a run script to run on the EBS volume
      • Modify YAML and then Update Parallel Cluster.
      • Submit a minimum of 2 benchmark runs
      • upgrade pcluster version to try Persistent 2 Lustre Filesystem
      • Query the stack formation log messages
  • 4. Performance and Cost Optimization
    • 4.1. Right-sizing Compute Nodes for the ParallelCluster Configuration
    • 4.2. An explanation of why a scaling analysis is required for Multinode or Parallel MPI Codes
    • 4.3. Slurm Compute Node Provisioning
    • 4.4. Spot versus On-Demand Pricing
    • 4.5. Benchmark Timings for CMAQv5.3.3 12US2 Benchmark
    • 4.6. Benchmark Scaling Plots for CMAQv5.3.3 12US2 Benchmark
    • 4.7. Cost Information
    • 4.8. Recommended Workflow for extending to annual run
    • 4.9. Side by Side Comparison of the information in the log files for 12x9 pe run compared to 9x12 pe run.
  • 5. Developer Guide to install and run CMAQv5.33 on Single VM or Parallel Cluster
    • 5.1. CMAQv5.3.3 on Single Virtual Machine Advanced (optional)
      • 5.1.1. Install Software and run CMAQv5.3.3 on c6a.2xlarge for the 2016_12US3 Benchmark
      • 5.1.2. Install I/O API libraries that support HDF5
      • 5.1.3. Upgrade to run CMAQ on larger EC2 Instance
    • 5.2. CMAQv5.3.3 Advanced Tutorial (optional)
      • 5.2.1. Use ParallelCluster without Software and Data pre-installed
      • 5.2.2. Create the c5n-4xlarge pcluster
      • 5.2.3. Update the compute nodes
      • 5.2.4. Create the c5n.18xlarge cluster
      • 5.2.5. Login to c5n.18xlarge cluster
      • 5.2.6. Install Input Data on ParallelCluster
      • 5.2.7. Install CMAQ sofware and libraries on ParallelCluster
      • 5.2.8. Run CMAQ
  • 6. Post-process and qa
    • 6.1. Post-process CMAQ and Install R
      • 6.1.1. Scripts to run combine and post processing
      • 6.1.2. Install R, Rscripts and Packages
    • 6.2. QA CMAQ
      • 6.2.1. Quality Assurance
      • 6.2.2. Run m3diff to compare the output data for two runs that have different values for NPCOL
      • 6.2.3. Use m3diff to compare two runs that have the same NPCOL
      • 6.2.4. Run an R script to create the box plots and spatial plots comparing the output of two runs
    • 6.3. Compare Timing of CMAQ Routines
      • 6.3.1. Parse timings from the log file
    • 6.4. Copy Output to S3 Bucket
      • 6.4.1. Copy Output Data and Run script logs to S3 Bucket
      • 6.4.2. Copy scripts and logs to /fsx
      • 6.4.3. Examine the output files
      • 6.4.4. Copy the output to an S3 Bucket
  • 7. Logout and Delete ParallelCluster
    • 7.1. Logout of cluster when you are done
    • 7.2. Delete Cluster
    • 7.3. Verify that the cluster was deleted
  • 8. Additional Resources
    • 8.1. FAQ
    • 8.2. Free Training
    • 8.3. Another workshop to learn the AWS CLI 3.0
    • 8.4. Youtube video
    • 8.5. Intro to AWS for HPC People - HPC Tech Shorts
    • 8.6. Benchmarking
    • 8.7. Help Resources for CMAQ
    • 8.8. Computing on the Cloud References
    • 8.9. Resources from AWS for diagnosing issues with running the Parallel Cluster
    • 8.10. Instructions on how to create Parallel Cluster Amazon Machine Image (AMI) from the command line
    • 8.11. ParallelCluster Update
    • 8.12. Use Elastic Fabric Adapter/Elastic Network Adapter for better performance
    • 8.13. VPC Management
    • 8.14. Using Cost Allocation Tags with ParallelCluster
  • 9. Future Work
    • 9.1. Future Work
  • 10. Contribute to this Tutorial
    • 10.1. Contribute to Pcluster-cmaq Documentation
Back to top
Edit this page

8.10. Instructions on how to create Parallel Cluster Amazon Machine Image (AMI) from the command line#

Tutorial How-to Create AMI from Command Line

We also need to have additional protections if we make these AMI’s public.

Building Shared AMIs

Securing Access to AMIs for AWS Marketplace

Building Pcluster from Existing AMI

Next
8.11. ParallelCluster Update
Previous
8.9. Resources from AWS for diagnosing issues with running the Parallel Cluster
Copyright © 2022, CMAS Center
Made with Sphinx and @pradyunsg's Furo