2.2. CMAQv5.3.3 on Single Virtual Machine Intermediate (software pre-installed)

Creating an EC2 instance from the Command Line is easy to do. In this tutorial we will give examples on how to create and run using ec2 instances that vary in size depending on the size of the CMAQ benchmarks.

Using Amazon EC2 with the AWS CLI SPOT Pricing

Benchmark Name	Grid Domain	EC2 Instance	vCPU	Cores	Memory	Network Performance	Storage	On Demand Hourly Cost	Spot Hourly Cost
2016_12SE1	(100x80x35)	c6a.2xlarge	8	4	16 GiB	Up to 12500 Megabit	EBS Only	0.306	0.2879

Data in table above is from the following: Sizing and Price Calculator from AWS

Run CMAQv5.33 on a single Virtual Machine (VM) using an ami with software pre-loaded to run on either a c6a.2xlarge, c6a.8xlarge or c6a.48xlarge instance with gp3filesystem.

• 2.2.1. Learn how to Use AWS CLI to launch c6a.2xlarge EC2 instance using Public AMI
  • Public AMI contains the software and data to run 2016_12SE1 benchmark using CMAQv5.33
    • Verify that you can see the public AMI on the us-east-1 region.
    • AWS Resources for the aws cli method to launch ec2 instances.
  • Use the publically available AMI to launch an ondemand c6a.2xlarge ec2 instance using a io2 volume with 100000 IOPS with hyperthreading disabled
  • Obtain the code using git
    • Use the following command to obtain the public IP address of the machine.
    • Login to the ec2 instance
    • Login to the ec2 instance again, so that you have two windows logged into the machine.
  • Load the environment modules
  • Update the pcluster-cmaq repo using git
  • Run CMAQv5.4 for 12US1 Listos Training 3 Day benchmark Case on 4 pe
    • Use command line to submit the job. This single virtual machine does not have a job scheduler such as slurm installed.
    • Use HTOP to view performance.
    • Successful output
    • Change to the scripts directory
    • Use lscpu to confirm that there are 4 cores on the c6a.2xlarge ec2 instance that was created with hyperthreading turned off.
    • Edit the 12US3 Benchmark run script to use the gcc compiler and to output all species to CONC output file.
    • Run the 12US3 Benchmark case
    • Use HTOP to view performance.
    • Successful output for 12 species output in the 3-D CONC file took 56 minutes to run 1 day
    • Find the InstanceID using the following command on your local machine.
    • Stop the instance
    • Terminate Instance
    • Verify that the instance is being shut down.
• 2.2.2. Learn how to Use AWS CLI to launch c6a.8xlarge EC2 instance using Public AMI
  • Public AMI contains the software and data to run 2016_12SE1 using CMAQv5.3.3
    • Verify that you can see the public AMI on the us-east-1 region.
    • AWS Resources for the aws cli method to launch ec2 instances.
  • Use the publically available AMI to launch an ondemand c6a.8xlarge ec2 instance using a io2 volume with 100000 IOPS with hyperthreading disabled
  • Obtain the code using git
    • Use the following command to obtain the public IP address of the machine.
    • Login to the ec2 instance (may need to wait 5 minutes for the ec2 instance to initialize and be ready for login)
    • Login to the ec2 instance again, so that you have two windows logged into the machine.
  • Load the environment modules
  • Update the pcluster-cmaq repo using git
  • Run CMAQv5.3.3 for 2016_12SE1 1 Day benchmark Case
    • Edit the run script to run on 16 cores
    • Use command line to submit the job. This single virtual machine does not have a job scheduler such as slurm installed.
    • Use HTOP to view performance.
    • Successful output
    • Find the InstanceID using the following command on your local machine.
    • Stop the instance
    • Terminate Instance
    • Verify that the instance is being shut down.
• 2.2.3. Learn how to Use AWS CLI to launch c6a.48xlarge EC2 instance using Public AMI
  • Public AMI contains the software and data to run 2016_12SE1 using CMAQv5.3.3
    • Verify that you can see the public AMI on the us-east-1 region.
    • AWS Resources for the aws cli method to launch ec2 instances.
  • Use the publically available AMI to launch an ondemand c6a.48xlarge ec2 instance using a gp3 volume with 16000 IOPS with hyperthreading disabled
  • Obtain the code using git
    • Use the following command to obtain the public IP address of the machine.
    • Login to the ec2 instance
    • Login to the ec2 instance again, so that you have two windows logged into the machine.
  • Load the environment modules
  • Update the pcluster-cmaq repo using git
  • Run CMAQv5.3.3 for 2016_12SE1 1 Day benchmark Case on 4 pe
    • Use command line to submit the job. This single virtual machine does not have a job scheduler such as slurm installed.
    • Use HTOP to view performance.
    • Successful output
    • Use lscpu to confirm that there are 8 processors on the c6a.2xlarge ec2 instance that was created with hyperthreading turned on.
    • Run 12US2 benchmark again using gp3 volume
    • Stop the instance
    • Terminate Instance
    • Verify that the instance is being shut down.
• 2.2.4. Documentation of Troubleshooting effort for CMAQv5.4+ on 12US1
  • Public AMI contains the software and data to run 12US1 using CMAQv5.4+
    • Verify that you can see the public AMI on the us-east-1 region.
    • AWS Resources for the aws cli method to launch ec2 instances.
  • Use a publically available AMI to launch a c6a.48xlarge ec2 instance using a gp3 volume with 16000 IOPS
    • Use the following command to obtain the public IP address of the machine.
    • Login to the ec2 instance
  • Load the environment modules
  • Run CMAQv5.4 for the 12km Listos Training Case
  • Run CMAQv5.4 for the full 12US1 Domain on c6a.48xlarge with 192 vcpus
    • Use command line to submit the job. This single virtual machine does not have a job scheduler such as slurm installed.
  • Once the model starts running (see Processing cmpleted …) in the log file, then use htop to view the CPU usage.
    • Using Cloudwatch to see the CPU utilization.
  • Save volume as a snapshot
  • Clean up Virtual Machine
    • Find the InstanceID using the following command on your local machine.
    • Terminate the instance
  • Create c6a.48xlarge with hyperthreading disabled
    • Retry the Listos run script.
    • Use HTOP to view performance.
    • Successful output
    • Used the AWS Web Interface to upgrade to an io1 system
• 2.2.5. Use new ami instance with faster storage (io1) to create c6a.48xlarge ec2 instance
  • Create new instance
  • Check that the ec2 instance is running using the following command.
  • Use the following command to obtain the IP address
  • Login
  • Load environment modules
  • Change to the scripts directory
  • Use lscpu to confirm that there are 96 processors on the c6a.48xlarge ec2 instance that was created with hyperthreading turned off.
  • Login to the ec2 instance again, so that you have two windows logged into the machine.
  • Run 12US1 Listos Training 3 Day benchmark Case on 32 pe (this will take less than 2 minutes)
  • Successful output
  • Download input data for 12NE3 1 day Benchmark case
  • Use the aws s3 copy command to copy data from the CMAS Data Warehouse Open Data S3 bucket.
  • Link the data directory on /shared/data
  • Edit the 12US3 Benchmark run script to use the gcc compiler and to output all species to CONC output file.
  • Run the 12US3 Benchmark case
  • Successful output for 12 species output in the 3-D CONC file took 7.4 minutes to run 1 day
  • Successful output for all species output in the 3-D CONC File (222 variables)
  • Run 12US1 2 day benchmark case on 96 processors
  • Verify that it is using 99% of each of the 96 cores using htop
  • Successful run timing
  • Find the InstanceID using the following command on your local machine.
  • Stop the instance
  • Terminate Instance
  • Try creating the gp3 version of the ami using the Nitro Hypervisor, and see if that improves the performance without the cost of the io1 volume.