slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Chapter 10 Cloud-Enabling Dust Storm Forecasting PowerPoint Presentation
Download Presentation
Chapter 10 Cloud-Enabling Dust Storm Forecasting

Loading in 2 Seconds...

play fullscreen
1 / 31

Chapter 10 Cloud-Enabling Dust Storm Forecasting - PowerPoint PPT Presentation


  • 112 Views
  • Uploaded on

Chapter 10 Cloud-Enabling Dust Storm Forecasting. Qunying Huang, Jizhe Xia, Manzhu Yu, Karl Benedict and Myra Bambacus. Learning Objectives. General computing challenges for computing intensive applications How cloud computing can help address those issues

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Chapter 10 Cloud-Enabling Dust Storm Forecasting' - aspen-palmer


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Chapter 10 Cloud-Enabling Dust Storm Forecasting

QunyingHuang, Jizhe Xia, Manzhu Yu,

Karl Benedict and Myra Bambacus

learning objectives
Learning Objectives

General computing challenges for computing intensive applications

How cloud computing can help address those issues

Configure HPC cluster on the cloud

Deploy duststorm model onto the cloud

Run the dust storm on the cloud

Performance and cost analysis

learning materials
Learning Materials
  • Videos:
      • Chapter_10_Video.mp4
  • Scripts, Files and others:
      • mirror.tar.gz
learning modules
Learning Modules
      • Dust storm modeling and challenges
      • Dust storm model cloud deployment
        • General steps
        • Special considerations
      • Use case: Arizona Phoenix 2011 July 05
  • Nested modeling
  • Cloud performance and efficiency analysis
      • Conclusion and discussions
dust storm hazards
Dust Storm Hazards

Illness & Diseases

Traffic & Car accidences

Air Pollution

Ecological System

Desertification

Global/regional Climate

Phoenix Dust Storm a "100-Year Event“, 2011, July 5th

dust storm models
Dust storm models
  • Eta-4bin (Kallos et al., 1997; Nickovic et al., 1997)
    • Low resolution (30 KM)
    • Large area
  • Eta-8bin (Nickovic et al., 1997; Nickovic et al., 2011)
    • 8 categories of dust particles
    • Low resolution (30 KM)
    • Large area
  • NMM-dust (Janjic et al., 2001; Janjic, 2003)
    • High resolution (3 KM)
    • Small area
dust storm modeling
Dust storm modeling
  • Dust storm modeling (Purohit et al., 1999)
  • Divide the domain into three-dimensional grid cells
  • Solve a series of numerical equations on each cell
  • Numerical calculations are repeated on each cell
dust storm forecasting challenges
Dust Storm Forecasting Challenges

O(n^4)

Input/output

Southwest U.S.

Finish One-day forecasting in 2 hours

Dust Storm Forecasting

learning modules1
Learning Modules
      • Dust storm modeling and challenges
      • Dust storm model cloud deployment
        • General steps
        • Special considerations
      • Use case: Arizona Phoenix 2011 July 05
  • Nested modeling
  • Cloud performance and efficiency analysis
      • Conclusion and discussions
dust storm model deployment onto the cloud
Dust Storm Model Deployment onto the Cloud

1. Authorize network access

12. Create two new AMIs from the

running instances

2. Launchone cluster instanceas the

head node

11. Configure and test the model

3. SSH to the instance

10. Export the NFS directory to the

computing node

4. Install the software dependency and

middleware, e.g., NFS, and MPICH2

9. Deploy the model on the NFS exporting

directory

5. Create a new AMI from the head node

and start an instance from the new AMI

8. Mount the volume to the NFS exporting

directory

6. Configure the middleware on both

nodes to enable communication

7. Create an EBS volume

Video: Chapter_10_Video.mp4

step 1 authorize network access
Step 1. Authorize network Access
  • Dust Storm Model Deployment onto the Cloud
  • Configure firewall rule for the firewall group “hpc”
  • Open port 22 for SSH
  • Open port 9000 -10000 for MPICH2

Video: Chapter_10_Video.mp4

0:00 – 1:35

slide12

Dust Storm Model Deployment onto the Cloud

  • Step 2. Lunch one cluster instance as the head node
  • Use Amazon cluster compute AMI
  • Create a ssh key pair “hpc”, and save the public key pair file to local storage as “hpc.pem”
  • Use the security group “hpc”
  • Step 3. SSH to the instance
  • Use the public ssh key file “hpc.pem”
  • Change the user permission for “hpc.pem” as 600 (user root read permission only)

Video: Chapter_10_Video.mp4

1:35 – 4:10

step 4 install software dependencies

Dust Storm Model Deployment onto the Cloud

Step 4. Install software dependencies

Install NFS and MPICH2:

  • [root@domU-head~] yum install gccgcc-c++ autoconfautomake
  • [root@domU-head~] yum –y install nfs-utilsnfs-utils-lib sytem-config-nfs#install NFS
  • [root@domU-head~] wgethttp://www.mcs.anl.gov/research/projects/mpich2staging/
  • goodell/downloads/tarballs/1.5/mpich2-1.5.tar.gz # download MPICH2 package
  • [root@domU-head~] tar –zvxfmpich2-1.5.tar.gz # Unzip

[root@domU-head~] mkdir /home/clouduser/mpich2-install # create an installation directory

  • [root@domU-head~] cd mpich2-1.5
  • [root@domU-head~] ./configure -prefix=/home/clouduser/mpich2-install --enable-g=all --enable-fc --enable-shared --enable-sharedlibs=gcc --enable-debuginfo
  • [root@domU-head~] make # Build MPICH2

[root@domU-head~] make install # Install MPICH2

step 5 create a new ami from the head node

Dust Storm Model Deployment onto the Cloud

Step 5. Create a new AMI from the head node

Step 4. Install software dependencies

Configure and start NFS:

[root@domU-head ~] mkidr /headMnt # create a NFS export directory

[root@domU-head ~] echo "/headMnt *rw " >> /etc/exports

[root@domU-head ~] exportfs -ra

[root@domU-head ~] service nfs start #start up NFS

  • Create a computing node from the new AMI

Video: Chapter_10_Video.mp4

4:10 – 13:02

step 6 configure the head node and computing node

Dust Storm Model Deployment onto the Cloud

Step 6. Configure the head node and computing node

Keyless access from the head node to the computing nodes

[root@domU-head ~] vi /etc/hosts #access to the hosts list of the head node

[root@domU-head ~] ssh-keygen -t rsa #create a public key at the head node

[root@domU-computing~] mkdir –p /root/.ssh/

[root@domU-computing ~] scproot@domU-head: /root/.ssh/id_ras.pub /root/.ssh/ #copy the public key from the head node to the computing node

[root@domU-computing ~] cat /root/.ssh/id_ras.pub >> /root/.ssh/authorized_keys

Video: Chapter_10_Video.mp4

13:02- 17:55

step 7 create an ebs volume
Step 7. Create an EBS volume
  • Dust Storm Model Deployment onto the Cloud
  • Attach to the head node

Step 8. Mount the volume to the NFS exporting directory

  • Make a file system for the EBS volume
  • Mount the volume to head node directory /headMnt

Video: Chapter_10_Video.mp4

13:02 – 20:40

step 9 deploy the model

Dust Storm Model Deployment onto the Cloud

Step 9. Deploy the model
  • Download the model under NFS directory (/headMnt)
  • Export the NFS directory (/headMnt) to the computing node
  • Step 10. Export the NFS directory to the computing node

[root@domU-computing ~] mkdir /computingMnt # create the directory

[root@domU-computing ~] mount -t nfs -o rwdomU-head:/headMnt /headMnt #Mount the volume to the NFS export directory

Video: Chapter_10_Video.mp4

20:40 – 24:19

step 11 configure and test the model

Dust Storm Model Deployment onto the Cloud

Step 11. Configure and Test the model
  • cd /headMnt/mirror/performancetest/scripts
  • ./run_test.sh ec2 >& ec2.log &
  • Step 12. Create two new AMIs from the running instances

Video: Chapter_10_Video.mp4

24:19 – 28:18

learning modules2
Learning Modules
      • Dust storm modeling and challenges
      • Dust storm model cloud deployment
        • General steps
        • Special considerations
      • Use case: Arizona Phoenix 2011 July 05
  • Nested modeling
  • Cloud performance and efficiency analysis
      • Conclusion and discussions
special considerations
Special considerations
  • Configuring a virtual cluster environment
    • Create placement group
  • Loosely coupled nested modeling and cloud computing
  • Auto-scaling
    • write scripts with the EC2 APIs
learning modules3
Learning Modules
      • Dust storm modeling and challenges
      • Dust storm model cloud deployment
        • General steps
        • Special considerations
      • Use case: Arizona Phoenix 2011 July 05
  • Nested modeling
  • Cloud performance and efficiency analysis
      • Conclusion and discussions
nested model
Nested model
  • Nested model: provide high resolution results for one or several area of interests over a large area.

Tightly Nesting

  • A model run with multiple resolutions
  • Modifications of models (Michalakes et al., 1998)
  • Knowledge of placement for high-resolution nested subdomains

AOI: NMM-dust

(3KM)

Nested Subdomain #2

(3KM)

Loosely Nesting

ETA-8bin (30KM)

Domain #1(30KM)

  • ETA-8bin identifies AOIs(Area of Interesting)
  • <Low resolution (30 KM) , Large area>
  • NMM-dust performs forecasting over AOIs
  • <High resolution (3 KM ), Small area>
  • Nested Subdomain #3
  • (3KM)

AOI: NMM-dust

(3KM)

loosely nested model
Loosely Nested Model

Low-resolution model Results

Figure a. 18 AOIs Distribution

Low-resolution model domain area and sub-regions (Area Of

Interests, AOIs) identified for high-resolution model execution

learning modules4
Learning Modules
      • Dust storm modeling and challenges
      • Dust storm model cloud deployment
        • General steps
        • Special considerations
      • Use case: Arizona Phoenix 2011 July 05
  • Nested modeling
  • Cloud performance and efficiency analysis
      • Conclusion and discussions
run under cloud performance analysis
Run Under Cloud >> Performance analysis

Use 2 hours for one-day forecasting over Southwest of U.S. (37 X 20 degree)

18 Subregions run on 18 Amazon EC2 virtual cluster

run under cloud cost analysis cont
Run Under Cloud >> Cost analysis cont

The yearly cost of a local cluster is around 12.7 times higher than that of the EC2 cloud service if 28 EC2 instances (with 400 CPU cores) are leveraged to handle the high resolution and concurrent computing requirements for a duration of 48 hours.

conclusion
Conclusion
  • Large-scale forecasting is Computable (2 hours)
    • Loosely coupled nested model
    • Cloud computing
      • Being capable of provisioning a large amount of computing power in a few minutes
      • Economically sustaining low access rates and low resolution models
discussion questions
Discussion questions
  • What are computing challenges for dust storm forecasting?
  • What are the general steps to deploy dust storm model on the cloud?
  • Which instance type is better for dust storm forecasting, regular instance or HPC instance? Why?
  • How to configure a virtual high performance computing (HPC) cluster to support computing-intensive applications?
  • How Elastic Block Storage (ebs) service is used in supporting the dust storm model deployment to the cloud?
  • How to create a placement group for HPC instances using both Amazon web console management and command line tools? Why we need this step?
  • Compared to Chapter 5 deployment for general applications onto the cloud, what are the special considerations for dust storm model?
  • Why can cloud computing achieve cost-efficiency?
  • Why cloud computing provides a good solution to support disruptive event (e.g., dust storm) simulation?
  • What are the remaining issues while using cloud computing to support dust storm simulation?
reference
Reference
  • Huang Q., Yang C., Benedict K., Chen, S., Rezgui A., Xie J., 2013. Enabling Dust storm Forecasting Using Cloud Computing, International Journal of Digital Earth. DOI:10.1080/17538947.2012.749949.
  • Huang Q., Yang C., Benedict K., Rezgui A., Xie J., Xia J., Chen, S., 2012. Using Adaptively Coupled Models and High-performance Computing for Enabling the Computability of Dust Storm Forecasting, International Journal of Geographic Information Science. DOI:10.1080/13658816.2012.715650.
  • Xie J., Yang C., Zhou B., Huang Q., 2010. High Performance Computing for the Simulation of Dust Storms. Computers, Environment, and Urban Systems, 34(4): 278-290