1 / 37

An ESG Walkthrough

An ESG Walkthrough. ESG Federation website - DCC File system for ESG. Muhammad Atif. ESG-NCI Gateway. Search. We highly recommend subscribing to our tweets. Latest news. Search and Access Data. You can search without login; but not download Quick Links → Create Account

lindsey
Download Presentation

An ESG Walkthrough

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An ESG Walkthrough ESG Federation website - DCC File system for ESG Muhammad Atif

  2. ESG-NCI Gateway Search We highly recommend subscribing to our tweets Latest news

  3. Search and Access Data • You can search without login; but not download • Quick Links → Create Account • Follow the on screen instructions • You will receive an email to confirm your registration; at the same time the administrators are also notified. • Admin has to validate you before you can download the data. • After confirmation email from admin; login to the NCI Node • Account → Apply for Group Membership • Mk-3.6 • CMIP-5 research (not in our control – request goes to PCMDI) • Requests to NCI are usually quick. • Same ID can be used on all the gateways. Your OpenID: https://esg.nci.org.au/esgcet/myopenid/<username>

  4. Searching for data • Recommend that you browse the website and get familiarized. • If asked about authentication, you may use a temporary openid • https://esg.nci.org.au/esgcet/myopenid/dcc000 • Password: abc123 • Please note that this openid will be removed after the workshop, it is highly recommended that you create your openid

  5. Download data from the Gateway • Three Download Methods • Using the web browser. • A set of wget-scripts. • Via GridFTP (Data mover lite)

  6. Download Method-1 (Web browser based) • Intuitive but slow • Follow on-screen instructions and nothing can go wrong. • Works like normal downloads from the browser of your choice i.e. click to download. • IE, FF, Chrome and Safari are supported. • Works well if you are after couple of files

  7. Download -2 (Wget Scripts) • Ability to download multiple files • Select the files (variables) you are interested in. • Presents a wget-download.sh script, that you need to save and run. • Command line based – No GUI • Two methods of Authentication • Authorization token (Depreciated) • PCMDI gateway only. • No login required, However the token expires in 24 hours and the script is of no use after that. • My Proxy Login (Official) • Needs a separate step for authentication. • Need to run Java applet or MyProxyClient software • If authentication expires, just run the MyProxyClient software • Note: If you are interested in doing lots of downloads, we can provide a custom script to speedup the process on DCC……… Example to follow later

  8. Process for MyProxyLogonDownload on the DCC • ssh abc123@dcc.nci.org.au -Y • Download MyProxyLogon-ESG.jar file • wget http://esg.nci.org.au/esgcet/webstart/myProxyLogon/MyProxyLogon-ESG.jar • Run MyProxyLogon.jar file; • module load java • Java –jar MyProxyLogon-ESG.jar • It writes the certificates to your $HOME/.esg folder • Run the wget-download.sh command

  9. Download-3 (DML) • Parallel Downloads • DML Preferences Concurrency • Faster than wget • Uses GridFTP • Caveat: Not available on all ESG nodes • NCI one of the few that has the facility

  10. ESG Data on the DCC • IPCC AR5/CMIP5 • CSIRO-QCCCE Mk-3.6 • CAWCR ACCESS • Replicated data from the other ESG nodes. • Other data • CMIP3 • Observational data • Processed data

  11. DCC File system organization • All ESG data in /projects/ESG: • Authoritative • Unofficial-ESG-replica • CAWCR_CVC_processed • /projects/ESG/Authoritative • Serves data using the policies of the ESG Federation • This is the directory that our ESG software serves data from • All data is the current official copy • User example: login to the DCC and have a look.

  12. Unofficial Replica • /projects/ESG/Unofficial_Replica • IPCC • The IPCC directory is where you can reference data that we have downloaded from other nodes (though not an official replica). The subdirectories could be partial datasets or complete ones. • IPCC_tmp_flat • Direct symlinks to files, flat directory structure • tmp • You can download your data here in $USER folder. • We can provide you with scripts to help download data here • GlobalObs_and_Reanalysis • data sourced from various places that Lawrie Rikus/Ben Hu have been maintaining. • Also served through a THREDDS service - for remote access

  13. Unofficial Replica • /projects/ESG/Unofficial-ESG-replica/IPCC • User downloads using wget scripts/DML. • Partial data; Not all of the data is downloaded. • Does not necessarily contain the most up to date version • Data may be changed by the remote node since last download. • ESG (and official replica directory) always has the latest version. • Organised as Data Reference Syntax (DRS)

  14. Data Reference Syntax (How files are organized @ DCC) • This is how the tree looks like compared to DRS • DRS cmip5.<product>.<institute>.<model>.<experiment>. <time_frequency>.<realm>.<cmor_table>.<ensemble> • File System /projects/ESG/unofficial-ESG-replica/IPCC/CMIP5/output1/NCC/NorESM1-M/historical/mon/seaIce/OImon/r1i1p1/v20110901/sic/<FILE>

  15. Downloading data to DCC File System • If you would like a significant amount of data that we don’t have, then … please contact us. • Reasons: • It may already be downloaded but not linked • Downloading data is still tricky • Space management • That said – we would like to facilitate downloads of priority data. • How? …. Lets do it

  16. Demo • Download the wget file from esg-gateway • ssh –Y user@dcc.nci.org.au • java –jar MyProxyLogon-ESG.jar • Copy wget-file to dcc (scp, copy n paste) in a new folder • ./esg-download.py wget-download.sh • View the directory, it should have a number of wget-split-* • ./esg-qsub-download.py –i wget-split- • Press “y” • Check the files after some time We will be streamlining it further

  17. Help • DCC and ESG both are evolving continuously • Comments and suggestions are always welcome • Help Desks • Anything related to ESG federation website/ other models are not native to NCI • Cmip5-helpdesk@stfc.ac.uk • Related to DCC compute cluster • help@nf.nci.org.au

  18. Downloads are managed by Us • GridFTP • Fast • Downloads managed by you as a user

  19. Controlled Vocab: • http://esg-pcmdi.llnl.gov/internal/esg-data-node-documentation/cmip5_controlled_vocab.txt

  20. ESG-NCI Gateway Search by categories We highly recommend subscribing to our tweets

  21. Search and Access Data • You can search without login; but not download • Quick Links → Create Account • Follow the on screen instructions • You will receive an email to confirm your registration; at the same time the administrator(s) are also notified. • Admin has to validate you before you can download the data. • After confirmation email; login to the NCI Node • Account → Apply for Group Membership • Mk-3.6 • CMIP-5 research (not in our control – request goes to PCMDI) • Requests to NCI are usually quick. • For others, this may take time (one – two days)

  22. Searching for data

  23. Download data from the Gateway • Three Methods • Using the web browser. • A set of wget-scripts. • Via GridFTP (Data mover lite)

  24. Download – 1 (Web based) • Intuitive but slow • Follow on-screen instructions and nothing can go wrong. • Works like normal downloads from the browser of your choice • IE, FF, Chrome and Safari are supported. • Works well if you are after couple of files

  25. Download -2 (Wget Scripts) • Ability to download multiple files • Presents you with a wget-download.sh script. • Command line based – No GUI • Two methods • Authorization token (Depreciated) • PCMDI gateway only. • My Proxy Login (Official) • Needs a separate step for authentication • Need to run Java applet or MyProxyClient software • Note: If you are interested in doing lots of downloads, we can provide a custom script to speedup the downloads on DCC.

  26. Process for MyProxyLogonDownload on the DCC • ssh abc123@dcc.nci.org.au -Y • Download MyProxyLogon-ESG.jar file • wget http://esg.nci.org.au/esgcet/webstart/myProxyLogon/MyProxyLogon-ESG.jar • Run MyProxyLogon.jar file; instructions are provided in the wget download script that you have already downloaded via web-browser. • module load java • Java –jar MyProxyLogon-ESG.jar • It writes the certificates to your $HOME/.esg folder

  27. Download-3 (DML) • Parallel Downloads • DML Preferences Concurrency • Faster than wget • Uses GridFTP • Caveat: Not available on all ESG nodes • NCI one of the few that has the facility

  28. Interacting with ESG data on DCC

  29. ESG Data on the DCC • IPCC AR5/CMIP5 • CSIRO-QCCCE Mk-3.6 • CAWCR ACCESS • Replicated data from the other ESG nodes. • Other data • CMIP3 • Observational data • Processed data

  30. DCC File system organization • All ESG data in /projects/ESG: • Authoritative • Unofficial-ESG-replica • CAWCR_CVC_processed • /projects/ESG/Authoritative • Serves data using the policies of the ESG Federation • This is the directory that our ESG software serves data from • All data is the current official copy. • User example: login to the DCC and have a look.

  31. Unofficial Replica • /projects/ESG/Unofficial_Replica • IPCC • The IPCC directory is where you can reference data that we have downloaded from other nodes (though not an official replica). The subdirectories could be partial datasets or complete ones. • IPCC_tmp_flat • Direct symlinks to files, flat directory structure • tmp • You can download your data here in $USER folder. • We can provide you with scripts to help download data here • GlobalObs_and_Reanalysis • data sourced from various places that Lawrie Rikus/Ben Hu have been maintaining. • Also served through a THREDDS service - for remote access

  32. Unofficial Replica • /projects/ESG/Unofficial-ESG-replica/IPCC • User downloads using wget scripts/DML. • Partial data; Not all of the data is downloaded. • Does not necessarily contain the most up to date version • Data may be changed by the remote node since last download. • ESG (and official replica directory) always has the latest version. • <MOVE TO NEW SLIDE>Organised as Data Reference Syntax (DRS) cmip5.<product>.<institute>.<model>.<experiment>.<time_frequency>.<realm>.<cmor_table>.<ensemble> Ref the official link to the standard.

  33. Downloading data to our ESG • If you would like a significant amount of data that we don’t have, then … please contact us. • Reasons: • It may already be downloaded but not linked • Downloading data is still tricky • Space management • That said – we would like to facilitate downloads of priority data. • How? …. (new slide)

  34. Downloads are managed by Us • GridFTP • Fast • Downloads managed by you as a user

  35. Controlled Vocab: • http://esg-pcmdi.llnl.gov/internal/esg-data-node-documentation/cmip5_controlled_vocab.txt

More Related