1 / 17

Neutron Science TeraGrid Gateway Update

Neutron Science TeraGrid Gateway Update. Vickie Lynch, Meili Chen, John Cobb Oak Ridge National Laboratory AUS Technical Presentation October 8, 2009. Outline. Sinergia Startup Project Fitting Service Transition to SNS. Sinergia Startup Project.

karlyn
Download Presentation

Neutron Science TeraGrid Gateway Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Neutron Science TeraGrid Gateway Update Vickie Lynch, Meili Chen, John Cobb Oak Ridge National Laboratory AUS Technical Presentation October 8, 2009

  2. Outline • Sinergia Startup Project • Fitting Service • Transition to SNS

  3. Sinergia Startup Project • Collaboration between the University of Zürich, ETH Zürich and Oak Ridge National Laboratory (ORNL) • Swiss Sinergia grant supports PostDoc at Zürich and graduate student at ORNL

  4. Diffuse Scattering Modeling • 3D fitting of large datasets • Needed for Single Crystal instruments • Topaz (SNS) • Snap (SNS) • MaNDi (SNS) • Corelli (SNS) • Four-circle Diffractometer (HFIR) • Imagine (HFIR)

  5. Sinergia July 09 Kickoff in Zurich • Organized by Hans-Beat Bürgi • Michel Chodkiewicz (PostDoc at Zürich) will develop new C++ diffuse scattering code over 3 years • Tara Clark (Grad student at ORNL) working with Vickie Lynch will get previous code running on TeraGrid and test • Jürg Hauser from University of Bern will provide a reference dataset with less noise

  6. Original diffuse scattering code • Used for publications: • Weber T.,Bürgi H.B., Determination and refinement of disordered crystal structures using evolutionary algorithms in combination with Monte Carlo methods, Acta Crystallographica A 58, 526-540 (2002) • H.B. Bürgi, J. Hauser, T. Weber, R.B. Neder, Supramolecular Architecture in a Disordered Perhydrotriphenylene Inclusion Compound from Diffuse X-ray Diffraction Data. Crystal Growth & Design, 5 2073-2083 (2005) • Master/slave code was written mainly in Perl with telnet to communicate between workstations (Calls Fortran and C codes for FFTs) • Much file communication between master and slave • Code assumed unique disk space for each slave • One run for over 200 generations took ~30 days on ~10 workstations

  7. VOP code by Phil Bentley • Vitess Optimisation Program written for optimizing parameters in instrument simulations • Genetic algorithm with tournament selection; Swarms • Written in C++ using MPI for parallel computing (Allreduce to find min; Bcast to sent best parameters) • Command line input for Vitess with parameters to be optimized preceded by @ • Limits file read by code • Written to maximize flux from Vitess, but easily modified to minimize c2

  8. Code port to TeraGrid • Replaced master telnet code with VOP MPI code • VOP calls Perl slaves with pipes • Unique directories created for each slave • Ported and ran successfully on NSTG • Needed faster turnaround and 1048 clones • Got Kraken Startup allocation • Popen for Perl slaves did not work with CNL

  9. Code port to Ranger • Tested that popen worked on Ranger • Transferred part of Startup to Ranger • Too much file communication in Perl slaves for 16 cores/node • Put OpenMP directives in FFT codes • Using 4 slaves/node with 4 OpenMP threads/slave

  10. VOP results with Perl slaves

  11. Sinergia accomplishments • Add differential evolution option to vop master • Compared differential evolution, genetic algorithm and swarm for disordered crystal data. Genetic algorithm converged fasted, but each generation takes twice as long using tournament selection • Added hybrid OpenMP directives • Have code running on NSTG and Ranger • Have Levenberg-Marquardt nonlinear least squares driver for slaves for further optimization • Running using reference dataset on Ranger now (160 cores)

  12. Future Sinergia work • Install PGPLOT Perl module and helper scripts • Testing with reference dataset • Number of individuals • Number of clones • Optimize mutation and crossover constants • Scale to thousands of cores • Automatically switch between genetic algorithms and least squares using history of chisq • Write slave code in C++, not Perl • Communicate between modules with memory, not files

  13. Fitting Service • Fast data fitting, the major and the most time consuming procedure of the data analysis, is the key of high performance data analysis and real-time data processing. • Uses NL2SOL or Dakota to fit experimental data • http://www.ornl.teragrid.org/fitting/FitSrv_method.doc • Instrument scientists writes functions • Read data • Model to fit data • http://www.ornl.teragrid.org/fitting/FitSrv_Portal_uInterface.pdf

  14. Fitting Service results • Fitting service used for experimental reflectometer data is shown • NL2SOL results show 5 times smaller Chisq than previous fitting code used for this data

  15. Fitting Service uses TeraGrid • New fitting service • going into production • available from the development portal • will be “monitored” for health in the future • Calculation scales linearly on TeraGrid on Mercury at NCSA • See TG09 slides by Meili Chen for more details.

  16. Fitting Service GUI in Portal

  17. NSTG Cluster Transition to SNS • Cluster: dual processor 3.06 GHz Intel Xeon nodes • 14 nodes with 2.5GB of memory are compute nodes • 4 nodes with 4GB memory are dedicated to GridFTP • Transitioning to Scientific Linux 5.2 • Move planned from ORNL to SNS machine room (~ 3 miles) • Cluster will link SNS with TeraGrid

More Related