N Tropy : A Framework for Knowledge Discovery in a Virtual Universe

1. N Tropy: A Framework for Knowledge Discovery in a Virtual Universe Harnessing the Power of Parallel Grid Resources for Astrophysical Data Analysis Jeffrey P. Gardner Pittsburgh Supercomputing Center University of Pittsburgh Carnegie Mellon University

2. Mining the Universe can be (Computationally) Expensive • Computational Astrophysics: • Size of simulations are frequently limited by inability to process simulation output (no parallel group finders). • Current MPPs are 1000’s of processors…this is already too large for serial processing. • Next generation of MPPs will be 100,000’s of processors!

3. Mining the Universe can be (Computationally) Expensive • Observational Astronomy: • Paradigm shift is astronomy: Sky Surveys • Observers now generates ~ 1TB data per night • With Virtual Observatories, one can pool data from multiple catalogs. • Computational requirements are growing at a faster rate than computational power. • There will be some problems that would be impossible without parallel machines. • There will be many problems for which throughput can be substantially enhanced by parallel machines.

4. The Challenge of Parallel Data Analysis • Parallel programs are hard to write! • Steep learning curve to learn parallel programming • Lengthy development time • Parallel world is dominated by simulations: • Code is often reused for many years by many people • Therefore, you can afford to spend lots of time writing the code. • Data Analysis does not work this way: • Rapidly changing scientific inqueries • Less code reuse • Data Mining paradigm mandates rapid software development!

5. Tightly-Coupled Parallelism(what this talk is about) • Data and computational domains overlap • Computational elements must communicate with one another • Examples: • Group finding • N-Point correlation functions • New object classification • Density estimation • Solution(?): • N Tropy

6. The Goal • GOAL: Minimize development time for parallel applications. • GOAL: Enable scientists with no parallel programming background (or time to learn) to still implement their algorithms in parallel. • GOAL: Provide seamless scalability from single processor machines to MPPs…potentially even several MPPs in a computational Grid. • GOAL:Do not restrict inquiry space.

7. Methodology • Limited Data Structures: • Astronomy deals with point-like data in an N-dimension parameter space • Most efficient methods on these kind of data use trees. • Limited Methods: • Analysis methods perform a limited number of fundamental operations on these data structures.

8. N Tropy Design PKDGRAV already provides a number of advanced services • PKDGRAV benefits to keep: • Flexible client-server scheduling architecture • Threads respond to service requests issued by master. • To do a new task, simply add a new service. • Portability • Interprocessor communication occurs by high-level requests to “Machine-Dependent Layer” (MDL) which is rewritten to take advantage of each parallel architecture. • Advanced interprocessor data caching • < 1 in 100,000 off-PE requests actually result in communication.

9. N Tropy New Features • Dynamic load balancing (available now) • Workload and processor domain boundaries can be dynamically reallocated as computation progresses. • Data pre-fetching?? (To be implemented) • Predict request off-PE data that will be needed for upcoming tree nodes. • mdlSlurp(): Prefetch a big block of data to a special block of local memory • Intelligent prediction: Investigate active learning algorithms to prefetch off-PE data.

10. Performance 10 million particles Spatial 3-Point 3->4 Mpc (SDSS DR1 takes less than 1 minute with perfect load balancing)

11. PHASE 1 Performance 10 million particles Projected 3-Point 0->3 Mpc

12. NTropy Conceptual Schematic Web Service Layer (at least from Python) WSDL? SOAP? Key: Framework Components Tree Services User Supplied VO Computational Steering Layer C, C++, Python (Fortran?) Framework (“Black Box”) Dynamic Workload Management Domain Decomposition/ Tree Building User tree and particle data Collectives Tree Traversal Parallel I/O User serial I/O routines User tree traversal routines User serial collective staging and processing routines

13. N Tropy Design • Use only Parallel Management Layer (pst) and Interprocessor Communication Layer (mdl) of PKDGRAV. • Rewrite everything else from scratch PKDGRAV Functional Layout Computational Steering Layer Executes on master processor Parallel Client/Server Layer (pst) Coordinates execution and data distribution among processors Serial Layer Executes on all processors Gravity Calculator Hydro Calculator Interprocessor Communication Layer (mdl) Passes data between processors

14. N Tropy Functional Layout Executed on master thread only Application Steering script ntropy_* functions NTropy Computational Steering API Key: Layers completely rewritten PKDGRAV Parallel Client/Server Layer Layers retained from PKDGRAV “User” Supplied Methods Thread services layer (executed on all threads) Application “Tasks” (serial code) Tree node and particle navigation methods: (e.g. Acquire node/particle, next node/particle, parent node, child node ntp_* functions MDL Interprocessor communication layer Executed on all threads

15. Using Ntropy (example: N-body)Computational Steering Layer (nbody.c) NTropy stuff App-specific stuff #include “ntropy.h” /* Mandatory */ #include “nbody.h” /* My application-specific stuff */ int main(int argc,char **argv) { NTROPY ntropy; /* Mandatory */ struct nbodyStruct nbody; /* My App-specific struct */ /* Start ntropy. Only the master thread will ever return from this function. * The slave threads will return to "main_ch" in ntropy.c. */ ntropy = ntropy_Initialize(argc, argv); /* Process command line arguments */ /* Add nbody-specific command line arguments */ nbodyAddParams(ntropy, &nbody); /* Read in command line arguments */ ntropy_ParseParams(ntropy, argc, argv); /* Now that the command-line parameters have been parsed, we can * examine them. */ nbodyProcParams(ntropy, &nbody); /* Open log file and write header */ fpLogFile = nbodyLogParams(ntropy, &nbody);

16. Using Ntropy (example: N-body)Computational Steering Layer (nbody.c) /* Start threads by calling ntropyThreadStart. This starts NTropy computational services on all compute threads, pushes the data in Global and Param structs to all threads, and runs the function tskLocalInit on all threads. tskLocalInit is the constructor for your thread environment, and is convenient if you want to set up any local structs. You can also provide tskLocalFinish which is destructor for your thread environment. */ nThreads = ntropy_ThreadStart(ntropy, &(nbody.global), sizeof(nbody.global), &(nbody.param), sizeof(nbody.param), (*tskLocalInit), (*tskLocalFinish)); And in task_fof.c: void *tskLocalInit(NTP ntp) { struct nbodyLocalStruct *local; /* A struct that I invent that will store * all “thread-global” variables that I * will need for this computation */ local = (nbodyLocalStruct *)malloc(sizeof(struct nbodyLocalStruct)); return (void *)local; } void tskLocalFinish(NTP ntp) { struct nbodyLocalStruct *local = ntp_GetLocal(ntp); free(local); }

17. Using Ntropy (example: N-body)Computational Steering Layer (nbody.c) N-body structs: /* This struct has everything that I read in from the command line */ struct nbodyParamStruct { double dOmega0; int nSteps; … } /* This struct has everything that I don’t read in from the command line, but still want all threads to know */ struct nbodyGlobalStruct { double dRedshift; double dExpansionFactor; … } /* This struct has everything that I was store store locally in each compute thread. */ struct nbodyLocalStruct { int nParticleInteractions; int nCellInteractions; … }

18. Using Ntropy (example: N-body)Computational Steering Layer (nbody.c) Main N-body loop: nbody->iStep = 0; nbody->iOut = 0; while(nbody->iStep==0 || nbody->iOut < nbodySteps(&nbody)) { /* Build the tree */ ntropy_BuildTree(ntropy, (*tskCellAccum), (*tskBucketAccum)); /* Do the Gravity walk using dynamic load balancing */ ntropy_Dynamic(ntropy, (*tskGravityInit), (*tskGravityFinish), (*tskGravity)); /* Or, if you don’t want dynamic load balancing for this function, use ntropy_Static() */ /* Write results if needed (nbodyWriteOutput checks if it needs to output. If so, it calls ntropy_WriteParticles(), then increments msr->iOut.) */ nbodyWriteOutput(ntropy, &nbody); ++(msr->iStep); } /* Finish */ ntropy_Finish(ntropy);

19. Using Ntropy (example: N-body)Task Layer (nbody_task.c) void tskGravity(NTPntp, NS *pNodeSpecStart) { /* Get all the structs that I will need */ struct nbodyParamStruct *param = ntp_GetParam(ntp); struct nbodyGlobalStruct *global = ntp_GetGlobal(ntp); struct nbodyLocalStruct *local = ntp_GetLocal(ntp); NS nodeSpecBucket; /* Handle for the current bucket node */ NS nodePtrBucket; /* Pointer to the data of the current bucket */ NS nodeSpecDone; /* The handle of the node that is “next” for nodeSpecStart. * When we walk to this node, we will be done. */ ntpNS_Next(ntp, pNodeSpecStart, &nodeSpecDone); ntpNS_Copy(ntp, pNodeSpecStart, &nodeSpecBucket); /*Find each bucket that is in pNodeSpecStart’s domain and do a treewalk for it*/ while(ntpNS_Compare(ntp, nodeSpecBucket, nodeSpecDone) { AQCUIRE_NODEPTR(ntp, nodePtr, nodeSpec); while(ntpNP_Type(ntp, nodePtr) != NTP_NODE_BUCKET) { ntpNS_ReplaceLower(ntp, &nodeSpec, nodePtr); ACQUIRE_NODEPTR(ntp, nodePtr, nodeSpec); } /* Now nodeSpecBucket and nodePtrBucket are pointing at the next bucket to process. */ MyWalkBucket(ntp, nodeSpecBucket, nodePtrBucket); } }

20. Using Ntropy (example: N-body)Task Layer (nbody_task.c) void MyWalkBucket(NTPntp, NS nodeSpecBucket, NP nodePtrBucket) { NS nodeSpec; /* Handle for the node currently being looked at */ NP nodePtr; /* Pointer to the data of the node currently being looked at */ ntpNS_PstStart(ntp, &nodeSpec); /* Get the handle of the root of the PST */ while (iWalkResult != NTP_NULL_PST) { ACQUIRE_NODEPTR(ntp, nodePtr, nodeSpec); iInteractResult = MyCellCellInteract(ntp, nodeSpecBucket, nodePtrBucket, nodePtr, nodeSpec); if (iInteractResult == MY_CELL_OPEN) iWalkResult = ntpNS_ReplaceLower(ntp, &nodeSpec, nodePtr); else /* iInteractResult == MY_CELL_NEXT */ iWalkResult = ntpNS_ReplaceNext(ntp, &nodeSpec, nodePtr); } }

21. N Tropy Functional Layout Executed on master thread only Application Steering script ntropy_* functions NTropy Computational Steering API Key: Layers completely rewritten PKDGRAV Parallel Client/Server Layer Layers retained from PKDGRAV “User” Supplied Methods Thread services layer (executed on all threads) Application “Tasks” (serial code) Tree node and particle navigation methods: (e.g. Acquire node/particle, next node/particle, parent node, child node ntp_* functions MDL Interprocessor communication layer Executed on all threads

22. Further NTropy Features • Custom cell and particle data • Selectable cache types • Read-only • Combiner • Explicit control over cache starting and stopping • Cache statistics • 10 user-selectable timers and 4 “automatic” timers

23. Further NTropy Features • Custom command-line and parameter file options • Automatic reduction variables • Range of collectives (AllGather, AllToAll, AllReduce) • I/O Primitives (TIPSY “array” and “vector” files) as well as flexible user-defined I/O.

24. Conclusions • Most data analysis in astronomy is done using trees as the fundamental data structure. • Most operations on these tree structures are functionally identical. • Based on our studies so far, it appears feasible to construct a general purpose parallel framework that users can rapidly customize to their needs.