Parallelizing Incremental Bayesian Segmentation (IBS)

1. Parallelizing Incremental Bayesian Segmentation (IBS) Joseph Hastings Sid Sen

2. Outline • Background on IBS • Code Overview • Parallelization Methods (Cilk, MPI) • Cilk Version • MPI Version • Summary of Results • Final Comments

3. Background on IBS

4. IBS • Incremental Bayesian Segmentation [1] is an on-line machine learning algorithm designed to segment time-series data into a set of distinct clusters • It models the time-series as the concatenation of processes, each generated by a distinct Markov chain, and attempts to find the most-likely break points between the processes

5. Training Process • During the training phase of the algorithm, IBS builds a set of Markov matrices that it believes are most likely to describe the set of processes responsible for generating the time series

6. Code Overview

7. Hi-Level Control Flow • main() • loops through input file • Runs break-point detection • Foreach segment: • check_out_process() • Foreach existing matrix • compute_subsumed_marginal_likelihood() • Adds segment to set of matrices or subsumes

8. Parallelizable Computation • compute_subsumed_marginal_likelihood() • Depends on a single matrix and the new segment • Produces a single score • The index of the best score must be calculated

9. Code Status

10. Parallelization Methods

11. MPI • Library facilitating inter-process communication • Provides useful communication routines, particularly MPI_Allreduce, which simultaneously reduces data on all nodes and broadcasts the result

12. Cilk • Originally developed by the Supercomputing Technologies Group at the MIT Laboratory for Computer Science • Cilk is a language for multithreaded parallel programming based on ANSI C that is very effective for exploiting highly asynchronous parallelism [3] (which can be difficult to write using message-passing interfaces like MPI)

13. Cilk • Specify number of worker threads or “processors” to create when running a Cilk job • No one-to-one mapping of worker threads to processors, hence the quotes • Work-stealing algorithm • When a processor runs out of work, asks another processor chosen at random for work to do • Cilk’s work-stealing scheduler executes any Cilk computation in nearly optimal time • Computation on P processors executed in time Tp≤ T1/P + O(T)

14. Code Status

15. Cilk Version

16. Code Modifications • Keywords: cilk, spawn, sync • Convert any methods that will be spawned or that will spawn other (Cilk) methods into Cilk methods • In our case: main(), check_out_process(), compute_subsumed_marginal_likelihood() • Main source of parallelism comes from subsuming current process with each existing process and choosing subsumption with the best score • spawn compute_subsumed_marginal_likelihood(proc, get(processes,i), copy_process_list(processes));

17. Code Modifications • When updating global_score need to enforce mutual exclusion between worker threads • Cilk_lockvar score_lock; ... Cilk_lock(score_lock); ... Cilk_unlock(score_lock);

18. Cilk Results Optimal performance achieved using 2 processors (trade-off between overhead of Cilk and parallelism of program)

19. Adaptive Parallelism • Real intelligence is in the Cilk runtime system, which handles load balancing, paging, and communication protocols between running worker threads • Currently have to specify the number of processors to run a Cilk job on • Goal is to eventually make the runtime system adaptively parallel by intelligently determining how many threads/processors to use • Fair and efficient allocation among all running Cilk jobs • Cilk Macroscheduler [4] uses steal rate of worker thread as a measure of its processor desire (if a Cilk job spends a substantial amount of its time stealing, then the job has more processors than it desires)

20. MPI Version

21. Code Modifications • check_out_process() first broadcasts the segment using MPI_Bcast() • Each process loops over all matrices, but only performs subsumption if (I % np == rank) • Each process computes best score, and MPI_Allreduce() is used to reduce this information to the globally best score • Each process learns the index of the best matrix and performs the identical subsumption

22. MPI Results Big improvement from 1 to 2 processors; levels off for 3 or more

23. Summary of Results

24. MPI vs. Cilk

25. Final Comments

26. MPI vs. Cilk • MPI version much more complicated, involved more lines of code, and much more difficult to debug • Cilk version required thinking about mutual-exclusion, which MPI avoids • Cilk version required few code changes, but conceptually more complicated to think about

27. References (Presentation) • [1] Paola Sebastiani and Marco Ramoni. Incremental Bayesian Segmentation of Categorical Temporal Data. 2000. • [2] Wenke Lee and Salvatore J. Stolfo. Data Mining Approaches for Intrusion Detection. 1998. • [3] Cilk 5.3.2 Reference Manual. Supercomputing Technologies Group, MIT Lab for Computer Science. November 9, 2001. Available online: http://supertech.lcs.mit.edu/manual-5.3.2.pdf. • [4] R. D. Blumofe, C. E. Leiserson, B. Song. Automatic Processor Allocation for Work-Stealing Jobs. (Work in progress)