LAC Coding Seminar

1. LAC Coding Seminar Lecture 7 March 15, 2007

2. Parallel Programming

3. 3 The Evolution of Programming Paradigm 60's – 70's: Computers can handle longer and more complicated software than assembly languages can support Emergence of high level programming languages: FORTRAN, C 80's – 90's: Software grows to millions of lines and need new programming languages to handle composibility and maintainability; High performance is not an issue any more because the hardware are faster. OOP: C++, Java Software development tools 2005 – 20??: Performance of single processor reaches a bottleneck and multi-core is a new trend. Sequential program also reaches the performance limit. Datasets is also larger thanks to inexpensive disk storage and network bandwidth.

4. 4 Multi-core Processors Intel Core 2 Duo Intel Core 2 Quad AMD Opteron Dual Core STI Cell

5. 5 Multi-processor & Multi-core Model Shared memory Core2 Duo, Quad, Opteron due core Shared network or distributed memory Each processor/core has its own memory space, and cannot access memory of others. Cell

6. 6 Parallel Programming Data Parallel Processor 1: exec DATA[1] Processor 2: exec DATA[2] Pipeline Data source is in streaming style, Processor 1 exec Job 1, Processor 2 exec Job 2 Data and job dependency Synchronization For the same data segment, Job 2 must not started before Job 1 finishes Both processors need to access and update same data

7. 7 Synchronization Race condition Deadlock Starvation Mutex Semaphore Critical section Signal

8. 8 Patterns of Parallel Programming Task decomposition Data decomposition Dependency analysis

9. 9 Parallel Programming Distributed & cluster environment Socket, MPI, PVM Multi-process Multi-threaded

10. Multi-threading Programming with Pthreads

11. 11 Multi-process vs. multi-thread Process vs. thread Process owns its own address space and system resources. Thread is an abstract data type representing flow of control within a process Threads share address space, share system resources (files, I/O etc), but have separate Status, execution stack, context: registers. Multi-process vs Multi-threaded programming Threads are cheaper to create than processes and switch between. Easier and faster to shared data. However, less robust against programming errors.

12. 12 Start to Use Pthreads Pthreads = POSIX Threads Header file #include <pthread.h> Library -lpthread

13. 13 Thread Creation and Termination pthread_t thread; int pthread_create(pthread_t * thread, pthread_attr_t * attr, void * (*start_routine)(void *), void * arg); void pthread_exit(void *retval);

14. 14 Example: Create a Thread

15. 15 Example

16. 16 Joining and Detaching Threads int pthread_join(pthread_t th, void **thread_return); Synchronize the thread execution int pthread_detach(pthread_t th); Make sure the system resources allocated for the thread will be released

17. 17 Example: Thread Joining and Detaching

18. 18 Mutex pthread_mutex_t mutex; int pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *mutexattr); int pthread_mutex_destroy(pthread_mutex_t *mutex); int pthread_mutex_lock(pthread_mutex_t *mutex); int pthread_mutex_trylock(pthread_mutex_t *mutex); int pthread_mutex_unlock(pthread_mutex_t *mutex);

19. 19 Example: Mutex

20. 20 Condition/Signal pthread_cond_t cond; int pthread_cond_init(pthread_cond_t *cond, pthread_condattr_t *cond_attr); int pthread_cond_destroy(pthread_cond_t *cond); int pthread_cond_signal(pthread_cond_t *cond); int pthread_cond_broadcast(pthread_cond_t *cond); int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex); int pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, const struct timespec *abstime);

21. 21 Example: Condition

22. 22 Thread Specific Data pthread_key_t key int pthread_key_create(pthread_key_t *key, void (*destr_function) (void *)); int pthread_key_delete(pthread_key_t key); int pthread_setspecific(pthread_key_t key, const void *pointer); void * pthread_getspecific(pthread_key_t key);

23. 23 Example: TSD

24. Debugging Multithreaded Program

25. 25 Debugging Multithreaded Program Same old bugs, plus synchronization issues gdb Info thread thread id Core dump Use static debugging method Code review Code analysis

26. 26 Timing & Syncronization When you add code to debug your program the bug disappears! Bugs that happen on this machine do not happen on that one. Most subtle bugs are from data racing Check all shared data Data used in this thread may be changed in another thread Others… Check communication link: data are correctly sent and received. Check deadlock "volatile" keyword and optimization

27. 27 Performance is too low? Granularity issues Overhead of threads Unbalanced computation