computer systems principles concurrency patterns n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Computer Systems Principles Concurrency Patterns PowerPoint Presentation
Download Presentation
Computer Systems Principles Concurrency Patterns

Loading in 2 Seconds...

play fullscreen
1 / 47
gavrilla-rios

Computer Systems Principles Concurrency Patterns - PowerPoint PPT Presentation

86 Views
Download Presentation
Computer Systems Principles Concurrency Patterns
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Computer Systems PrinciplesConcurrency Patterns Emery Berger and Mark Corner University of Massachusetts Amherst

  2. Web Server webserver • Client (browser) • Requests HTML, images • Server • Caches requests • Sends to client not found http://server/Easter-bunny/200x100/75.jpg client

  3. Possible Implementation while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; }

  4. Possible Implementation while (true) { wait for connection; // net read from socket & parse URL; // cpu look up URL contents in cache; // cpu if (!in cache) { fetch from disk / execute CGI;//disk put in cache; // cpu } send data to client; // net }

  5. Problem: Concurrency webserver • Sequential fine until: • More clients • Bigger server • Multicores, multiprocessors • Goals: • Hide latency of I/O • Don’t keep clients waiting • Improve throughput • Serve up more pages clients

  6. Building Concurrent Apps • Patterns / Architectures • Thread pools • Producer-consumer • “Bag of tasks” • Worker threads (work stealing) • Goals: • Minimize latency • Maximize parallelism • Keep progs. simple to program & maintain

  7. Thread Pools • Thread creation relatively expensive • Instead: use pool of threads • When new task arrives, get thread from pool to work on it; block if pool empty • Faster with many tasks • Limits max threads (thus resources) • ( ThreadPoolExecutor class in Java)

  8. Producer-Consumer • Can get pipeline parallelism: • One thread (producer) does work • E.g., I/O • and hands it off to other thread (consumer) producer consumer

  9. Producer-Consumer • Can get pipeline parallelism: • One thread (producer) does work • E.g., I/O • and hands it off to other thread (consumer) producer consumer

  10. Producer-Consumer • Can get pipeline parallelism: • One thread (producer) does work • E.g., I/O • and hands it off to other thread (consumer) producer consumer

  11. Producer-Consumer • Can get pipeline parallelism: • One thread (producer) does work • E.g., I/O • and hands it off to other thread (consumer) producer consumer

  12. Producer-Consumer • Can get pipeline parallelism: • One thread (producer) does work • E.g., I/O • and hands it off to other thread (consumer) producer consumer LinkedBlockingQueueBlocks on put() if full, poll() if empty

  13. Producer-Consumer Web Server • Use 2 threads: producer & consumer • queue.put(x) and x = queue.poll(); while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } while (true) { do something… queue.put (x); } while (true) { x = queue.poll(); do something… }

  14. Producer-Consumer Web Server • Pair of threads – one reads, one writes while (true) { wait for connection; read from socket & parse URL; queue.put (URL); } while (true) { URL = queue.poll(); look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; }

  15. Producer-Consumer Web Server • More parallelism –optimizes common case (cache hit) while (true) { wait for connection; read from socket & parse URL; queue1.put (URL); } while (true) { URL = queue1.poll(); look up URL contents in cache; if (!in cache) { queue2.put (URL); return; } send data to client; } 1 2 while (true) { URL = queue2.poll(); fetch from disk / execute CGI; put in cache; send data to client; }

  16. When to Use Producer-Consumer • Works well for pairs of threads • Best if producer & consumer are symmetric • Proceed roughly at same rate • Order of operations matters • Not as good for • Many threads • Order doesn’t matter • Different rates of progress

  17. Producer-Consumer Web Server • Should balance load across threads while (true) { wait for connection; read from socket & parse URL; queue1.put (URL); } while (true) { URL = queue1.poll(); look up URL contents in cache; if (!in cache) { queue2.put (URL); } send data to client; } 1 2 while (true) { URL = queue2.poll(); fetch from disk / execute CGI; put in cache; send data to client; }

  18. Bag of Tasks • Collection of mostly independent tasks worker worker worker worker

  19. Bag of Tasks • Collection of mostly independent tasks worker worker worker worker

  20. Bag of Tasks • Collection of mostly independent tasks worker worker worker worker

  21. Bag of Tasks • Collection of mostly independent tasks worker worker worker worker

  22. Bag of Tasks • Collection of mostly independent tasks worker worker worker worker

  23. Bag of Tasks • Collection of mostly independent tasks worker worker worker worker

  24. Bag of Tasks • Collection of mostly independent tasks • Bag could also be LinkedBlockingQueue(put, poll) addWork worker worker worker worker

  25. Exercise: Restructure into BOT • Re-structure this into bag of tasks: • addWork & worker threads • t = bag.poll() or bag.put(t) while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; }

  26. Exercise: Restructure into BOT • Re-structure this into bag of tasks: • addWork & worker • t = bag.poll() or bag.put(t) addWork: while (true) { wait for connection; t.URL = URL; t.sock = socket; bag.put (t); } Worker: while (true) { t = bag.poll(); look up t.URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client via t.sock; }

  27. Bag of Tasks Web Server • Re-structure this into bag of tasks: • t = bag.poll() or bag.put(t) addWork: while (true){ wait for connection; bag.put (URL); } worker addWork worker: while (true) { URL = bag.poll(); look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } worker worker

  28. Bag of Tasks vs. Prod/Consumer • Exploits more parallelism • Even with coarse-grained threads • Don’t have to break up tasks too finely • What does task size affect? • possibly latency… smaller might be better • Easy to change or add new functionality • But: one major performance problem…

  29. What’s the Problem? addWork worker worker worker worker

  30. What’s the Problem? • Contention – single lock on structure • Bottleneck to scalability addWork worker worker worker worker

  31. Work Queues • Each thread has own work queue (deque) • No single point of contention • Threads now generic “executors” • Tasks (balls): blue = parse, yellow = connect… executor executor executor executor

  32. Work Queues • Each thread has own work queue (deque) • No single point of contention executor executor executor executor

  33. Work Queues • Each thread has own work queue (deque) • No single point of contention executor executor executor executor

  34. Work Queues • Each thread has own work queue (deque) • No single point of contention executor executor executor executor

  35. Work Queues • Each thread has own work queue • No single point of contention • Now what? executor executor executor executor

  36. Work Stealing • When thread runs out of work,steal work from random other thread worker worker worker worker

  37. Work Stealing • When thread runs out of work,steal work from top of random deque • Optimal load balancing algorithm worker worker worker worker

  38. Work Stealing Web Server • Re-structure:readURL, lookUp, addToCache, output • myQueue.put(new readURL (url)) while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; }

  39. while (true) { wait for connection; read from socket & parse URL; look up URL contents in cache; if (!in cache) { fetch from disk / execute CGI; put in cache; } send data to client; } readURL, lookUp, addToCache, output class Work {public: virtual void run();}; class readURL : public Work {public: void run() {…} readURL (socket s) { …}};

  40. readURL lookUp addToCache worker output

  41. class readURL {public: void run() { read from socket, f = get file myQueue.put (new lookUp(_s, f)); } readURL(socket s) { _s = s; }};

  42. class lookUp {public: void run() { look in cache for file “f” if (!found) myQueue.put (new addToCache(_f)); else myQueue.put (new Output(s, cont)); } lookUp (socket s, string f) { _s = s; _f = f; }};

  43. class addToCache {public: void run() { fetch file f from disk into cont add file to cache (hashmap) myQueue.put (new Output(s, cont)); }

  44. Work Stealing Web Server • Re-structure:readURL, lookUp, addToCache, output • myQueue.put(new readURL (url)) readURL(url) { wait for connection; read from socket & parse URL; myQueue.put (new lookUp (URL)); }

  45. Work Stealing Web Server • Re-structure:readURL, lookUp, addToCache, output • myQueue.put(new readURL (url)) readURL(url) { wait for connection; read from socket & parse URL; myQueue.put (new lookUp (URL)); } lookUp(url) { look up URL contents in cache; if (!in cache) { myQueue.put (new addToCache (URL)); } else { myQueue.put (new output(contents)); } } addToCache(URL) { fetch from disk / execute CGI; put in cache; myQueue.put (new output(contents)); }

  46. Work Stealing • Works great for heterogeneous tasks • Convert addWork and worker into units of work (different colors) • Flexible: can easily re-define tasks • Coarse, fine-grained, anything in-between • Automatic load balancing • Separates thread logic from functionality • Popular model for structuring servers

  47. The End