CSC407: Software Architecture Winter 2007 Pipes and Filters

CSC407: Software ArchitectureWinter 2007Pipes and Filters Greg Wilson BA 4234 gvwilson@cs.utoronto.ca

Overview • Some things are best learned by example • So let's look at some software architectures • First target: pipes and filters • One of the reasons for Unix's success and longevity • The world's first (and simplest) component object model

High-Level View • Want to be able to do complex data processing by composing simple operations in flexible ways • More specifically: • Data to process is a stream of records (possibly of infinite length) • Processing is done by filters, whose only interactions are the effects they have on the data streams • Restrict model to directed acyclic graph (DAG)

System Components • Filters encapsulate operations • Pipes connect filters • Sources provide data • Sinks consume data source sink in.txt grep sort out.txt filter pipe

A More Elaborate Example program text semantic checker bytecode generator scanner parser program • We need a join as well • …which implies a tee + + + + error stream

…A More Elaborate Example in.txt cat tee sort comm out.txt mknod pipeA p mknod pipeB p sort pipeA > pipeB & \ cat in.txt | tee pipeA | sort –u | \ comm –13 pipeB > out.txt sort

Going All The Way

CRC Cards Class: pipe Collaborators: Source Sink Filter Class: filter Collaborators: Pipe • Responsibility • Transfer data • Buffer data • Synchronize • Responsibility • Gets input data • Transforms data • Supplies output data Class: sink Collaborators: Pipe Class: source Collaborators: Pipe • Responsibility • Consume data • Responsibility • Deliver data

Who's Driving? • Option #1: active filters push data into pipes • But what about sinks? • Option #2: passive filters are driven by pipes • But what about sources? • Option #3: a controller manages the data flow • Note: have not yet considered: • Whether processing runs in parallel or not • Whether components can be distributed

Active Filters • Filter is a loop that pulls input from one pipe and pushes output into another • E.g., stdin and stdout by default while true: data = inputPipe.read(…) data = transform(data) outputPipe.write(data)

Passive Filters (Pull) source filter 1 filter 2 sink • The push strategy is the converse

Controllers • There are no "pipes" per se • Just call filters in a loop • Problems with concurrency and composability • But it makes it a lot easier to pass objects… • …and avoids data-copying performance problems while true: data = input.read(…) for f in filters: data = f(data) output.write(data)

Design Questions • Data format? • Just one, or many? • How to signal end-of-stream? • How to implement pipe/filter connections? • How to implement error handling? • Differentiate between stream and batch processing? • I.e., between filtering and sorting?

Active Expressions • What does a|b|c mean? • Bitwise or • Pipe • Whatever it has been overloaded to mean • What does a|(20*b)|c mean? • It’s a task farm • Tempting to let people “type in parallelism” • But what about grids? And trees?

Back to the Future • Microsoft PowerShell (formerly Monad) passes objects instead of strings • "Revolution" is not too strong a word • Shame about the syntax, though… $ get-process | where {$_.handles -gt 500} | sort {$_.handles} Handles NPM(K) PM(K) WS(K) VS(M) CPU(s) Id ProcessName ------- ------ ----- ----- ----- ------ ---- ---------- 516 13 1676 1476 23 68.6 1088 svchost 546 10 4464 2760 44 7772.7 832 lsass 582 24 27148 7708 129 1777.3 1440 msnmsgr 599 60 7868 4644 56 110.3 776 winlogon

Take a Few Minutes… • …and think about the architecture of PowerShell • Kruchten’s views once again • What if we want to be able to distribute the processing?

CSC407: Software Architecture Winter 2007 Pipes and Filters