Lessons for High Performance Services or Towards Transaction Processing Engineering

1. Lessons for High Performance ServicesorTowards Transaction Processing Engineering Dr. Alfred Z. Spector azs@azs-service.com Views herein are my own, and do not represent any other organization.

2. Towards Transaction Processing EngineeringAbstract • Transaction processing as a field of engineering and science has a 50-year long record: Thus, due consideration of the engineering lessons of systems from Saber on (and on and on) and the academic work, say, on atomicity or replication, should now be yielding high quality, non-speculative design principles. Perhaps some elements would be marked as understood and codified; some as explored, but not yet fully understood; and some as emerging and still unexplored. But, there would at least be the beginnings of an unambiguous, engineering backbone to the field. • But, the reality is that we have many too many unpooled experiences and too many proposed techniques, structures, and algorithms on which no critical consensus has been reached. For example, I have my own opinions on the value of our teams' work on distributed transaction management (done 10-20 years ago at Carnegie Mellon and Transarc), but I've yet to write (or see) a broad critical analysis that attempts to put the relevant topics conclusively to bed. There are many more examples like this. • My motivation for desiring to begin this classification is a feeling that our field (and computer science as a whole) does not sufficiently and clearly document what we know. We are not good at reducing large aspects of software/systems project to a more traditional type of engineering. Due to the breadth of my recent management experiences, I recognize I may lack qualifications to contribute details to this effort, but indeed no one can see the whole elephant. Perhaps, the right approach is to come up with sharedpublication mechanisms (and needed structural elements) that can leverage the great successes of social networking technologies to harness the collective wisdom of the community. • I will present this critique, illustrate it with categorized examples (some of which may generate debate), and propose some conceptual structures that could lead to a better codification of the science and engineering principles of transaction processing.

3. Outline • Our field is old, grand, and of the essence • Anecdotal problems • Deep Issues in TP • From “Towards a Software Science of Design1” • A Step for TP: The codification of what we know • Conclusions 1 National Science Foundation Science of Design PI Mtg., Arlington, VA, 3/07, Towards a Software Science of Design

4. The Grand Ole Field • Transaction Processing, or “service-oriented computing”, is ~50 years old: • Highly successful through many waves: • The few and the big (1st decade) • Growth of the many (2nd two decades) • Distributed systems (4rd decade) • Web support, universal access, & massive utilities (5th decade) • Powered by: • Practical engineering by computer industry and users • Substantial researcher interest • A great and balanced community (!) • A Strong Future: • So much more than 6 billion/(ThinkTimeInSecs) TPS • Goal: To minimize overhead, support optimization of all systems, and provide universal aggregation and access of all knowledge • We envision highly interconnected systems making life better • However, with substantial design and engineering challenges

5. Our Inventory of Technology Supporting the Agenda Source: Table 2.1, Current-Cost Net Stock of Private, Commercial Fixed Assets, Equipment and Software, National Bureau of Economic Analysis • I/T (particularly S/W) a Growing Share of U.S. Commercial Capital Stock1 • Note: Global I/T expenditure per year is roughly $1.5 x 10^12 • My analysis: Growth in I/T value is probably much steeper; why? • In terms of value, I/T has grown much faster due to price deflation • Software depreciation/expensing is demonstrably fast 1and by implication, the World’s.

6. We Should and We Do Know a Lot, but… • We make huge, reliable systems with great brilliance • We install small systems to run stores with minimal fuss • Our gurus are indeed gurus • But, • Why are the technical leaders so overwhelming key to major, new systems design and implementation? • Why does (or can) the industry develop many system with minimal advances, but significant incompatibilities? • That is, why are there so many different ways to do the same thing? Why do we regularly re-invent the wheel? • Why is it virtually impossible to get the details right? • Where is the regularity in system design? • Why are achieving reuse, failure analysis, security, etc. either so difficult or requiring of genius?

7. Do We Truly Understand the Deep Issues in TP (1)? • The Data Abstraction Layer: • Relational, XML, Text w/arbitrary annotation, Chunks, Posix • … • The Service Runtime: • Apache and load balancing • Heavier-weight transaction a la J2EE, or Microsoft .Net • Stored procedures • … • Security and Privacy Architecture: • Confidentiality • Access control: data layer, service layer, … • Availability Architecture: • Replication techniques • Problem determination: • Architecture and techniques • Service Architecture with embedded services • Balancing encapsulation and transparency

8. Deep Issues in TP (2) • Reuse • How to support and reduce reinvention: e.g., O.O., vs. Open Source, vs… • Huge opportunities for engineering benefits across society • The limits to atomicity • Where, when, and how much? • Disaster recovery • Mirroring vs. event transmission/logging • Systems management • Lower cost management • Design for evolution • Risk analysis • Performance Analysis • Software Engineering/Project management • And many more…

9. Score high on many bad metrics: Amount of code # of dependencies # of programmatic interfaces # of layers Administrative interface size & configuration options Non-uniformity Non-orthogonality Defects Documentation # of programmers involved Added: dynamic change Problems in TP Design are part of the Software Design Problem (from 2004 talk I gave at MIT1) More generally, we have a software design problem 1From The Conudrum of Systems, MIT, 2004, http://www.csail.mit.edu/events/DLStalks/dlsspector03.html

10. Stepping Back: Design, Broadly1 • The process of design refers to the application of synthetic and analytic processes to plan and create new objects. • My perspective – great commonality in design across many human endeavors • Protocols • (Medical) Differential Diagnoses • Bridge/Building Design2 • Proofs • Plays, Music, Novels • Urban Areas • Political Systems • Complex Systems • While design is central, it is not today a holistic discipline, studied in the same ways as the classical humanities or even science/engineering. Should it be? • Cross-fertilization to be expected: • AASHTO specifications for requirements of highway bridges • Design workbenches • Project design • Software design is often a part of broader designs so design is often inextricably linked 1 National Science Foundation Science of Design PI Mtg., Arlington, VA, 3/07, Towards a Software Science of Design 2 Bridge Design and Construction, CACM 29(4): 267-283, 4/86, A. Z. Spector and D.K. Gifford

11. Difficulty of Software Design • Diversity of objectives and scale • Quantity of artifacts • Growing scale even for individual artificats • Post facto, dynamic assembly • Malleability • Longevity • Absence of manufacturing cost • Changing development possibilities • Difficulty of verification

12. A First Step in TP: The Codification of What We Know • Social Networking approaches to knowledge capture, organization, and refinement have been more successful than many thought: • Wikipedia is an obvious example • Linux is another • Other domain specific social networking sites • Grand Question: • Could we devise a going-in structure for our field that organized requirements, algorithms, etc…along various dimensions (Recursive application of technology)? • Could this be debated and refined over time? • Could we then provide answers • Facts and Proofs • Designs • Software templates • Code • Some tagging for elements: some would be marked as understood and codified; some as explored, but not yet fully understood; and some as emerging and still unexplored • Could we use social networking technologies based on our systems and using our community to put Transaction Processing Engineering on a firm foundation?

13. Do We Need A TP Knowledge Repository (TPKR)? • Yes, here’s why: • As a field, we don’t get to the bottom of things very well: • Industry has challenges: • The good (or bad) old thing • The new, new thing • Academe has challenges: • Oft., lack of detailed knowledge on big systems • The new, new thing (or LPU) • The time to really answer hard problems • The whole field is not really disciplined towards achieving design principles • TPKR could motivate: • More discourse and information exchange • Challenges and concomitant debate on important design issues • A publication vehicle, more in keeping with the times than the Journal • Contributions to the compendium of definitive information would be judged highly • Perhaps, more so than individual journal/conference articles • The social phenomena might unleash a lot of energy • I note that Jim Gray was very interested in alternative forms of publication (e.g., as compared to traditional journals), in my last conversation with him

14. Towards a Software-wide Science of Design A Software Science of Design would facilitate software production across the lifecycle by helping to understand, codify and integrate currently understood approaches to design, but also by creating better approaches and tools. • A broad, integrative mission • Key foci: • Cross-fertilization with other design disciplines • Holistic approach • Scientific analysis (not, “here’s a new approach, it’s real cool ”) • Education and standardization of approaches • Interdisciplinary work • Elements probably not included • Domain-specific abstractions and their implementations • Elements directly covered in other areas of the field: • Programming Language Design • Verification • Etc. • These are admittedly blurry lines • Note: • I don’t think that S/W design, broadly, will admit to as much standardization as exists in, say, road design • Nonetheless, there can still be families of approaches, parameterized (or meta-), approaches & tools…

15. Elements of a Research Agenda • Establishing Convincing Truths • Terminology and reference models • Metrics • Modularity and reuse • Workbenches/platforms • Development methodologies • Automated assembly • Design for change • Design for security • Complexity • Education • Team culture • Impact of accounting and taxation • Impact of liability, intellectual property, and anti-trust law

16. Observations and Conclusions • The state of software design is not sound. (1) We don’t know enough (2) we rarely codify (in the broadest sense) what we know (3) we don’t consistently use or educate what we codify. • Design is everywhere. While particularly important in software, there should be a cross-discipline design community looking for synergies in innovation, education, and practice. • Design in software is multi-faceted. It applies to all aspects of the lifecycle. A Science of Design should be broad enough to address all aspects. • Design is of growing importance. Related not only to amount of software, more serious applications of software, greater diversity of it’s development across time and space, and post-facto, dynamic assembly, with a greater base of components, design (but not programming) is oft. needed! Implications are manifold. • The research agenda has great opportunity. There are plenty of topics for a broad research agenda. Work should proceed towards categorization and completeness: tops down and bottoms up. • Interdisciplinary research is of particular important. The design of software is not purely a problem of computer science. • Establishment of objectives. We should establish quantifiable objectives. • And we could do a big thing in our subfield: TPKR

17. Thank you!