The Software Matrix Simplifying Software Salvage

1. The Software MatrixSimplifying Software Salvage RiddhimanGhosh Advisor: Dr. James Fawcett

2. Introduction • Approximately 100 billion lines of source code at work in the world today. • Large fractions of code in systems are functionally equivalent. • However systems are often constructed without leveraging existing code bases – less than 15% of new code serves an original purpose. • “Reinvention of the wheel” in the software industry.

3. Introduction • Software construction is needlessly error-prone and expensive. • We’re maintaining multiple copies of essentially the same software. • Time and cost of developing, testing and documenting a piece of software is multiplied by the number of equivalent copies in existence.

4. Software Reuse • Study of reuse – recycling of software assets – has been an important branch in the software engineering discipline. • Effective reuse promises • Reduced development and maintenance costs • Gains in development schedule and quicker time-to-market • Increased robustness and quality • Not a new idea.

5. Software Reuse • Systematic Reuse: “an institutionalized organizational approach to product development in which software assets are intentionally created or acquired to be reusable” • However, few organizations practice systematic reuse, in spite of a recognition of its potential benefits. • Reuse is hard!

6. Gap between “theory” and “practice” in Reuse “ Not all the “theoretical oriented” and “sophisticated” solutions presented by researcher[s] have ...thrilled the practitioners... – Zand, M. ...the reuse community has worked on complex technologies and methods with high ceremony, yet most of the software community seems to be looking for simpler solutions... High ceremony methods require an organization with high process maturity to achieve success. – Griss, M. ”

7. Software Salvage • What we seen in the industry is not software reuse, but rather “software salvage” • Salvage: lifting of a significant block of existing systems and inserting them into a newly developed system. • Radar Systems Department, General Electric Company (Syracuse, NY) routinely attempted salvage in the building of a new radar.

8. Reuse vs. Salvage • We make a distinction between the terms software reuse and software salvage. • Reuse connotes immutability • the individual pieces meant for reuse cannot be modified; they’re designed and implemented to be adaptable but with no intent to change even a single character of source code. • Salvage makes no guarantee of immutability • very often source code of individual pieces being salvaged is modified.

9. Problem • Effective salvage is difficult to accomplish • The large pieces we want to recycle often have many dependencies on parts we don’t want. • These dependencies usually require expensive changes to the part being salvaged. • Pulling out pieces from different systems, and getting these extricated pieces to work together is challenging. • Think of salvage as analogous to transplanting the heart from one living organism to another! Similar problems, when we pull out a part from a software system due to the connectedness inherent in typical software.

10. Problem It is difficult to salvage existing parts systems and build new systems from them, with ease.

11. Goal • Enable leveraging of major parts of systems • Eliminate or reduce changes required by salvage so that salvage moves closer to the reuse model (immutability), without requiring high-process organization. • Simplify software salvage to make it a useful paradigm.

12. Prior Approaches • Two major (and fairly recent) approaches towards encouraging reuse have been • object-oriented reuse • component-oriented reuse.

13. Object-Oriented Reuse • Reuse has been one of the classic motivations of object-orientation • Object oriented theory • mandated discipline in writing code through best-practice guidelines such as separating interface from implementation • Encouraged reuse through inheritance, composition, parameterization

14. Object-Oriented Reuse • However OO technologies didn’t quite engender the reuse revolution that was hoped for. • Perhaps the only OO libraries widely used user-interface frameworks (such as MFC) and libraries for data structures. • While OO techniques have made compiler libraries an effective means of reuse, business organizations have had a harder time getting leverage from their pre-existing software assets only through the creation and consumption of OO libraries.

15. Object-Oriented Reuse • The definition of an object is purely “technical” • Defined as an encapsulation of state and behavior • No direct mapping to the physical unit that is actually deployed, versioned and potentially reused. • Objects don’t fit the “salvage” model well • In terms of granularity, they are of an inappropriate size to be mixed and matched. • They are the size of grains of sand, while what salvagers are typically looking for in building systems, are bricks.

16. Component-Oriented Systems • The logical next step was the notion of components • a coherent package of software implementation that can be independently deployed and composed with other components. • of coarser granularity than objects • independent and deployable implies executable or loadable code • The defining feature of component technology is that a system, which is composed of components, can be repaired or made better by updating components, without rebuilding the whole system.

17. Component-Oriented Systems • There are several component technology offerings • COM, CORBA, JavaBeans/EJB, .NET Components • Adopting a component-oriented approach (rather than only concentrating on programming language as in OO theory) is beneficial. • But the existence of these component technologies has not solved the problems that plague software salvage, discussed earlier • Packaging techniques alone are not enough

18. Our Approach • We are of the view that components are useful only if there is a framework that actively supports and promotes their reuse. • .NET and J2EE have framework support, however it is focused on generic industry problems (network communication, web publishing, application security, etc.) • We seek to support vertical, perhaps proprietary, applications, where building a huge framework is impractical from a return on investment point of view.

19. Our Approach • Make salvage easier by • viewing applications as compositions of different pieces • having a framework of collaborating pieces from which applications can be composed dynamically • Try to achieve benefits of the “Software-IC” model— a plug-and-play approach to software construction.

20. Our Approach • We are limiting the hard problem of general reuse to a smaller domain—of salvage within an organization, to be used in the construction of modest-sized systems (about 200,000 lines of code). • To address this domain would be to provide solutions of value to the small and medium-sized software shops that have been resistant to adopt systematic reuse.

21. Software Matrix • The Software Matrix is a framework that actively supports and promotes the salvage of components. • We focus on the reuse of major blocks of code rather than low-level components • By employing message-passing and mediator structures and by supporting the discovery of needed types, we’ve built a pluggable architecture that can gracefully adapt to salvage operations.

22. Software Matrix • In particular, the Matrix is a runtime infrastructure that acts as a substrate into which individual pieces of an application – different blocks of code – can plug.

23. Software Matrix From these “plugged-in” individual pieces the Matrix dynamically composes applications.

24. Software Matrix • This infrastructure was named the Software Matrix in order to connote a very structured pattern of building software. • System elements are embedded in this substrate – a matrix /collection – through which applications are dynamically composed. • The image of endless banks of incubators of humans in a matrix, as from “The Matrix” motion picture, isn’t entirely unintentional, if somewhat flippant.

25. Cells • The individual blocks of code that plug-in to the Matrix are called “Cells” and are the building blocks out of which applications are built.

26. Cells • Cells represent the unit of composition and reuse in our system.

27. Cells • An application is built through the collaboration of Cells, and all Cells communicate with each other strictly through messages (we use XML encoded messages)

28. Cells • One of the problems of salvaging: extracting parts of monolithic (or very tightly-coupled) applications. • By enforcing a separation of concern between Cells by inserting a message bus between them we are insisting on loose-coupling between different pieces of an application, which makes salvaging these pieces for use in future applications much easier.

29. A closer look at Cells… • Every Cell contains: • a message queue: holds request and response messages during collaboration with other cells.

30. A closer look at Cells… • Every Cell contains: • a capability list: used to advertise the capabilities of a cell to other cells, via the Matrix. (E.g. SU.Math.Convolution) The capability list is used by the Matrix in order to discover the right cells for system construction.

31. A closer look at Cells… • Every Cell contains: • a globally unique identifier (GUID): Cells can be uniquely identified using a GUID. This is used by the Matrix for several operations such as discovery and registration.

32. A closer look at Cells… • Every Cell contains: • functionality: Cells also contain the functionality that allows them to be considered as software assets with potential for reuse.

33. A closer look at Cells… • All cells subscribe to a common protocol (ICell) that specifies how to: • register and un-register with the Matrix • advertise capabilities • send and accept messages • collaborate with other cells (could be synchronous, asynchronous or one-way) ICell MyCell

34. A closer look at Cells… • Every cell also has an entry-point (start), and is given a chance to execute once it is plugged-in. • This entry-point (empty/non-empty) decides whether a cell will be only a passive ‘server’, or itself actively seek collaboration from other cells.

35. Example • Sample application: needs to read data samples from an input file, perform a signal processing operation on the a data (e.g., filtering), plot the results of the operations on the system display, and log the results to a file. • The major pieces of the application would be responsible for: • file operations • signal processing • graphical plotting

36. Example • These would be written as cells and plugged-in to the Matrix. • The Matrix would then assume the responsibility of constructing the application from these individual cells. • In order to perform its task a cell may need services of another cell. But cells do not explicitly bind to other cells – they only specify what message type they need handled, and the Matrix discovers cells capable of handling that message. • If no suitable cell found, a “not supported” message is generated.

37. Example

38. Example Sequence of events in the construction of the sample application

39. Example • So what we see here is dynamic composition – we are building a system from pieces that exist on the Matrix at runtime. The compositional aspects (as opposed to only computational aspects) of software are being taken care of by the Matrix. • It automatically connects the right pieces at run-time without having to bind to anything explicitly at compile-time. • This method of software construction, with its dynamic nature and loosely coupled building blocks, is amenable to salvage operations. The very same Cells could be used to build other applications.

40. How do cells “plug-in”? • Cells are implemented as “plug-in” modules, and are realized using .NET components. • The Matrix uses the ‘reflection’ and ‘late-binding’ (Fusion) features of the .NET framework to discover and register plug-ins. • Given an .NET assembly, the Matrix will reflect over the contained types to determine which of them implement the ICell interface in order to recognize valid plug-ins.

41. How do cells “plug-in”? • Throughout its lifetime, the Matrix runtime infrastructure monitors a known location – the plug-in directory, in order to discover if any new cells are present. • On detecting a valid cell, the Matrix ‘registers’ the cell with itself. This newly registered cell (along with previously registered cells) is then available for system construction.

42. How do cells “plug-in”? At periodic intervals the Software Matrix interrogates a known location in order to discover if any new Cells are present, and if so registers them.

43.  In order to enable salvage, pieces of an application (at the time of writing it, or existing pieces) are wrapped in a Cell. Create a wrapper that inherits from ICell (e.g. we wish to create FileManager Cell responsible for common file operations ) FileManager : ICell { ... } Steps to take advantage of the Matrix

44.  The Capability List of this FileManager Cell is then populated to indicate the types of messages it is capable of handling. Steps to take advantage of the Matrix CapabilityList.Add(“SU.FileManager.Files.Read”); CapabilityList.Add(“SU.FileManager.Files.Write”); CapabilityList.Add(“SU.FileManager.Files.Search”); CapabilityList.Add(“SU.FileManager.Files.Compression”);

45.  Override the “process” method to add appropriate message processing Basically you specify what is to be done in response to a particular message type – delegate calls to your implementation. Steps to take advantage of the Matrix

46.  Compile, and copy the resulting binary into the “plug-in” directory of the Software Matrix. The Matrix will automatically detect and register the cell, and it will be available for composition. Steps to take advantage of the Matrix

47. If a cell wishes to use other cells (or is a program executive for instance), it will probably say: ... Result = syncSend( "SU.FileManager.Files.Search", Params); ... Here it is trying to locate a cell that can handle the named message type. Steps to take advantage of the Matrix

48. Features • Fine-grained message-passing • Many problems associated with software salvage arise from extracting one of several tightly-coupled pieces of a system. • We are using message passing at a much finer level of granularity than is normally seen • The Matrix requires messages be the only mode of collaboration between different parts of an application, thereby decreasing the degree of coupling. • This loose coupling is critical to the success of salvage operations.

49. Features • Dynamic Composition • The Matrix takes care of the compositional aspects of software by automatically discovering and connecting the right pieces and building an application at runtime • By adding a few more cells into the Matrix if needed, we can build new applications by reusing existing cells.

50. Features • Support for System Evolution • Since the appropriate cell to serve a particular message type is selected at runtime, the Matrix supports evolution of software systems. • Evolving requirements can be accommodated easily by unplugging or modifying only those cells that represent the affected part of the system. • Easy to field-replace cells – all one has to copy the new cell over the old cell in the plug-in directory. • Effective way to support program maintainability, bug fixes/upgrades.