Toward a framework for cots classification and evaluation
1 / 57

Toward a framework for COTS classification and evaluation - PowerPoint PPT Presentation

  • Uploaded on

Toward a framework for COTS classification and evaluation Letizia Jaccheri This talk I present me, IDI, NTNU, Trondheim, Norway Project goals Context (education, research) Brainsto rming COTS classification COTS evaluation Related work Conclusions and further work Letizia Jaccheri

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Toward a framework for COTS classification and evaluation' - omer

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

This talk l.jpg
This talk

  • I present me, IDI, NTNU, Trondheim, Norway

  • Project goals

  • Context (education, research)

  • Brainstorming

  • COTS classification

  • COTS evaluation

  • Related work

  • Conclusions and further work

Bari 19/12/01

Letizia jaccheri l.jpg
Letizia Jaccheri

  • Pisa 1965

  • Master in CS Pisa 1988

  • Programmer 1988-1989

  • PhD in Software Engineering, Torino 1994

  • Guest researcher in Trondheim from 1990 to 1992

  • Assistant professor in Software Engineering Torino 1994-1997

  • Førstamanuensis IDI/NTNU 1997

Bari 19/12/01

Teaching and research l.jpg

Assistant from 1991/2 at Politecnico Programming, Introduction to CS, TO (PT)

Designed and run new course in Software engineering 2 at PT

Software quality and process improvement (NTNU)

Software architecture (NTNU)

Software development environment

Software process modelling



Software engneering education

Software architecture

Empirical studies

Human aspects in SE

Teaching and research

Bari 19/12/01

Idi ntnu l.jpg
IDI NTNU Introduction to CS, TO (PT)

Bari 19/12/01

Slide6 l.jpg

Bari 19/12/01 Introduction to CS, TO (PT)

Slide7 l.jpg

Bari 19/12/01 Introduction to CS, TO (PT)

Project goals l.jpg
Project Goals Introduction to CS, TO (PT)

  • Component out of the Shelf (COTS)

    • Learning/pedagogical goals

    • Research goals

Bari 19/12/01

P edagogical context and goals l.jpg

A 5th year course, SE master students Introduction to CS, TO (PT)

Best Norwegian students, 10% of applicants can start

how to evaluate technology

how to create value by help of new technology

how to learn new technology

Pedagogical context and goals

Bari 19/12/01

Research context and goals l.jpg

Papers about COTS Introduction to CS, TO (PT)

Difficulties in defining what a COTS is

Few examples

Little empirical work

Classify and evaluate existing COTS

Develop research hypotheses

A mean not an end

Research context and goals

Bari 19/12/01

Motivation l.jpg
Motivation Introduction to CS, TO (PT)

  • The motivation for constructing taxonomy is to provide both researchers and practitioners a tool that allow communicating, generalizing conclusions, and applying research findings (Glass).

  • Which classes are empy? Why?

  • Being able to evaluate COTS, also by comparison

  • The final DB is not an end but a mean

Bari 19/12/01

Classification and evaluation framework l.jpg
Classification and evaluation Introduction to CS, TO (PT) framework

  • How to make classes:

    • A COTS classification schema

  • How to evaluate

    • a set of evaluation attributes

Bari 19/12/01

Brainstorming l.jpg
Brainstorming Introduction to CS, TO (PT)

  • how did (or would) you solve the problem of COTS classification and evaluation?

  • List 3 examples of COTS

  • List some classes of COTS (browsers, DBMS, …)

  • List 3 evaluation attributes (price, reliability, …)

Bari 19/12/01

The pedagogical context l.jpg
The pedagogical context Introduction to CS, TO (PT)

  • Active Reading

  • Doing

  • Teaching

  • Connecting people


Bari 19/12/01

The process l.jpg
The process Introduction to CS, TO (PT)

  • Define classes, Propose COTS, propose evaluation attributes (Students 1. Assignment)

  • Choose classes, items, evaluation attributes, and scenarios (teacher and other researchers)

  • Evaluate COTS (Students 2. Assignment)

Bari 19/12/01

1 phase deliverables l.jpg
1. Phase deliverables Introduction to CS, TO (PT)

  • Provide at least six attributes which you believe are significant to describe/evaluate COTS

  • Provide a list of COTS’s to be evaluated. Start from links

Bari 19/12/01

List of software technology l.jpg
List of software technology Introduction to CS, TO (PT)

  • Java;

  • Microsoft;

  • Rational;

  • Argo UML;

  • Ibm,

  • Orion;

  • WAP Empreror (WAP utvikling);

  • Paint Shop Pro 7 (tegning);

  • Corba;,

  • Winwap;


  • PDA

  • Perl

  • Macromedia,

Bari 19/12/01

List of software technology 2 l.jpg
List of software Technology (2) Introduction to CS, TO (PT)

  • HTML/XHTML/dynamic HTML XML;

  • Opera

  • Lynx

  • Web TV SMIL

  • Amaya

  • Fetch!

  • Neoplanet

  • Mobile Access

  • Together ControlCentre

  • Oracle

  • Jasmine

  • Sybase

Bari 19/12/01

Finding classes l.jpg
Finding classes Introduction to CS, TO (PT)

Our classification is based in three different axes:

  • architectural level

  • kind

  • phase

Bari 19/12/01

3 tier l.jpg
3-tier Introduction to CS, TO (PT)

Bari 19/12/01

Example architectural level l.jpg
Example (Architectural level) Introduction to CS, TO (PT)

  • Client

    • Examplesare: HTML, Opera, Netscape, Internet Explorer, macromedia flash, java client compiler, etc..

  • Server

    • Examples are: servlet, javac, JSP, Perl, PhP, Microsoft .net, Microsoft ASP,, etc..

  • Data

Bari 19/12/01

Example kind l.jpg
Example (kind) Introduction to CS, TO (PT)

  • executable statements either in source or binary form.

    • Examples are javac, macromedia flash, etc..

  • Standards available and approved in some forum

    • Examples are the html specification as available at

  • service provided by other through some network

    • Examples are we-based version control systems such as, etc..

Bari 19/12/01

Phase l.jpg
Phase Introduction to CS, TO (PT)

  • during development

    • Examples are: Macromedia Flash is used during development of client level components,

  • part of the final running system

    • Examples are Internet Explorer, mysql, etc..

Bari 19/12/01

Evaluation attributes l.jpg
Evaluation Introduction to CS, TO (PT)Attributes

  • Acquisition cost Number

  • Ownership cost Number

  • Market size Number

  • Market Number

  • License type Nominal (site, personal, evaluation)

Bari 19/12/01

Generating classes l.jpg
Generating classes Introduction to CS, TO (PT)

  • 3 x 3 x 2 =18

Bari 19/12/01

Examples l.jpg
Examples Introduction to CS, TO (PT)

  • HTML (client, standard, part)

  • Netscape (client, executable, part)

  • IE (client, executable, part)

  • Servlet (server, standard, part)

  • PhP (server, standard, part)

  • (client, service, operation)

  • Ambiguities may arise.

Bari 19/12/01

Example java l.jpg
Example, java Introduction to CS, TO (PT)

Bari 19/12/01

Assignment 2 l.jpg
Assignment Introduction to CS, TO (PT)2

  • Thursday 27th September meeting

    • evaluation table

    • list of software items (derived from those provided by students)

    • students will be assigned to items and will evaluate them according to the common evaluation table

  • Delivery at October 15th

  • Results presented and discussed in a workshop, which will be held the 18th of October. Persons from industry participate to this workshop

Bari 19/12/01

Results from assignment 1 l.jpg
Results from assignment 1 Introduction to CS, TO (PT)

  • Proposed items

    • 174, 555, min 4, max 36

Bari 19/12/01

Frequency of classes of proposed items l.jpg
Frequency of classes of proposed items Introduction to CS, TO (PT)

Bari 19/12/01

Architecture l.jpg
Architecture Introduction to CS, TO (PT)

  •  3-tier

  • 2-tier

  • Standalone / Centralized

  • Blackboard

  • Peer to Peer

  • Client-Server

Bari 19/12/01

Slide32 l.jpg
Kind Introduction to CS, TO (PT)

  • Executable: compiled, ready-to-run item.

  • Standard: describing a documented standard.

  • Service: which can be used from another item or directly by a user.

  • Source: source code.

  • Documentation: documentation for example describing other software items.

Bari 19/12/01

Phase33 l.jpg
Phase  Introduction to CS, TO (PT)

  • development

  • compilation: items helping compilation.

  • execution

  • maintenance: change of modules, upgrades etc.

  • start-up: items for starting another software item (for example boot programs).

  • phase-out: items for ending the life cycle of another item.

Bari 19/12/01

Possibilities l.jpg
Possibilities  Introduction to CS, TO (PT)

  • Resulting in 210 (7x5x6) distinctive classification possibilities????

Bari 19/12/01

Context l.jpg
Context Introduction to CS, TO (PT)




Bari 19/12/01

Evaluation attributes36 l.jpg

product maturity Introduction to CS, TO (PT)

market share




hw requirements

product support






change frequency

licence type

acquisition cost

sw requirements

standard configuration

domain specific

Evaluation attributes

Bari 19/12/01

What do we expect students to do l.jpg
What do we expect students to do? Introduction to CS, TO (PT)

  • Reasoning about sw products with respect to 3 scenarios

  • Not so much installation and testing

Bari 19/12/01

My proposal l.jpg
My proposal Introduction to CS, TO (PT)

  • Architecture =

    • 3-tier (client,

    • server,

    • data)

    • Stand alone

  • Kind (executable, standard, service)

  • Phase (development, deployment/execution)

    • We are not interested in architecture of development tools

Bari 19/12/01

Slide39 l.jpg

Bari 19/12/01 Introduction to CS, TO (PT)

Scenarioes for technology evaluation l.jpg
Scenarioes for Technology Evaluation Introduction to CS, TO (PT)



  • Your current project if applicable, else a former project.

Bari 19/12/01

Amazon l.jpg
Amazon Introduction to CS, TO (PT)

  • Very many users.

  • Personalised, dynamic web pages.

  • Recommandations based on former purchases, clicked items in session.

  • Large number of products, product presentations.

  • On-line shopping (single-click), payment policies.

  • Email confirmations of purchasing, delivery.

Bari 19/12/01

Sif80at l.jpg
SIF80AT Introduction to CS, TO (PT)

  • Few users, protective environment for input (password protected).

  • Provides information.

  • Normal users provides data/result input interactively through a GUI.

  • Normal users presents, change own data/results.

  • Authorised users perform analysis and statistics based on the delivered data/results from normal users.

Bari 19/12/01

Own project l.jpg
Own project Introduction to CS, TO (PT)

  • If your project is technology-based, evaluate the list software items to be applied in your project.

  • Else select a former technology project to be evaluated.

Bari 19/12/01

What and how to measure l.jpg
What and how to measure Introduction to CS, TO (PT)

  • product maturity: write a story

  • Market share: is a percentage, ratio

  • Performance: number of users it scales to

  • Security/safety: write a story

Bari 19/12/01

Slide45 l.jpg
Cont. Introduction to CS, TO (PT)

  • Reliability: write a story, try to mention faults

  • hw requirements: list

  • product support: write a summary

  • sw requirements: list

  • standard configuration: ?

  • domain specific: if yes, list domain

Bari 19/12/01

Slide46 l.jpg
Cont. Introduction to CS, TO (PT)

  • Documentation: kind (web, on line, etc.), number of manual pages

  • Usability: degree of satisfaction on a scale from 1 to 5 and write a story

Bari 19/12/01

Slide47 l.jpg
Cont. Introduction to CS, TO (PT)

  • Learnability: how much time does it take to learn to use this component in this scenario? Use historical data, or literature. Discuss relationship to the different scenarios.

  • Modifiability: how easy is it to modify this item? Not modifiable, parametrizable, provide interface

Bari 19/12/01

Slide48 l.jpg
Cont. Introduction to CS, TO (PT)

  • Maintenability: this does not apply to third part software

  • change frequency: how many releases/versions in the last year?

  • licence types: list

  • acquisition cost: we will have to distinguish between licence types

Bari 19/12/01

Results so far l.jpg
Results so far Introduction to CS, TO (PT)

  • Why is this class empty?” A class may be empty because existing COTS have been forgotten or because there do not exist COTS with those characteristics on the market. If such COTS do not exist, an obvious reason may be that that class does not make sense. A more interesting reason may be that COTS like that have not been yet developed, and this may open for new research. One can for example ask “Is the class of tools for development of server level standard not interesting?”

Bari 19/12/01

Slide50 l.jpg
Cont. Introduction to CS, TO (PT)

  • Other research questions may be generated by looking at the existing or not existing relationships between classes of COTS. Like, “are there any relationships between the COTS for client standard development and the COTS for server standard development?”.

Bari 19/12/01

Slide51 l.jpg
Cont. Introduction to CS, TO (PT)

  • Another interesting consequence of our work is to assess its validity for learning purposes both in academia and in industry. We believe that one gets a deep knowledge of COTS components by taking them in use. Another interesting question is “which is the relationship between a classification work like the one we propose and a learning program based on active use of COTS components for, for example, prototype development?”.

Bari 19/12/01

Slide52 l.jpg
Cont. Introduction to CS, TO (PT)

  • Finally,to our knowledge, there is no work in the literature that either proposes the distinction of classification and evaluation attributes or emphasizes the benefits of classification for learning new technologies.

Bari 19/12/01

References classification l.jpg
References (classification) Introduction to CS, TO (PT)

  • Glass, R., Vessey, I. “Contemporary Application Domain Taxonomies”, IEEE Software, Vol. 12, No. 4; July 1995, pp. 63–76.

  • Carney, D., Long, F. “What Do You Mean by COTS? Finally a Useful Answer”, IEEE Software, 17(2): 83-86, March/April 2000.

  • Morisio, M., Torchiano, M. “Definition and Classification of COTS: a proposal” Accepted at International Conference on COTS Based Software Systems (ICCBBS), Orlando (FL), February 4-6, 2002.

Bari 19/12/01

References evaluation l.jpg
References (Evaluation) Introduction to CS, TO (PT)

  • Boloix, G., Robillard, P.N. “A software system evaluation framework”, IEEE Computer, 28(10): 17–26, December 1995.

  • Kontio, J. “A Case Study in Applying a Systematic Method for COTS Selection”, Proc. of IEEE-ACM 18th Int. Conf. on Software Engineering (ICSE), Berlin, Germany, March 25-29, 1996, pp 201-209.

Bari 19/12/01

Slide55 l.jpg
Cont. Introduction to CS, TO (PT)

  • Morisio, M., Tsoukiàs, A., “IusWare: A methodology for the evaluation and selection of software products”, IEE Proceedings Software Engineering, 144(3):162-174, June 1997.

  • Ochs, M.A.; Pfahl, D.; Chrobok-Diening, G.; Nothhelfer-Kolb, B. “A Method for Efficient Measurement-based COTS Assessment ad Selection – Method Description and Evaluation Results” in Proceedings IEEE 7th International Software Metrics Symposium, London, England, 4-6 April 2001. pp. 285-296

Bari 19/12/01

Slide56 l.jpg
Cont. Introduction to CS, TO (PT)

  • Ovum home page, available at

  • SIF80AT - A course in new software technologies, home page available at

Bari 19/12/01

Further work l.jpg

To assess our classification/evaluation model Introduction to CS, TO (PT)

Code/work on the evaluation material

Is it meaninfull to compare items in the same class?

Do we have too many, too few classes?

Run statistics correlations to find/assess relationships between attributes

Further work

Bari 19/12/01