Quality Determination
Download
1 / 55

Quality Determination - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

Quality Determination. for Web-based Applications. Hazura Zulzalil. Overview of MCDA General definition MCDM process MCDA methods Evaluation of WBA Quality Attribute Relationships Aggregation by Choquet Integral Implementation Case study and results. Outlines.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Quality Determination ' - loc


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Quality Determination

for Web-based Applications

Hazura Zulzalil


Outlines

Overview of MCDA

General definition

MCDM process

MCDA methods

Evaluation of WBA

Quality Attribute Relationships

Aggregation by Choquet Integral

Implementation

Case study and results

Outlines


What is mcda

Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account.

What is MCDA?


What is mcdm

Highly structured, disciplined and formal approach to decision making

evaluating the alternatives in the given set A against the set C of criteria

Aggregating the individual evaluations to produce global evaluation

Could be used for selection the best possible alternatives or for ranking the alternatives

What is MCDM?


Mcdm process

Set of Alternatives decision making

Set of Criteria

C1, C2,………Cn

A1 x11……..………x1n

A2 x21……..………x2n

.

.

Am xn1……..………xmn

Weights wi /

Importance of Criteria

Aggregation Measure

Overall worth of an alternative Ai

MCDM Process


Evaluation of mcda methods

Criteria – interdependence, completeness, non-linear preferences

Weights – transparency of process, type of weights, meaning

Solution finding procedure – ranking, option

Project constraints – cost, time

Evaluation of MCDA methods


Evaluation of mcda methods1

Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication

Data Situation

Type of data - qualitative or quantitative

Risk/uncertainties – probabilities, thresholds, fuzzy numbers, sensitive analysis

Data processing amount

Non-substitutability

Evaluation of MCDA methods


Evaluation of wba

Quality of web site is hard to evaluate participant, tool for learning transparency, actors communication

Consists of multiple criteria to be measured

Simple weighted average cannot be used to summaries the various quality measurements into a single score.

Inability to account for dependency among the quality criterion.

Tend to construct independent criteria, or criteria that are supposed to be so

Causing some bias effect in evaluation

Evaluation of WBA


Wba evaluation approaches

Single criteria participant, tool for learning transparency, actors communication

usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005),

content and structure (Bauer & Scharl, 2000).

accessibility (Vigo et al., 2007)

WBA Evaluation Approaches


Wba evaluation approaches1

Multi-criteria participant, tool for learning transparency, actors communication

WEBQEM (Olsina et al., 1999)

EWAM (Schubert & Selz, 1998)

WebQual (Barnes and Vidgen, 2002)

WAI (Miranda et al., 2006)

FQT4Web (Davoli et al., 2005)

WBA Evaluation Approaches


Stated or implied needs participant, tool for learning transparency, actors communication

ISO 9126 & other technical info

Requirement definition

Managerial requirement

Quality requirement specification

Quality

Requirement

Definition

Metric Selection

Rating level definition

Assessment criteria definition

Preparation

Software Development

Products

Measured value

Measurement

Evaluation

Rated value

Rating

Result (acceptable or unacceptable)

Assessment

ISO/IEC 9126 Evaluation Process


Quality model

e-commerce participant, tool for learning transparency, actors communication

e-learning

e-education

e-government

etc.

APPLICATION

DOMAIN

S

C

I

T

S

Y

I

T

R

I

Functionality

Reliability

Usability

Efficiency

Maintainability

Portability

L

E

A

T

U

C

A

Q

R

A

H

C

understandability

S

Adaptability

C

time behaviour

learnability

Analysability

maturity

suitability

I

installability

T

resource

operability

changeability

fault tolerance

accuracy

S

coexistence

I

utilisation

attractiveness

stability

recoverability

R

interoperability

replaceability

efficiency

expliciteness

testability

availability

E

T

security

portability

customisability

compliance

manageability

degradability

C

compliance

traceability

clarity

reusability

reliability

A

R

helpfulness

maintainability

compliance

functionality

A

user-friendliness

compliance

H

compliance

C

usability compliance

B

U

S

Indicators, scales and preferred values

Quality Model


Quality attributes for wba

Define software product qualities as a hierarchy of factors, criteria and metrics.

Quality factor represents behavioral characteristics of the system

Quality criterion is an attribute of a quality factor that is related to software production and design

Quality metrics is a measure that captures some aspect of a quality criterion.

Quality Attributes for WBA


Factor A is split up into three criteria a criteria and metrics. 1, a2, and a3. Criteria a1 with the weight 4 is considered four times as important as criteria a2 and twice as important as criteria a3.

Similarly, we can set different weight for each factor to indicate its importance.

Overall Quality Score

Factor A

Factor B

Factor C

Criteria a1, weight 4

Criteria a2, weight 1

Criteria a3, weight 2


Definition of quality attributes

Name criteria and metrics.

Description

Functionality

The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions

Usability

The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions

Reliability

The capability of the Web site to maintain a specified level of performance when used under specified conditions.

Efficiency

The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions

Maintainability

The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications

Portability

The capability of the site to be transferred from one environment to another

Definition of Quality Attributes


Quality attributes relationships

Three types of relationships criteria and metrics.

Positive, i.e. a good value of one attribute result in a good value of the other (synergistic goals).

Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be enhanced (+)

Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals).

Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be degraded (-)

Independent, i.e. the attributes do not affect each other.

Relationships definitions: If characteristics A is enhanced, then characteristics B is unlikely to be affected (0)

Quality Attributes Relationships



Relationship chart gillies 1997
Relationship Chart (Gillies, 1997) criteria and metrics.


Techniques to explore the relationships

Ref criteria and metrics.

Attributes

Purpose

Techniques used

[8, 9]

Correctness, Reliability

Integrity, Usability

Efficiency, Maintainability

Testability, Flexibility

Portability. Reusability

Interoperability

To study the relations of different quality goals attribute in developing software

Survey -questionnaire

[10]

Performance

Adaptability

Maintainability

To address the importance of design decision made during software development

Case Study - Interview

[11]

Usability

Time to market

Reliability, Usability

Correctness, Portability

To increase the understanding of software quality attributes and their relations

Research Literature and Survey –structured interview

[12]

Quality attributes in 3 different perspectives: management, developer and user perspective

To merge different view and discuss the relationships between the quality attributes

Discussion (meeting and offline discussion)

Techniques to explore the relationships



What is aggregation

method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values

What is Aggregation?


Aggregation issues

use simple weighted average approach one, so that the result of aggregation takes into account in a given manner all the individual values

methods are not transparent

assume independency

the choice of summarization method somehow should depend on the certain parameters

E.g. the kind of importance parameters (weights) and the type of dependency and interaction

the definition of the quality factors and their relationships must be clearly specified

Aggregation issues


Common aggregation operators
Common aggregation operators one, so that the result of aggregation takes into account in a given manner all the individual values

Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.)

Not stable under linear transformation and consider criteria as non interacting

Median

Typical ordinal operator – defined the middle value of the ordered list

Weighted minimum and maximum

Possible to increase one of the weights without having any change in the result

Ordered weighted averaging operators

Can express vague quantifiers

23


Properties of an aggregation operator

mathematical properties one, so that the result of aggregation takes into account in a given manner all the individual values

Properties of extreme values

Idempotence

Continuity

Monotonicity

Commutativity

Decomposability

Stability under the same positive linear transformation

Properties of an aggregation operator


Properties of an aggregation operator1

behavioural properties one, so that the result of aggregation takes into account in a given manner all the individual values

express the decisional behavior, interaction between criteria, interpretability of the parameters and weights on the arguments

Properties of an aggregation operator


Aggregation by fuzzy integral
Aggregation by fuzzy integral one, so that the result of aggregation takes into account in a given manner all the individual values

Different methods have been developed according to

type of information to be aggregated and

the properties have to be satisfied.

26


Fuzzy measures and integral
Fuzzy measures and integral one, so that the result of aggregation takes into account in a given manner all the individual values

Definition 1: A fuzzy measure on the set X of criteria is a set function  : Ƥ (X) [0,1], satisfying the following axioms

 ()=0,  (X)=1.

A  B  X implies (A)  (B)

(A) represent the weight of importance of the set of criteria A.

Additive : if (AB) = (A) + (B); A  B=

Superadditive: if (AB)  (A) + (B); A  B=

Subadditive if (AB)  (A) + (B); A  B=

If a fuzzy measure is additive, then it suffices to define n coefficients (weights) ({ I}), … ({ n})

27


Choquet integral
Choquet integral one, so that the result of aggregation takes into account in a given manner all the individual values

Definition 2: Let  be a fuzzy measure on X.

The choquet integral of a function

ƒ: (X) [0,1] with respect to  is defined by

n

C (f(x1),…. f(xn)):=  (f(x(i)) - f(x(i-1))) (A(i))

ƒ ((0)) = 0

i = 1

  • Fuzzy integral model does not need to assume independency

  • Fuzzy integral of ƒwith respect to  gives the overall evaluation of an alternative

28


Importance and interaction of criteria
Importance and interaction of criteria one, so that the result of aggregation takes into account in a given manner all the individual values

Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L).

By weighted sum (3 , 3, 2) result:

29


Solved by fuzzy measure one, so that the result of aggregation takes into account in a given manner all the individual values  and the choquet integral

Scientific subjects are more important than literature;

 ({M}) =  ({P}) =0.45;  ({L}) = 0.3

M and P are redundant,

 ({M, P}) = 0.5 < 0.45 + 0.45

Students equally good at scientific subjects and literature,

 ({L, M}) = 0.9 > 0.45 + 0.3

 ({L, P}) = 0.9 > 0.45 + 0.3

 ()=0,  ({M, P, L})=1

30


Result by applying fuzzy measure: one, so that the result of aggregation takes into account in a given manner all the individual values

* The initial ratio of weight (3, 3, 2) is kept unchanged

31


Complexity of the model
Complexity of the model one, so that the result of aggregation takes into account in a given manner all the individual values

Number of coefficients grows exponentially with the number of criteria to be aggregated.

3 approaches (to reduce the number of coefficients)

Identification based on semantics

Importance of criteria

Interaction between criteria

Symmetric criteria

Veto effects

Identification based learning data

Minimization of squared error

Constraint satisfaction

Combining semantics and learning

32


Proposed solution

Apply 2-additive Choquet integral one, so that the result of aggregation takes into account in a given manner all the individual values

provide the information about the relationships among criteria (redundancy or support among criteria) and the preference among alternatives

Derive fuzzy measures by constraint satisfaction

Proposed solution


Explore relationships

Techniques to explore how the different attributes are related to each other:

Experience Based Approach

Mathematical Modeling

Statistical Technique (Correlation Analysis)

measures the strength of a linear relationship among different quality factors

The main result of a correlation is called the correlation coefficient (r)

Explore relationships


Correlation result
Correlation Result related to each other:


Implementation of choquet integral

Definition of the initial preferences. related to each other:

Convert into Choquet integral form

Identify threshold values.

If solution exists, calculate the Choquet integral, Shapley value and Interaction indices

Implementation of Choquet Integral


Define preference related to each other:

thresholds


Convert into Choquet integral form related to each other:


Three preference thresholds related to each other:C, Sh &I have to be determined before the aggregation take part.

Range of : 0 to 1

no rule to fix the , we need to compare the solutions obtain with different value of .

Once the solution exist, Choquet integral will be calculated

Define preference thresholds


Calculate the Choquet integral related to each other:


Calculate the Shapley value related to each other:

with

Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions.

Summation of these Shapley values for a given set of elements would represent the importance of the complete set


Calculate the Interaction Index related to each other:

With

The interaction index Iij can be interpreted as a kind of average value of the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is said to be positive (resp. negative).


Case study

Perform on 3 types of WBA related to each other:

Academic

E-commerce

Museum

Four quality factor were evaluated

Usability,Functionality, Reliability, Efficiency

Each has different preference, importance and interaction

Case Study


Result for academic website
Result for academic website related to each other:


Threshold related to each other:

C= 1,

Sh = 0.1,

I =0.1,


Summary(1) related to each other:


Summary(2) related to each other:


Comparison with other approaches
Comparison with other approaches related to each other:


Conclusion

Aggregation by Choquet integral can be alternated if there is interaction exist between quality factors.

The proposed approach can be applied for non-interactive criteria as well. If there is no interaction between the criteria, then the fuzzy measure will be additive measures.

Results show that the global evaluation obtained is compatible with the weighted average method.

Conclusion


Future works

The evaluation of WBA which cater the dynamic changes of the quality factors.

Behavior (Preferences, importance,interaction, etc.) can be change continuously.

Investigate more than 2 quality attribute interactions

Future works


ad