slide1
Download
Skip this Video
Download Presentation
Quality Determination

Loading in 2 Seconds...

play fullscreen
1 / 55

Quality Determination - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

Quality Determination. for Web-based Applications. Hazura Zulzalil. Overview of MCDA General definition MCDM process MCDA methods Evaluation of WBA Quality Attribute Relationships Aggregation by Choquet Integral Implementation Case study and results. Outlines.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Quality Determination' - loc


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
Quality Determination

for Web-based Applications

Hazura Zulzalil

outlines
Overview of MCDA

General definition

MCDM process

MCDA methods

Evaluation of WBA

Quality Attribute Relationships

Aggregation by Choquet Integral

Implementation

Case study and results

Outlines
what is mcda
Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account.What is MCDA?
what is mcdm
Highly structured, disciplined and formal approach to decision making

evaluating the alternatives in the given set A against the set C of criteria

Aggregating the individual evaluations to produce global evaluation

Could be used for selection the best possible alternatives or for ranking the alternatives

What is MCDM?
mcdm process
Set of Alternatives

Set of Criteria

C1, C2,………Cn

A1 x11……..………x1n

A2 x21……..………x2n

.

.

Am xn1……..………xmn

Weights wi /

Importance of Criteria

Aggregation Measure

Overall worth of an alternative Ai

MCDM Process
evaluation of mcda methods
Criteria – interdependence, completeness, non-linear preferences

Weights – transparency of process, type of weights, meaning

Solution finding procedure – ranking, option

Project constraints – cost, time

Evaluation of MCDA methods
evaluation of mcda methods1
Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication

Data Situation

Type of data - qualitative or quantitative

Risk/uncertainties – probabilities, thresholds, fuzzy numbers, sensitive analysis

Data processing amount

Non-substitutability

Evaluation of MCDA methods
evaluation of wba
Quality of web site is hard to evaluate

Consists of multiple criteria to be measured

Simple weighted average cannot be used to summaries the various quality measurements into a single score.

Inability to account for dependency among the quality criterion.

Tend to construct independent criteria, or criteria that are supposed to be so

Causing some bias effect in evaluation

Evaluation of WBA
wba evaluation approaches
Single criteria

usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005),

content and structure (Bauer & Scharl, 2000).

accessibility (Vigo et al., 2007)

WBA Evaluation Approaches
wba evaluation approaches1
Multi-criteria

WEBQEM (Olsina et al., 1999)

EWAM (Schubert & Selz, 1998)

WebQual (Barnes and Vidgen, 2002)

WAI (Miranda et al., 2006)

FQT4Web (Davoli et al., 2005)

WBA Evaluation Approaches
slide11
Stated or implied needs

ISO 9126 & other technical info

Requirement definition

Managerial requirement

Quality requirement specification

Quality

Requirement

Definition

Metric Selection

Rating level definition

Assessment criteria definition

Preparation

Software Development

Products

Measured value

Measurement

Evaluation

Rated value

Rating

Result (acceptable or unacceptable)

Assessment

ISO/IEC 9126 Evaluation Process

quality model
e-commerce

e-learning

e-education

e-government

etc.

APPLICATION

DOMAIN

S

C

I

T

S

Y

I

T

R

I

Functionality

Reliability

Usability

Efficiency

Maintainability

Portability

L

E

A

T

U

C

A

Q

R

A

H

C

understandability

S

Adaptability

C

time behaviour

learnability

Analysability

maturity

suitability

I

installability

T

resource

operability

changeability

fault tolerance

accuracy

S

coexistence

I

utilisation

attractiveness

stability

recoverability

R

interoperability

replaceability

efficiency

expliciteness

testability

availability

E

T

security

portability

customisability

compliance

manageability

degradability

C

compliance

traceability

clarity

reusability

reliability

A

R

helpfulness

maintainability

compliance

functionality

A

user-friendliness

compliance

H

compliance

C

usability compliance

B

U

S

Indicators, scales and preferred values

Quality Model
quality attributes for wba
Define software product qualities as a hierarchy of factors, criteria and metrics.

Quality factor represents behavioral characteristics of the system

Quality criterion is an attribute of a quality factor that is related to software production and design

Quality metrics is a measure that captures some aspect of a quality criterion.

Quality Attributes for WBA
slide14
Factor A is split up into three criteria a1, a2, and a3. Criteria a1 with the weight 4 is considered four times as important as criteria a2 and twice as important as criteria a3.

Similarly, we can set different weight for each factor to indicate its importance.

Overall Quality Score

Factor A

Factor B

Factor C

Criteria a1, weight 4

Criteria a2, weight 1

Criteria a3, weight 2

definition of quality attributes
Name

Description

Functionality

The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions

Usability

The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions

Reliability

The capability of the Web site to maintain a specified level of performance when used under specified conditions.

Efficiency

The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions

Maintainability

The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications

Portability

The capability of the site to be transferred from one environment to another

Definition of Quality Attributes
quality attributes relationships
Three types of relationships

Positive, i.e. a good value of one attribute result in a good value of the other (synergistic goals).

Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be enhanced (+)

Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals).

Relationships definitions: If characteristics A is enhanced, then characteristics B is likely to be degraded (-)

Independent, i.e. the attributes do not affect each other.

Relationships definitions: If characteristics A is enhanced, then characteristics B is unlikely to be affected (0)

Quality Attributes Relationships
techniques to explore the relationships
Ref

Attributes

Purpose

Techniques used

[8, 9]

Correctness, Reliability

Integrity, Usability

Efficiency, Maintainability

Testability, Flexibility

Portability. Reusability

Interoperability

To study the relations of different quality goals attribute in developing software

Survey -questionnaire

[10]

Performance

Adaptability

Maintainability

To address the importance of design decision made during software development

Case Study - Interview

[11]

Usability

Time to market

Reliability, Usability

Correctness, Portability

To increase the understanding of software quality attributes and their relations

Research Literature and Survey –structured interview

[12]

Quality attributes in 3 different perspectives: management, developer and user perspective

To merge different view and discuss the relationships between the quality attributes

Discussion (meeting and offline discussion)

Techniques to explore the relationships
what is aggregation
method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values What is Aggregation?
aggregation issues
use simple weighted average approach

methods are not transparent

assume independency

the choice of summarization method somehow should depend on the certain parameters

E.g. the kind of importance parameters (weights) and the type of dependency and interaction

the definition of the quality factors and their relationships must be clearly specified

Aggregation issues
common aggregation operators
Common aggregation operators

Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.)

Not stable under linear transformation and consider criteria as non interacting

Median

Typical ordinal operator – defined the middle value of the ordered list

Weighted minimum and maximum

Possible to increase one of the weights without having any change in the result

Ordered weighted averaging operators

Can express vague quantifiers

23

properties of an aggregation operator
mathematical properties

Properties of extreme values

Idempotence

Continuity

Monotonicity

Commutativity

Decomposability

Stability under the same positive linear transformation

Properties of an aggregation operator
properties of an aggregation operator1
behavioural properties

express the decisional behavior, interaction between criteria, interpretability of the parameters and weights on the arguments

Properties of an aggregation operator
aggregation by fuzzy integral
Aggregation by fuzzy integral

Different methods have been developed according to

type of information to be aggregated and

the properties have to be satisfied.

26

fuzzy measures and integral
Fuzzy measures and integral

Definition 1: A fuzzy measure on the set X of criteria is a set function  : Ƥ (X) [0,1], satisfying the following axioms

 ()=0,  (X)=1.

A  B  X implies (A)  (B)

(A) represent the weight of importance of the set of criteria A.

Additive : if (AB) = (A) + (B); A  B=

Superadditive: if (AB)  (A) + (B); A  B=

Subadditive if (AB)  (A) + (B); A  B=

If a fuzzy measure is additive, then it suffices to define n coefficients (weights) ({ I}), … ({ n})

27

choquet integral
Choquet integral

Definition 2: Let  be a fuzzy measure on X.

The choquet integral of a function

ƒ: (X) [0,1] with respect to  is defined by

n

C (f(x1),…. f(xn)):=  (f(x(i)) - f(x(i-1))) (A(i))

ƒ ((0)) = 0

i = 1

  • Fuzzy integral model does not need to assume independency
  • Fuzzy integral of ƒwith respect to  gives the overall evaluation of an alternative

28

importance and interaction of criteria
Importance and interaction of criteria

Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L).

By weighted sum (3 , 3, 2) result:

29

slide30
Solved by fuzzy measure  and the choquet integral

Scientific subjects are more important than literature;

 ({M}) =  ({P}) =0.45;  ({L}) = 0.3

M and P are redundant,

 ({M, P}) = 0.5 < 0.45 + 0.45

Students equally good at scientific subjects and literature,

 ({L, M}) = 0.9 > 0.45 + 0.3

 ({L, P}) = 0.9 > 0.45 + 0.3

 ()=0,  ({M, P, L})=1

30

slide31
Result by applying fuzzy measure:

* The initial ratio of weight (3, 3, 2) is kept unchanged

31

complexity of the model
Complexity of the model

Number of coefficients grows exponentially with the number of criteria to be aggregated.

3 approaches (to reduce the number of coefficients)

Identification based on semantics

Importance of criteria

Interaction between criteria

Symmetric criteria

Veto effects

Identification based learning data

Minimization of squared error

Constraint satisfaction

Combining semantics and learning

32

proposed solution
Apply 2-additive Choquet integral

provide the information about the relationships among criteria (redundancy or support among criteria) and the preference among alternatives

Derive fuzzy measures by constraint satisfaction

Proposed solution
explore relationships
Techniques to explore how the different attributes are related to each other:

Experience Based Approach

Mathematical Modeling

Statistical Technique (Correlation Analysis)

measures the strength of a linear relationship among different quality factors

The main result of a correlation is called the correlation coefficient (r)

Explore relationships
implementation of choquet integral
Definition of the initial preferences.

Convert into Choquet integral form

Identify threshold values.

If solution exists, calculate the Choquet integral, Shapley value and Interaction indices

Implementation of Choquet Integral
slide41
Three preference thresholds C, Sh &I have to be determined before the aggregation take part.

Range of : 0 to 1

no rule to fix the , we need to compare the solutions obtain with different value of .

Once the solution exist, Choquet integral will be calculated

Define preference thresholds

slide43
Calculate the Shapley value

with

Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions.

Summation of these Shapley values for a given set of elements would represent the importance of the complete set

slide44
Calculate the Interaction Index

With

The interaction index Iij can be interpreted as a kind of average value of the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is said to be positive (resp. negative).

case study
Perform on 3 types of WBA

Academic

E-commerce

Museum

Four quality factor were evaluated

Usability,Functionality, Reliability, Efficiency

Each has different preference, importance and interaction

Case Study
slide48
Threshold

C= 1,

Sh = 0.1,

I =0.1,

conclusion
Aggregation by Choquet integral can be alternated if there is interaction exist between quality factors.

The proposed approach can be applied for non-interactive criteria as well. If there is no interaction between the criteria, then the fuzzy measure will be additive measures.

Results show that the global evaluation obtained is compatible with the weighted average method.

Conclusion
future works
The evaluation of WBA which cater the dynamic changes of the quality factors.

Behavior (Preferences, importance,interaction, etc.) can be change continuously.

Investigate more than 2 quality attribute interactions

Future works
ad