slide1
Download
Skip this Video
Download Presentation
CMMI and Metrics BCS SPIN SG, 19 February 2008 Clifford Shelley OXFORD SOFTWARE ENGINEERING Ltd

Loading in 2 Seconds...

play fullscreen
1 / 31

CMMI and Metrics BCS SPIN SG, 19 February 2008 Clifford Shelley OXFORD SOFTWARE ENGINEERING Ltd - PowerPoint PPT Presentation


  • 69 Views
  • Uploaded on

CMMI and Metrics BCS SPIN SG, 19 February 2008 Clifford Shelley OXFORD SOFTWARE ENGINEERING Ltd 9 Spinners Court, 53 West End, Witney, Oxfordshire OX28 1NH www.osel.co.uk [email protected] Tel. +44 (0) 1993 700878. Objective.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' CMMI and Metrics BCS SPIN SG, 19 February 2008 Clifford Shelley OXFORD SOFTWARE ENGINEERING Ltd' - amela-simmons


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
CMMI and Metrics

BCS SPIN SG, 19 February 2008

Clifford Shelley

OXFORD SOFTWARE ENGINEERING Ltd

9 Spinners Court, 53 West End,

Witney,

Oxfordshire

OX28 1NH

www.osel.co.uk

[email protected]

Tel. +44 (0) 1993 700878

objective
Objective
  • Present the background and history of measurement within CMMI
  • Place measurement within the context of CMMI (or CMMI within the context of measurement?)
  • Identify common issues and concerns with measurement within the context of CMMI, and their resolution
  • Look at some measurement approaches that can be really useful

(it is assumed here that ‘measurement’ is equivalent to ‘metrics’)

background
Background…
  • Original SW CMM (a tool to measure software capability)
    • Included measurement and analysis as a ‘common feature’ – briefly described
    • Expected product size estimates in planning
    • Measurement of processes (problematic at L2, expected at L3)
    • SPC like KPA at L4
background1
…Background…

Carried forward and developed to CMMI

  • Staged Representation
    • ML2
      • Measurement and Analysis is a PA in its own right
        • NB: Includes GQM as SP 1.1-1.4
      • PP requires product sizing
    • ML3
      • OPD SP 1.4 - process data defined and collected (using MA)
      • IPM SP 1.2 – use historical data for estimating
    • ML4
      • SPC expectations (OPP)
      • Control common causes of variation?
      • Exploited by QPM
    • ML5
      • Quantitiative Process Improvement – pervasive measurement
      • CAR uses measurement as analysis tool
      • OID uses measurement as an analysis tool
background2
…Background

Continuous Representation

  • CL2:
    • All PAs expected to be monitored and controlled (GP2.8 – measurement is expected)
  • CL3:
    • Standard process measures [defined and] stored (GP3.2)
  • CL4:
    • Quantitative objectives for processes established (GP4.1)
    • SPC required - stabilize sub-process performance (GP4.2)
      • Control special causes of variation – part of process capability as understood by production engineering
  • CL5:
    • Establish quantitative process improvement (GP5.1 sub-practice)
    • Manage common causes too – to enable process capability
measurement within cmmi
Measurement within CMMI…
  • MA PA is the enabler
  • “…is to develop and sustain a measurement capability that is used to support management information needs”
  • Interpretation?
measurement within cmmi1
…Measurement within CMMI…
  • MA Scope
    • SG1 Align Measurement and Analysis Activities
      • SP 1.1 Establish Measurement Objectives
      • SP 1.2 Specify Measures
      • SP 1.3 Specify Data collection and Storage Procedures
      • SP 1.4 Specify Analysis Procedures
    • SG2 Provide Measurement Results
      • SP 2.1 Collect Measurement Data (includes verification)
      • SP 2.2 Analyze Measurement Data
      • SP 2.3 Store Data and Results
      • SP 2.4 Communicate Results (to aid decision making)
measurement within cmmi2
…Measurement within CMMI
  • MA PA – Applicability for ML2
    • “…support management information needs…”
    • Project management (initially)
    • “…at multiple levels in the organization…”
    • and process management (GP 2.8)
    • and products (product components provided by suppliers)
  • Not considered explicitly
    • testing? – testers tend to be a measurement ‘centre of excellence’
    • development? – developers don’t (design not amenable to measurement?)
practical concerns 1
Practical concerns 1:
  • Fear
    • ‘I don’t want to be measured’
  • It is abstract and can be difficult (Pfleeger)
    • Distinguish between metrics designers and metrics users
    • and train accordingly
  • Where does it fit?
    • Tactical capability or organizational infrastructure, or mix?
practical concerns 2
Practical concerns 2:
  • What should we measure?
    • RTFM
    • SG1
    • Ask what do you* need to know? Why?
  • Who monitors and controls (measures)processes
    • At ML2, at ML3?
  • Who monitors and controls the MA process?
  • Metrics Repository
    • Central/organizaton, or local/project

* ‘you’, perhaps ‘we’, but not ‘they’

common circumstances 1
Common Circumstances 1:
  • Product sizing rarely done, or done well
    • Difficulty with identifying entities and their quantitative attributes
    • Fix: analysts extract identities, counts and attributes of system elements from developer/estimators – start with task based estimation spreadsheet and work upstream, ‘What were they thinking?’
  • Organizational Measurement Infrastructure in place
    • Collects lots of data (if it moves measure it)
    • No traceability back to objectives (SG1 missing) data is orphaned from rationale – can’t reuse
    • Fix: discard collected data without definitions or rationale, then reverse engineer objectives (SG1) for existing data collection systems – usually results in opportunity to shed data collection activity – reduce costs – although rarely taken up
  • Organizational measurement data is unverified – of unknown accuracy
    • used for admin/billing
    • Not credible, known to be invalid (timesheets)
    • Not used by collectors
    • Data is for reporting, not using
    • Fix: verify data – presumes SG1
common circumstances 2
Common Circumstances 2:
  • Good measurement
    • Developed, owned and used locally, within teams
    • Can be undervalued (seems obvious)
    • MA SG1 implicit
  • There is a limited ‘information horizon’
    • Visibility is limited, and may be better that way
    • Measurement data doesn’t travel well
    • ‘Drill down’ is limited – even if it looks like it isn’t
good measurement 1
Good Measurement 1:
  • Purpose clear and understood by collectors, analysts and decision makes
  • Measures are defined (not just described)
  • Data collectors are users (short feedback loops)
  • Accuracy and validity known (as minimal requirement)
  • Can stop collecting when no longer needed
good measurement 2
Good Measurement 2:
  • KISS
    • Minimal arithmetic, especially multiplication and division
    • (this includes percentages)
    • Arithmetic obscures much, reveals little
  • Non parametric approaches
    • Robust, widely applicable to ‘messy’ software engineering data
    • Not the usual statistical approach
  • Consider ‘Exploratory Data Analysis’ (EDA)
    • Tukey
  • Use Graphics
    • But not pie charts
    • Let data show its information content – patterns, trends, outliers
    • Tufte
  • GQMGtutu
    • Goal Question Metric, Graphics – guided by Tufte and Tukey
    • SEI GQ(I)M
  • SPC
    • Later, much later
    • SPC Rule #1, know what you’re doing
slide16
O X F O R D

S O F T W A R E E N G I N E E R I N G

L I M I T E D

9 Spinners Court, 53 West End,

Witney,

Oxfordshire

OX28 1NH

www.osel.co.uk

[email protected]

Tel. +44 (0) 1993 700878

slide19

Empirical

relational

system

slide20

real world

mathematical world

Empirical

relational

system

Formal

relational

system

measurement

slide21

real world

mathematical world

Empirical

relational

system

Formal

relational

system

measurement

mathematics

and

statistics

Results

slide22

real world

mathematical world

Empirical

relational

system

Formal

relational

system

measurement

mathematics

and

statistics

Relevant

empirical

information

Results

interpretation

slide23

real world

mathematical world

Empirical

relational

system

Formal

relational

system

measurement

decisions

and

actions

mathematics

and

statistics

Relevant

empirical

information

Results

interpretation

From Pfleeger 1998

slide24

real world

mathematical world

Empirical

relational

system

Formal

relational

system

refined

measurement

better

decisions

and

actions

mathematics

and

statistics

Relevant

empirical

information

Results

improved

interpretation

slide27
N = 11

Mean of X’s = 9.0

Mean of Y’s = 7.5

Regression line = Y + 0.5X

Standard error of estimate of slope = 0.118

t = 4.24

Sum of squares X – X = 110.0

Regression of sum of squares = 27.50

Residual sum of squares of Y = 13.75

Correlation coefficient = 0.82

R2 = 0.67

_

process capability
Process Capability:

Indices for measuring process goodness

  • Cp = USL - LSL / 6 or 2T / 6
    • Cp < 1 process is incapable
    • Cp > 1 process is capable (6 processes have Cp of 2)
    • does not account for process drift so...
  • Cpk = the lesser of (USL - X) / 3 or (X - LSL) / 3

=

=

ad