test process and test process development l.
Skip this Video
Loading SlideShow in 5 Seconds..
Test process and test process development PowerPoint Presentation
Download Presentation
Test process and test process development

Loading in 2 Seconds...

play fullscreen
1 / 22

Test process and test process development - PowerPoint PPT Presentation

  • Uploaded on

Test process and test process development. Jussi Kasurinen, Software Engineering Laboratory. What were MASTO and ESPA?. ESPA was a 3-year academic project conducted by LUT and Aalto University, funded by Tekes and a group of companies and organizations.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Test process and test process development

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
test process and test process development
Testprocess and testprocessdevelopment

Jussi Kasurinen, Software Engineering Laboratory

what were masto and espa
What were MASTO and ESPA?
  • ESPA was a 3-year academic project conducted by LUT and Aalto University, funded by Tekes and a group of companies and organizations.
  • The focus on the project was on software testing in all different organizational levels.
    • MASTO at LUT was the part focusing on test process and organizational aspects.
    • SQUID at Aalto was the part focusing on the testing work and project-level aspects.

Software-producing organization

Organizational level, one company has one set of these

Project-level work, one company may have several of these

Pof these


Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

test standard and masto publications
Test Standard and MASTO Publications

Journal article “Software Test Automation in Practice: Empirical Observations”

Conference paper “How Test Organizations Adopt New Testing Practices and Methods?”

Conference paper: “A Study on Agility and Testing Processes in Software Organizations”

Covered in article “Software Test Automation in Practice: Empirical Observations”

Conference paper: “Test Case Selection and Prioritization: Risk or Design-based?”

Conference paper “A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard”

Conference paper: “Exploring Quality Concepts in Software Organizations”

Conference paper “Analysis of Problems in Testing Practices”

Test strategy constructs (and layers)

self assessment framework
Self-assessment framework
  • Combination of Test Improvement Model (TIM) (Ericson et al. 1997) maturity levels and ISO/IEC 29119 processes.
  • Two results:
    • General maturity and conformance with the standard model.
    • Process improvement objectives to develop test process.
  • Here is an example of the self-assessment results from one case organization:

Maturity levels from TIM

Processes from ISO/IEC 29119

General maturity/conformance estimation

Individual assessment of each process area, development ideas

Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

other major implications for industry and research
Other Major implications for industry and research
  • 1. The test strategy establishes a framework for testing work at the project level. The following hypotheses promote the development of a test strategy to address the factors important for the testing work:
    • The development of a test plan can be characterised as applying two stereotypical approaches. The first approach promotes a design-based approach, in which the testing work focuses on validating the object under testing. The second approach promotes the risk-based approach, where the testing work focuses on minimising the potential losses caused by the object under testing.
    • There is only a loose association between development methods and test processes. The applied development method does not restrict the practical testing work to any large degree, or require compromises in the test process definitions.
    • The most important aspects in the test process which have positive association with the perceived end-product quality are trust between the customer and producer, a test process which conforms to the self-optimising processes similarly as defined in the ISO/IEC 29119 standard and the communication of the preferred quality characteristics to all of the process stakeholders.
    • In the test process resourcing, the organisations have an average of 70% of their self-defined ―optimal amount of resources and dedicate on average 27% of the total project effort to testing. Based on the study results presented in the literature and the survey data, the test process hindrances are also based on the efficiency factors and test management, in addition to simple resourcing issues.

Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

other major implications for industry and research6
Other Major implications for industry and research
  • 2. The ISO/IEC 29119 test standard is a feasible process model for a practical organisation with the following limitations as regards for applicability:
    • The standard model can be characterised as being overtly detailed in the definition of roles and activities. In the practical test organisations, the boundaries of different levels, processes and roles are less organised than the model presents.
    • The process model is top heavy and places a considerable emphasis on the organisational aspects of the test process. Based on the qualitative analysis, the model defines several responsibilities for the upper management, many of which are performed, in real-life organisations, at the project-level management or at least not as systematically as defined in the model.
    • The current standard model does not include a roadmap or phased process for adopting the model. This hinders the applicability of the model in organisations, as the organisations had difficulties in creating an approach which their existing process could adopt for the concepts presented in the standard.

Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

other major implications for industry and research7
Other Major implications for industry and research
  • 3. The organisations do not actively try out new test methods and prefer a status quo in their test process. The following hypotheses relate to the test process development at the organisational level:
    • The organisations do not test out new testing tools or apply new testing methods unless they have a significant external incentive to do so. Based on the qualitative analysis, these incentives are things like the current state of the existing process or business needs in the operating domain.
    • The organisational test process may have feedback processes in place to allow continuous development, but in practice, the organisations tend to disregard the evidence of process enhancement needs if the existing process still performs at least at an acceptable efficiency.
    • In test process development, the organisations need a way of relating their existing process to the proposed changes to understand the objectives, and more pragmatically, the requirements the process improvement needs to succeed.

Jussi Kasurinen, Software Engineering Lab. LUT Information Technology

software testing in the cloud

Software Testing in the Cloud

Leah Riungu-Kalliosaari


Empirical study aimed at understanding how organizations can successfully use the cloud for testing

Cloud computing: A model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, servers, storage, application and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction (NIST definition)

STaaS: A model for software testing whereby testing of an application is provided as a service across the internet.

Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used

current situation
Current Situation

The trend towards cloud services is growing

Traditional desktop applications  online software services e.g. SaaS, StaaS

Mixed reactions  Generally positive

Positive – due to e.g. cost reduction, flexibility, performance, access to global markets

Negative – due to concerns regarding e.g. security, domain knowledge, test data management

Neutral – mainly explorative approach

implications on testing
Implications on Testing

With the cloud becoming more common, the quality expectations are growing

New issues and complexities surrounding these services will need to be addressed

Testing will become more important

Need for testing increases


3. Testing the cloud

2. Testing environments

in the cloud

1b. Non-SaaS software

1a. SaaS


Testing in the Cloud

  • The system or application under test is available online
  • Testing infrastructure and platforms are hosted in the cloud (Including crowdsourcing/Human as a Service-(Haas))
  • Testing of the cloud itself

Facets of testing in the cloud

what next
What next?

To better understand how cloud based services can be achieved with intended quality

Identify the most important quality requirements for cloud-based services

Provide recommendations for achieving overall intended quality of cloud-based services

Difference between intended and achieved quality

What are the tradeoffs, and what is the optimal balance.

HOW Through interviews during Autumn

Collect data to uncover the needs, problems, and practices requiring improvement

Learn how OUs develop software, test software and build quality

questions and comments
Questions and comments

What would you like to achieve during this project?

What is most important for your organization?

What kind of results?

What kind of collaboration do you expect?

Suggestions/improvement proposals regarding our research approach so far.


To show how software development, software testing and intended quality depend on one another.

The research area includes, in addition to traditional software development and service models, also the emerging XaaS (Everything as a Service) architectures, technologies and service models.

The project results help the participating companies in improving the efficiency of their quality management and software testing and hence the efficiency of their software development as a whole.

Testing techniques

Testing as a service

research problem
Research Problem

Intended Software Quality

Software Development

  • Software Products
  • ISO/IEC 12207
  • ISO/IEC 15504
  • New Services
  • SaaS

ISO/IEC 25010, Software Product Quality

Software Testing

ISO/IEC 29119, Testing Spice, IEEE Std 1012


Assessment framework through which each OU is evaluated

research problem18
Research Problem

The three-level view to software development:

Software quality

Specify the intended quality

Quality attributes & metrics according to the ISO/IEC 25010 quality model.

Software development

Product or service oriented development

Traditional or SaaS software

Software development methods, software life cycle models

ISO/IEC 12207, 15504 software development process improvement

research problem19
Research Problem

Software testing

Ensure that the software development process produces the targeted quality

Testing strategy & policy → testing processes → testing techniques

ISO/IEC 29119, IEEE Std 1012,

Testing process assessment, Testing SPICE

research problem20
Research Problem

The framework used as a ”grid” through which each OU is assessed.

Intended quality vs. achieved quality.

Top-down and bottom-up analysis (testing work  project management  organizational level).

The assessment framework is defined through international software engineering standards.

We need to anchor the assessment to a specific, concrete view on what quality, software testing and software development are.

Based on standards, the model remains coherent throughout the assessment.

Project results and usability; contribution to standardization work.

The results include improvement proposals for the OU’s.

Data collection methods include interviews and surveys.

work division entire project
Work division, entire project

3 year project

12 months / work package; WP 1 starts 8/2011

WP1: Requirements for intended quality

Identifies the quality requirements and attributes that the OU's perceive as key for achieving the intended quality.

Defining intended quality, the importance of different quality characteristics in achieving it, traditional vs. SaaS

WP2: Building the intended quality

Achieved vs. Intended quality vs. the software development process

Action research

WP3: Assessment

How the OU’s can improve their quality management and software testing to achieve the intended quality more efficiently?

WP 1

Begins in August 2011

Quality requirements and the intended quality, software development and software testing

Study units are OU's.

What are the quality requirements, and hence the quality attributes, that the OU perceives as key for achieving the intended quality (level & characteristics)?

Are some quality attributes more important than others when determining whether the targeted quality has been achieved? Do OU's or projects differ here?

Does SaaS development emphasize different kind of targeted quality (level, characteristics) to traditional software development?

The data are collected from the participating companies


Additionally, supplementary data from the previous MASTO project may also be used.

The findings are published as scientific publications during the WP.