Software testing research. Ossi Taipale November 2011 Lappeenranta University of Tech. Contents. Background Research process Research methods Research projects, research objectives and results: Basic research of software testing, ANTI-project
of SW testing,
2004 - 2007
of SW testing,
2008 - 2011
2011 - 2014
SW testing in
ISO/IEC and IEEE :
29119 and Testing Spice,
Selection of standards and constructs: e.g. ISO/IEC 12207, 15504 and 29119
Beliefs of the essence of phenomena under investigation
Concrete selections of the study: Delphi, survey, and grounded theory methods
Beliefs of the suitable method
The objective of the Delphi method is to achieve the most reliable consensus of opinion of a group of experts. Delphi method can be used in finding good arguments about an ongoing process (Directing the study).
The survey method is used to gather information about feelings, motivations, plans, and beliefs. Tools for gathering such information are usually questionnaires or interviews (Fink & Kosecoff 1985).
Grounded theory (Strauss & Corbin 1990). The objective of a qualitative study is rather to find or reveal facts than to prove existing theorems. Strauss and Corbin (1990) define qualitative research as any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification.
1. Finding and Ranking Research Directions for Software Testing, EuroSPI, Budapest, Taipale, O., K. Smolander, H. Kälviäinen.
2. Cost Reduction and Quality Improvement in Software Testing, SQM, Southampton, UK, Taipale, O., K. Smolander, H. Kälviäinen.
3. A Survey on Software Testing, SPICE Conference , Luxembourg, SPICE ,Taipale, O., K. Smolander, H. Kälviäinen (2006).
4. Factors Affecting Software Testing Time Schedule, ASWEC, Sydney, Australia, Taipale, O., K. Smolander, H. Kälviäinen (2006).
5. Improving Software Testing by Observing Practice, ISESE, Rio de Janeiro, Brazil, Taipale, O., K. Smolander.
6. Observing Software Testing Practice from the Viewpoint of Organizations and Knowledge Management, ESEM, Madrid, Spain, Taipale, O., K. Karhu, K. Smolander.
7. Triangulating Testing Schedule Over-runs from Knowledge Transfer Viewpoint, Lappeenranta University of Technology, Research Report 104, Taipale, O., K. Karhu, K. Smolander .
8. Outsourcing and Knowledge Management in Software Testing, EASE, Staffordshire, UK, K. Karhu, O. Taipale, K. Smolander.
Organizational level, one company has one set of these
Project-level work, one company may have several of these
Journal article “Software Test Automation in Practice: Empirical Observations”
Conference paper “How Test Organizations Adopt New Testing Practices and Methods?”
Conference paper: “A Study on Agility and Testing Processes in Software Organizations”
Covered in article “Software Test Automation in Practice: Empirical Observations”
Conference paper: “Test Case Selection and Prioritization: Risk or Design-based?”
Conference paper “A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard”
Conference paper: “Exploring Quality Concepts in Software Organizations”
Conference paper “Analysis of Problems in Testing Practices”
Test strategy constructs (and layers)
Things that do not affect the perceived quality that much even if they seem to have a big influence
Maturity levels from TIM
Processes from ISO/IEC 29119
Individual assessment of each process area, development ideas
General maturity/conformance estimation
1. Test Case Selection and Prioritization: Risk-Based or
Design-Based? Jussi Kasurinen, Ossi Taipale and Kari Smolander, ESEM
2. A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC 29119 Test Standard, Jussi Kasurinen, Per Runeson, Leah Riungu and Kari Smolander, EuroSPI
3. Software Test Automation in Practice: Empirical
Observations, Jussi Kasurinen, Ossi Taipale, Kari Smolander, AiSE
4. Exploring Perceived Quality in Software Organizations, Jussi Kasurinen, Ossi Taipale, JariVanhanen and Kari Smolander, IEEE
5. Analysis of Problems in Testing Practices, Jussi Kasurinen, Ossi Taipale and Kari Smolander, APSEC
6. A Study on Agility and Testing Processes in Software Organizations, VesaKettunen, Jussi Kasurinen, Ossi Taipale, and Kari Smolander, ISSTA
7. How Test Organizations Adopt New Testing Practices and Methods? Jussi Kasurinen, Ossi Taipale and Kari Smolander, TAICPART
8. Exploring the Perceived End-Product Quality in Software-Developing Organizations, accepted for publication in International Journal of Information System Modeling and Design, IGI Global,Jussi Kasurinen, Ossi Taipale, JariVanhanen and Kari Smolander.
The trend towards cloud services is growing
Traditional desktop applications online software services e.g. SaaS, StaaS
According to a 2009 Gartner report, cloud computing changes the way software is delivered and used. This change means that traditional licensing of software seems to loose ground where as hiring of software services seems to be on the rise.
Mixed reactions Generally positive
Positive – due to e.g. cost reduction, flexibility, performance, access to global markets
Negative – due to concerns regarding e.g. security, domain knowledge, test data management
Neutral – mainly explorative approach
With the cloud becoming more common, the quality expectations are growing.
New issues and complexities surrounding these services will need to be addressed.
Testing will become more important
Need for testing increases
Testing is seen as arena of cloud computing where the barriers to entry are relatively low. Testing in the cloud simply entails the use of computing resources and models in the cloud to perform testing. Cloud-based testing is provided on-demand and billed on pay-per-use basis, so that the user pays only for the resources they have used.
2. Testing environments
in the cloud
1b. Non-SaaS software
Testing in the Cloud
Facets of testing in the cloud
Develop understanding of cloud computing
Cloud computing is increasingly becoming a feasible choice for testing.
Carry out pilot projects
We recommend that organizations carry out pilot projects that enable them to fully explore the potential benefits.
Come up with elaborate strategies
Cloud testing vendors as well as testing and quality assurance consulting firms will be called upon to offer advice and direction.
Enhance team interaction and prepare for complexities
Organizations need to be prepared for additional testing brought about by the complexities and new requirements for cloud-based applications and systems.
Enhance co-operation between research and industry
These include, among others, application issues e.g. the types of applications best suited for testing in the cloud; management issues like how to organize the human resources for cloud-based testing (e.g. crowdsourcing); and legal and financial issues such as the management of test data across different global jurisdictions and how to device appropriate pricing models.
To better understand how cloud based services can be achieved with intended quality
Identify the most important quality requirements for cloud-based services
Provide recommendations for achieving overall intended quality of cloud-based services
Difference between intended and achieved quality
What are the tradeoffs, and what is the optimal balance.
Publications from the Testing in Cloud – project (2010-2013)
1. Research Issues for Software Testing in the Cloud, Leah MuthoniRiungu, Ossi Taipale, Kari Smolander, 2nd IEEE International Conference on Cloud Computing Technology and Science.
2. Software Testing as an Online Service: Observations
from Practice, Leah MuthoniRiungu, Ossi Taipale, Kari Smolander, Third International Conference on Software Testing, Verification, and Validation Workshops.
3. Testing in the Cloud: Exploring the Practice, IEEE Software, Leah Riungu-Kalliosaari, Ossi Taipale, Kari Smolander.
To show how software development, software testing and intended quality depend on one another.
Traditional software development and service models
Emerging XaaS (Everything as a Service) architectures, technologies and service models.
The project results help the participating companies in improving the efficiency of their quality management and software testing and hence the efficiency of their software development as a whole.
Testing as a service
Intended Software Quality
ISO/IEC 25010, Software Product Quality
ISO/IEC 29119, Testing Spice, IEEE Std 1012
OU’s are evaluated through an assessment framework
The framework used as a ”grid” through which each OU is assessed.
Intended quality vs. achieved quality.
Top-down and bottom-up analysis (testing work project management organizational level).
The assessment framework is defined through international software engineering standards.
We need to anchor the assessment to a specific, concrete view on what quality, software testing and software development are.
Based on standards, the model remains coherent throughout the assessment.
Project results and usability; contribution to standardization work.
The results include improvement proposals for the OU’s.
Data collection methods include interviews and surveys.
3 year project
12 months / work package; WP 1 starts 8/2011
WP1: Requirements for intended quality
Identifies the quality requirements and attributes that the OU's perceive as key for achieving the intended quality.
Defining intended quality, the importance of different quality characteristics in achieving it, traditional vs. SaaS
WP2: Building the intended quality
Achieved vs. Intended quality vs. the software development process
How the OU’s can improve their quality management and software testing to achieve the intended quality more efficiently?
Study units are OU's.
What are the quality requirements, and hence the quality attributes, that the OU perceives as key for achieving the intended quality (level & characteristics)?
Are some quality attributes more important than others when determining whether the targeted quality has been achieved? Do OU's or projects differ here?
Does SaaS development emphasize different kind of targeted quality (level, characteristics) to traditional software development?
Data collection for the first qualitative study September – November / 2011.
Additional data from other projects also incorporated.
Data analysis may produce first leads.
Second data collection round will constitute quantitative studies based on round 1 observations.
1. Observations on eBusiness implementation capabilities in heterogeneous business networks", 11th IFIP International Conference on e-Business, 2011, Kaunas, Lithuania Pesonen T., Smolander K.