OpenStack QA Project- Roles & Its Key Activities

1. OpenStack QA Project- Roles & Its Key Activities Ghanshyam Mann Ken’ichi Ohmichi NEC

2. Agenda • Who are we ? • Overview Of OpenStack • Overview Of OpenStack QA • OpenStack QA Projects – Overview & Their Key Activities • Tempest • Tempest-lib • DevStack • Grenade • Hacking • Summary

3. Whoare we ? • Ghanshyam Mann • Software developer from NEC • OpenStack community member since 2012 • Core developer of Tempest and active contributor in Nova. • Ken’ichi Ohmichi • Software developer from NEC • OpenStack community member since 2012 • Core developer of Nova and Tempest

4. Overview of OpenStack

5. OpenStack • Open source software for creating private and public clouds. • OpenStack software controls large pools of compute, storage, and networking resources throughout a datacenter, managed through a dashboard or via the OpenStack API.

6. OpenStack • OpenStack Community has grown to lot bigger level. • Around 22K developers across over 160 countries, who have created around 3M LOC. • This is one of the fastest growing open source community.

7. OpenStack • OpenStack provides lot of services over cloud through many projects. • List Of OpenStack Projects-

8. Upstream Stability Quality of OpenStack ? OpenStack QA

9. OpenStack QA Project • A project that works on testing and quality assurance efforts across all OpenStack projects. • Main mission is to develop, maintain, and initiate tools and plans to ensure- • The upstream stability • Quality of OpenStack

10. OpenStack QA Project • The OpenStack QA project is made of multiple projects where each run independently towards the goals of the project. • Each project maintains a separate core team based on who provides reviews in those projects.

11. Tempest - OpenStack Integration Test Suite • This is a set of integration tests to be run against a live OpenStack cluster. • Tempest is designed to be useful for a large number of different environments. This includes being useful for gating commits to OpenStack core projects, being used to validate OpenStack cloud implementations.

12. Tempest - OpenStack Integration Test Suite… • Type of tests in Tempest-

13. Tempest - OpenStack Integration Test Suite… • API tests • API tests are validation tests for the OpenStack APIs. • Make sure stability and backward compatibility of APIs. • Scenario tests • Scenario tests are complex “through path” tests for OpenStack functionality. • Scenario tests interact with many services and tests cross projects features. • Stress tests • Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks.

14. Tempest - OpenStack Integration Test Suite… • Third Party API tests • Third party tests are tests for non native OpenStack APIs that are part of OpenStack projects. • An example is that Nova Compute currently has EC2 API support in tree. • Those should be kept separate from the normal OpenStack validation. • Unit tests • Unit tests are the self checks for Tempest.  • They provide functional verification and regression checking for the internal components of Tempest.

15. Tempest - OpenStack Integration Test Suite… • Key Activities-

16. Tempest - OpenStack Integration Test Suite… • Focused Area- • Move common testing framework/modules to Tempest-lib. • Tempest Bug Triage and Fix • Since last year we started Bug Triage rotation for QA bugs. • This is helping us to keep new bug count less and provide fast response to logged bugs. • Recently in OpenStack summit, we discussed about providing External plugin interface.

17. Tempest-lib - OpenStack Functional Testing Library Motivation Behind Tempest-lib

18. Tempest-lib - OpenStack Functional Testing Library • As number of OpenStack projects keep growing, some issues comes up regarding their testing - • It becomes difficult to tests all projects in Tempest. • There will be duplicate effort for developing functional tests in each Projects. • To solve such issues, its better to provide common testing framework through some library so that projects can develop their own tests by using common framework.

19. Tempest-lib - OpenStack Functional Testing Library • A library to assist in creating functional or integrated test suites for OpenStack projects. • Tempest-lib is a library of common functionality that was originally in Tempest. Common Testing Framework Tempest Tempest-lib

20. Tempest-lib - OpenStack Functional Testing Library… • Key Activities-

21. Tempest-lib - OpenStack Functional Testing Library… • Project Functional tests can be developed using Tempest-lib. • Tempest tests started using Tempest-lib. Tempest tests Project-1 Functional tests Tempest-lib Project-n Functional tests

22. DevStack - OpenStack development environment • DevStack is an opinionated script to quickly create an OpenStack development environment. • It can also be used to demonstrate starting/running OpenStack services and provide examples of using them from a command line. Developer Work on OpenStack Install OpenStack Development Environment DevStack

23. DevStack - OpenStack development environment… •  Key Activities-

24. DevStack - OpenStack development environment… • Supported Components -

25. DevStack - OpenStack development environment… • Focused Area - • Making Nova v2.1 API as default • Nova has two set of APIs, v2 and v2.1. Currently DevStack configure v2 API as default and provide option to configure v2.1 API on /v2 endpoint in targeted environment. As v2 API will be deprecated in future, it will be better to make v2.1 as default Nova API in DevStack to tests v2.1 API thoroughly. • Make external plugins for projects currently having their scripts in DevStack tree.

26. Grenade – OpenStack Upgrade Testing • Grenade is an OpenStack test harness to exercise the upgrade process between releases. • It uses DevStack to perform an initial OpenStack install and as a reference for the final configuration. • Currently Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift, Ceilometer, Horizon and Ironic in their default DevStack configurations.

27. Grenade – OpenStack Upgrade Testing… • How Grenade ensure that any Code modification does not break Upgrade. Upgrade From KILO to Master • OpenStack • Master Branch • OpenStack • Target Branch • (Master) • OpenStack • KILO 2 4 Merge Do Not Merge FAIL Grenade 3 PASS Code modification 1 Tests changes for upgrade through grenade gate job Developer

28. Grenade – OpenStack Upgrade Testing… • Key Activities-

29. Hacking • Hacking is a set of flake8 plugins that test and enforce the OpenStack Style Guidelines. • Most of the additional style guidelines that OpenStack has taken on came from the Google Python Style Guide. • http://google-styleguide.googlecode.com/svn/trunk/pyguide.html • Since then, a few more OpenStack specific ones have been added or modified.

30. Conclusion & Key Points • OpenStack QA helps to validate your OpenStack Cloud. • Migrate more testing common code from Tempest to Tempest-lib. • Tempest-lib to avoid duplicate test development effort. • External Plugins for DevStack, Tempest, Grenade. • More Info - • https://wiki.openstack.org/wiki/QA • https://wiki.openstack.org/wiki/Meetings/QATeamMeeting • IRC - #openstack-qa