Sindu Vijayan – Principal QE Vidhya Nandhini Paramanathan – Sr Principal QE Manhattan Associates

1. Cognition in Testing Cognitive Solutions Sindu Vijayan – Principal QE Vidhya Nandhini Paramanathan – Sr Principal QE Manhattan Associates

2. Abstract • The technology advancements over the last decade has made storage and connectivity so affordable that the Digital world right now is transforming at a speed unimaginable. No field is untouched by this digital revolution and so is software testing. The enormous data demands the usage of cognitive or machine learning logic to derive meaningful information. The once niche of programs that use cognitive logic better termed as “Non-testable Programs” is now becoming an integral part of any business solution. • This article touches upon the best practices, techniques and challenges in testing Cognitive Solutions, taking examples from supply chain. While traditional testing techniques can be used in such applications where ever applicable; often assessing the correctness of the solution remains a challenge. • While discussing about “Testing the non-testable programs” The article touches upon the following areas at a broader level: • Effectiveness of the Algorithm/Logic • Correctness of Implementation • End to End Solution Accuracy • Quality Bench Mark • Cognitive solutions are often about optimization or better insights over the traditional models. There are no fixed boundaries or pass fail criteria. Taking an example of forecast projections, we use what we call the “Time Travel Technique” – where validations would be conducted against the actuals by moving back in time. Other measures that assess the accuracy with key performance indicators can also help test the effectiveness. The closer you are to reality the better the predictions. • There are no one-fits-all solutions available while addressing the “Non-testable programs”. Every such solution needs a unique test plan and methods to ensure that the primary objective of the Quality function is met – The solution provided meets the customer needs.

3. The World Right now • The technology advancements over the last decade has made storage and connectivity so affordable that the Digital world right now is transforming at a speed unimaginable. • 50x Data growth expected from 2010 to 2020 • Data doubles every 2 years or less • The potential information the data holds is enormous • The Potential Impacts!!! • Technology is no longer the differentiator! • Business Intelligence Transformation demands: • Right information which steers the business in the right direction. • Detection of Trends early enough for better response time. • The emerging trends are capture over a period of time. • Rise of Cognitive Computing. • AI – Once a luxury is now a necessity! • An integral part of software solution is an integral part of testing too !!

4. Cognitive Solutions what do they do? • Add optimization to the otherwise traditional approach • Can be used to provide better insights • Mostly statistical in nature Major Challenges in Testing the Enormous Data • Testing the Non-Testable Programs • No Test oracles applicable • No direct way of assessing the results of most such solutions • The level of optimization is different from solution to solution • Testing Enormous Data • Identifying the right data for validation • Validate the Effectiveness of the Solution

5. The big deal about data – The solution provided is as good as the data! • The Data determines the effectiveness and accuracy of the solution. • When a cognitive solution feeds into another; the effects of poor data can be multifold. • Assume that the features(predictors) are accurately measured.

6. The Right Data for validation • Generate a Standard Data Set • Create Data set from snapshots of proven data sources. • Data crafted carefully with the “actuals” captured. • Capture impact of different flavors of data. • Data is Static in nature. • Sales matrix gathered from different verticals over a period of time – Eg: a sales forecasting system • Real-Time Data feeds – See what the customer sees • Real-time production like data. • Testing solutions that analyze trends or customer sentiments would benefit from this approach. • The actual results are not readily available. • It is a challenge to assess the effectiveness of the solution. • The patterns may be diluted as compared to a standard data set manually crafted • The two methods serve two different purpose. The most beneficial would be to use both the techniques in the order above – if applicable.

7. Validate the effectiveness of the solution Test the Non-Testable: How optimized are the results with and without the cognitive solution? The Test methodologies: • Effectiveness of the Algorithms • Correctness of the implementations • End to End Solution Accuracy • Quality Bench Mark The level of optimization required for each solution is unique!

8. Effectiveness of Algorithms • Compare algorithms (Choose the right model or algorithm) • Compare Prototypes with other Algorithms • Utilize open source tools to compare results • Compare the calculated results with the actuals available • Error/Variance Calculations • Accuracy, Precision, Recall for classification problems • Cross Validation Techniques • Segregate the data as training and test set and capture the variance

9. Effectiveness of Algorithms • Data Seeding Testing Techniques • The technique can be used to artificially inject a pattern. • Evaluate the response of the system. • Identifies if the cognitive solution can capture changing trends/react to changes • Eg: Association rule learning where injecting large no: of records of a particular • association would identify the newly fed association with a strong affinity score. • Identify the matrices that can support the outcome of the algorithm.  • Eg: Are there any measures of interestingness that can help evaluate Data Data SQUINCH/PG1

10. Correctness of Implementation • While using prototyping for the algorithmic logic. A well tested prototype eases the implementation testing. • Bench mark testing against prototype. • Benchmarking against the same data is the way to go. • Automation Testing can be used. • Validate the OTS solutions • Off-The-Shelf solutions are becoming popular for cognitive solutions where the component with the algorithmic logic can be purchased. • Challenges: • The algorithmic logic is usually Intellectual Property or not shared. • Optimization of the Cognitive logic is a challenge. • However, we could assess the effectiveness by benchmarking techniques.

11. End to End Solution Accuracy • The Rule of Thumb: • Compare the results with and without the cognitive solution. • The extend of improvement is the gauge! • Time Travel Technique: • Collect the data with actuals over a period of time. • Go back in time make the predictions. • Compare the predictions to the actuals • Taking the example above: • Evaluate KPI’s like loss sales and Service Level Agreements • A well optimized cognitive solution should reduce the Lost Sales and there by improve the SLAs. How Close?

12. 2013 2015

13. Conclusion Testing the ‘Non-Testable programs’ remains a challenge No one-fits-all test solution available. Every solution needs a unique test plan that suits the problem it tries to address. The Above mentioned are techniques that can be applied as per requirements Primary objective of a Quality Function – Ensure the solution meets the customer needs.

14. References & Appendix https://academic.oup.com/comjnl/article/25/4/465/366384/On-Testing-Non-Testable-Programs

15. Author Biography Sindu Vijayan: Principal Quality Engineer with Manhattan Associates. Sindu leads the testing activities in the Inventory suite of product which are associated with Planning, forecasting and optimization of supply chain inventory. She is also responsible for Machine Learning initiatives within the QE department. With an over all industry experience of 14+ years she has been associated with Manhattan for 11+ years. Sindu is a Masters in computer application from Bharathiyar university and also holds a Post Graduate Diploma in Retail Management. She is a QAI Certified Manager in Software Quality. Vidhya Nandhini Paramanathan: Vidhya is a Sr Principal Quality Engineer with Manhattan Associates. She leads the testing activities in the Inventory suite of product which are associated with Planning, forecasting and optimization of supply chain inventory. She also has domain expertise in ERP, Healthcare. A Computer Engineering graduate from PSG College of Technology, Vidhya specializes in Automation and Performance Engineering. She has an overall 14+ years of industry experience and has been associated with Manhattan for 11+ years.

16. Thank You!