chicago quality assurance assocation n.
Skip this Video
Loading SlideShow in 5 Seconds..
Chicago Quality Assurance Assocation PowerPoint Presentation
Download Presentation
Chicago Quality Assurance Assocation

Loading in 2 Seconds...

play fullscreen
1 / 31

Chicago Quality Assurance Assocation - PowerPoint PPT Presentation

  • Uploaded on

Chicago Quality Assurance Assocation. Secrets of Successful Automation Projects Presented by Paul M. Grossman Software Quality Assurance Analyst QTP Automation Framework Designer. Portfolio of Custom Software Software Development: .Net , Java, C# eCommerce (Oracle ATG)

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Chicago Quality Assurance Assocation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
    Presentation Transcript
    chicago quality assurance assocation

    Chicago Quality Assurance Assocation

    Secrets of Successful Automation ProjectsPresented byPaul M. GrossmanSoftware Quality Assurance AnalystQTP Automation Framework Designer

    portfolio of custom software software development

    Portfolio of Custom Software

      • Software Development: .Net, Java, C#
      • eCommerce (Oracle ATG)
      • Mobile Applications: iOS, Android, Windows Mobile/WP7, Blackberry
    • Quality Assurance Testing
      • LoadRunner, QTP, HP ALMS
      • Custom Tool Development and Frameworks
      • Performance Testing
      • Mobile Testing: iOS, Android, Windows Mobile, Tablets
    • Developed Applications in the following sectors
      • Games & Entertainment
      • Healthcare
      • Social Networking
    paul m grossman
    PaulM. Grossman
    • 11 years automation experience with HP Tools
    • HP Quality Center 9.0 Certified Product Consultant
    • HP Quick Test Professional 9.0 Certified Product Specialist
    • Beta Tester of QTP 9.5 – 11.5
    • Speaker at Mercury/HP Events
    • Became Automation Engineer due to Perfect Timing!

    Manager caught me practicing magic tricks while waiting for system to reboot.

    session goals
    Session Goals
    • Key success factors in test automation today
    • Demonstrating test automation ROI
    • Effective framework design techniques
    • Successful script design for Agile environments
    automation challenges
    Automation Challenges
    • Lack of communication of changes/details to testing group
    • Lack of understanding of should be automated
    • Lack of variation from plan in test scripts
    • Insufficient standards
    • Insufficient Analysis & Design
    • “Silver bullet syndrome” (automated testing is the “silver bullet” that will end all testing problems on a project)
    • Lack of change control
    case study
    Case Study

    Automation Project Failure at a Medical Device Manufacturer

    Year 1: QA Manager wants automated testing.

    Buys WinRunner, accepts offer at new company.

    Year 2: New QA Manager wants automated testing.

    Hired Lead Developer (C++), given an Assistant (VBscript).

    Goal for automation: 100% coverage.

    Year 3: Lead Developer moves to new department.

    Assistant told automation is high-priority, spare-time project.

    Year 4: QA Manager moves to new division

    Assistant retires.

    Results: 0 Defects found. Negative Return On Investment.

    Unsuspecting manual tester assigned to automation team.

    New Division Manager expresses interest in


    Manager promotes me to Team Lead… of defunct Automation Project.

    top 5 project pitfalls
    Lack of research & planning

    Lack of apparent progress

    Lack of experienced resources

    Not a full time project

    No buy-in

    100% automation goal resulted in development in all product areas

    20% development in all areas with nothing to show

    Project handed over to assistant with no product training

    Misperception: FT not needed because automation by it’s definition is automatic, right?

    Testers & developers fueled rumors of tool’s stability

    Top 5 Project Pitfalls
    mitigation strategies
    Lack of research & planning

    Lack of apparent progress

    Lack of experienced resources

    Not a full time project

    No buy-in

    10% Automation goal

    POC Pilot Project


    Full-time commitment

    Repeat defects manually on a non-automated system

    Mitigation Strategies
    tips and tricks avoiding 100 automation trap
    Tips And TricksAvoiding "100% Automation" Trap
    • 90% Automation can be approached but it is generally not cost effective
    • New releases will have more functionality with constantly decreasing code coverage
    • A percentage of the code could be unreachable code, such as uncalled functions
    • Pick one: Functionality or Browsers Do the math: You won't have time for everything!

    20 - 40% Automation makes a significant impact on the testing process.

    Math: App Developers outnumber your team?

    Do they have 6-18 Months head start?

    Do they have more than one tool?

    case study 1
    Case Study

    A - HA !

    The Pilot Project

    • Automate the Acceptance Test (“Smoke Tests”)
      • Short tests, but too wide of coverage
    • Commitment to six weeks of full-time development
      • Became four weeks, 10 days of manual testing
    • Used “Record & Playback” to create the entire test script
      • Overwhelming maintenance
    • Demonstration to follow at the end of the project
      • Management convinced they purchased the right tool
      • Asked for training on Winrunner

    – Lessons Learned

    Request to get training approved!

    case study 2
    Case Study

    Functional Decomposition

    • Crafted functions from Demo Script to create “TimeSaver” utilities
    • Wrote discrete state change functions: TRUE, Worked! FALSE, Uh-Oh!
    • Modular functions made maintenance manageable
    • Automated the common interfaces of two additional products
    • Proposed automating easy features first

    –ShowStopper #1:Memory Leak. First 13-hours of endurance test.

    –Developers stop blaming tool, now ask for ad-hoc scripts

    Request to attend Mercury World Conference Approved!

    estimated return on investment
    Estimated Return On Investment

    “Best Bang For The Buck”: Project TimeSavers

    • Automate tedious manual tasks
      • Create 64 Annotations - 30 Seconds
      • Perform 128 End to End Procedures - 30 Minute Drive Space Test
      • Register 128 New Patients - 30 Minute Boundary Test
    • These tests were small, little maintenance
    • Gets testers on your side, reduces their “pain”
    • Scripts were easy to change into reusable functions

    First Quantifiable Return On Investment: $40 (1 Hour)

    Final estimated ROI for the company:$2 Million

    estimated return on investment 1
    Estimated Return On Investment

    Manual Tester: $40 Avg. hourly wage, including overtime and benefits.

    estimated return on investment 2
    Estimated Return On Investment

    ROI Formula: Estimate Equivalent Cost of Automation

    How much would it cost to hire someone to do this work manually?

    Estimated hourly wage (Plus OT & Benefits): $40Per Hour Average

    Determine hours of runtime

    - 10 hour runs, 5nights a week 10 x 5= 50 Hours per system

    - on 3systems 50 x 3 = 150Hours per week


    150x $40 = $6,000

    Extrapolated to the end of the yearX 52 Weeks



    Four years of run time: $1,248,000

    estimated return on investment 3
    Estimated Return On Investment

    What would some of the worst defects found by automation have cost the company had they gotten to the field?

    • Case Study Examples
      • ShowStopper #2: Transient Boot Up Failure
        • Replicated with automation and high end UPS
      • ShowStopper #3: Data Transfer Problem
        • Duplicated issue within five days.
        • Reason: Related to Network Load limitation.
    • We pegged the ROI of the worst detected defects by estimating the cost of a single lost customer sale: $100,000
    • Potential revenue loss from 7 ShowStoppers: $700,000
    • Plus 1.2 Million not paid to a manual tester: 1.9 Million Est. ROI

    Used Functional Test Tool to uncover Network Load issue?

    estimated return on investment that was then this is now
    Estimated Return On InvestmentThat was then…this is now

    What would some of the worst defects found by automation have cost your companyhad they gotten to the field?

    • ShowStopper #1: Credit cards fail to authenticate?
      • Store is closed world wide.
    • ShowStopper #2: Search path leads to infinite loop?
      • Customers frustrated with site jump to competitor.
    • ShowStopper #3: 12-second+ response time?
      • Customers abandon order
      • Is customer support being effected?
      • Slows down automation code development and debugging
      • Slows down development programmers
      • Slows down manual testers

    What if it's Dev, QA or Staging Environment?

    let s take a break
    Let's take a break!

    When we come back…

    • Tips And Tricks
    • Dealing with Failure
    • What is a Framework anyway?
    what is a framework framework options






    Lexical Analyzer


    What is a Framework?Framework Options
    what is a framework
    What is a Framework?
    • Driver
      • Loads & Executes Multiple Tests
      • Facilitates Result Reporting
    • Object Repository or Identification
      • Identifies objects by Class and Property Values
      • Uses Regular Expressions for dynamic values
    • Data
      • Pre-Defined
      • Generated
    • Recovery Mechanism
      • Browser closes
      • Object invalid
      • Error generated

    A framework is a set of techniques:

    • Reduce Maintenance
    • Abstract Complexity
    • Extend Coverage
    tips and tricks initial project design
    Tips And TricksInitial Project Design
    • Return To Home State Functionality
      • Get back to the starting point from anywhere in the application, even if it is not running.
    • Navigationand Population
      • Low level custom actions to Click buttons, Open menus, Selectitem fromlists.
    • getValidObject (sClass, sName)
      • A single function that returns a valid object reference from many close matches.
      • Object exists (height, width & Abs_x > 0), is enabled, above underlay, closest to text.
      • Uses Descriptive Programming or xPath to identify object
    • Tokens
      • Stores reusable values, such as an Order Number
      • Can return intrinsic values like Today's Date.
    • Assume Failure, Prove Success
      • Use a global variable & Return Codes for overall success. This creates a single exit point for script execution.
    • Modify, Recover, Repeat

    HomeState "^QA_URL", "Admin" 'URLToken, UserRole

    EnterInto "Order No.", "^dt_OrderNumber" 'Token Lookup

    SelectFrom "Month", "^ThisMonth" 'Intrinsic Token Replacement

    Click "Generate New Order"


    VerifyText "Failed to submit order"

    CaptureAfter "Order Number:", "^OrderNo" 'Token capture for reuse


    Cancel_New_Order "^OrderNo"



    tips and tricks modular design
    Tips And TricksModular Design
    • Functions… but No Subs
      • Don’t waste time deciding which to use
      • Subroutines have no built in functionality for return codes
      • Sub FileExists(strFile, rc) ‘ Return Code Workaround
    • All Functions need Return Codes
      • FileExists = TRUE
      • IF FileExists(strFile) THEN ProcessFile (strFile)
    • Use Option Explicit in VBscript
      • Reduces errors due to variable misspellings
    • Three Level Keyword Framework
      • Components (Param1, Param2)
        • Keywords (Param1, Param2)
          • All other supporting functions

    Test Scenario# 3

    Purchase Order

    With Discount Code

    and Pay with Declined

    Credit Card



    CLICK "Ok"

    tips and tricks global variables and dictionaries
    Tips And TricksGlobal Variables and Dictionaries
    • Minimize global variables:
      • gloObject: Reference set by GetValidObject
    • Use a Dictionary Object
      • Global
      • No Naming Convention
      • Stores Object References
      • Can be Persistent!
      • Case Sensitive Key Names
    tips and tricks dealing with failure
    Tips And TricksDealing with Failure

    dicFramework("AlreadyFailed") = FALSE

    Function CLICK (sButtonName)

    If dicFramework("AlreadyFailed") = TRUE Then

    Exit Function

    End If

    blnFound = getValidObject "WebButton", sButtonName'Sets gloObject

    If blnFound = TRUE then



    dicFramework("AlreadyFailed") = TRUE

    End If

    End Function

    That's an odd naming convention?

    "glo" = "Global Object

    tips and tricks dealing with failure 1
    Tips And TricksDealing with Failure

    dicFramework("AlreadyFailed") = FALSE

    Function CLICK (sButtonName)

    If blnFound = TRUE then

    On Error Resume Next


    reportErr ("Click")

    On Error Goto 0


    dicFramework("AlreadyFailed") = TRUE

    End If

    End Function

    Function reportErr (sFunc)

    If err.Number <> 0 Then

    Report err.Description & " in " & sFunc

    End If

    End Function

    thinking outside the box getvalidobject sclass stext
    Thinking Outside The Box getValidObject (sClass, sText)
    • getValidObject ("Link", "Login") 'Sets gloObject
    • Three Attempts
      • Descriptive Programming (Exact Match)
        • oBrowser.oPage.Link("innerhtml:=(.*)?Login(.*)?")
        • False? Add "Index:=0" - True if there multiple objects
      • Search Object Collection (First Valid Match)
        • oDesc("class") = "Link" : oDesc("Name") = ".*Login.*"
        • Set oLinksCollecton = oBrowser.oPage.ChildObjects(oDesc)
      • Switch Class
        • oBrowser.oPage.WebButton("innertext:=Login")
        • oBrowser.oPage.WebElement("innertext:=Login")
    thinking outside the box getvalidobject sclass stext 1
    Thinking Outside The Box getValidObject (sClass, sText)
    • getValidObject ("Link", "Login") 'Sets gloObject
    • Validation – Is this object Visible?
        • Width > 0
        • Height > 0
        • Abs_x > 0
        • Source_index: Appears over an underling object
    • This is not the List object you are looking for
      • List contains the string we want to select
    agile environments
    Agile environments
    • Version Control
      • Not vulnerable to local system crash
      • Code Change Comparison
      • Merge Changes
      • Revert Changes
    agile environments 1
    Agile environments
    • Continuous Integration with Jenkins
      • Run scripts when changes are made or at Night
      • Report overall Pass / Fail
      • Replaces Driver
      • Customizable

    We are real programmers:

    We should use similar tools and processes as the App Developers.

    secrets of successful automation projects
    Secrets of Successful Automation Projects
    • Successful Projects Require
      • Modular Design, Demonstration, Low Maintenance
      • Plan for Common Pitfalls
      • Leverages Agile Processes
    • Automated tests
      • Keywords & KW-Based Components
      • Supporting functions
      • Dynamic Reusable data
      • Object identification
      • Recovery & Cleanup
    • Potential loss can be quantified as ROI
    • Training Increases ROI
    • Always think outside the box!
    • Training Classes
    • HP Certification classes
    • HP Discover
    • LinkedIn Users Group Meetings

    On the Web

    thank you
    Thank you!

    Learn more at


    Additional Thanks to

    Lee Barnes

    Utopia Solutions