Future Directions For The Windows CE Test Kit

1. Future Directions For The Windows CE Test Kit James ZwygartLead Program ManagerMobile and EmbeddedMicrosoft Corporation

2. Agenda • Testing Windows Devices • Overview of Microsoft Testing Process • Tools and Resources

3. Who Should Test What? Added Applications InROM ISVs, OEM/ODM Customization Windows Mobile Microsoft Windows CE Drivers and Radio OEM/ODM

4. Testing WindowsEmbedded Devices • Fact • Quality is morethan just Software and Hardware requirements. There must be integration testing • Testing is an on-going process that starts fromday one, and evolves throughout the product cycle • Testing is a mix of different methods, tools, people, and approaches • Your InROM ISVs should be using all the toolsyou have available to you

5. Testing WindowsEmbedded Devices • Fiction • Passing the test in the CE Test Kit and requirements for the Mobile Logo Test Kit = High Quality Device. There must be integration testing • The tools and resources Microsoft provides are all you need

6. Agenda • Testing Windows Mobile Devices • Overview of Microsoft Testing Process • Tools and Resources

7. Definitions • Bug – a defect in the product that is tracked in a database • Triage – process in which decisions are made whether to fix, postpone, or dismiss a bug • MTTF – mean time to failure. Our primary metric for Stress and Stability testing • Exit Criteria – set of criteria that must be met to exit a phase of the product • Dogfood – using your own product while it is in development

8. Overview Of MicrosoftTesting Process

9. Agenda • Testing Windows Mobile Devices • Overview of Microsoft Testing Process • Tools and Resources

10. Tools And Resources • Platform Builder for Windows Mobile 5.0 (Magneto) • Remote tools • Debugger • OEM Documentation • Windows Mobile CETK (WMCETK) • Application Verifier, TUX, DATK, Radio, Hopper, and more • Windows Media Testing • PlaysForSure Test Kit or the Basic Media Test Kit (bMTK) • Community/Support • TAMs • Windows Mobile Device Maker Portal • General CETK Newsgroups (great to reference)

11. WCETK Communications • How the WCETK communicates with Host and client

12. Connecting The CETK • Connect the CETK to a target device by using Platform Manager • Connect the CETK to a target device by using Sockets • Command-line from PB or the device • Clientside /n=“ServerName” /p=5555 /a /s=“james • cetk.txt file • This can be on the device in the same directory as clientside.exe or in the release directory • Registry of the device • Create a HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CETT registry key on the target device

13. MTTF And Stress (Test Lab Setup) • Stress manages test modules that perform the actual testing • Stress runs a specified number of test modules at all times • Stress tracks memory and object store usage over time, records whether each test module passes or fails, and detects when a test hangs • The harness provides the ability to set the level of logging verbosity for the test modules • The harness can be configured from the Windows Mobile Stress dialog box

14. Lab Diagram

15. Understanding FailuresThere are 2 types of failures • Expected failures • Know the driver functionality. So you will understand if a test is failing because the driver does not support some functionality • At this point edit the test command line to eliminate this specific case • Unexpected failures • Use the source code to the tests to help debug why a test is failing

16. Using Test Sources • You can install source code for the tests that ship with the CETK • Microsoft provides source code to help you debug your device drivers and create custom tests for the CETK • However, you cannot build some of the source code, and source code for some tests is not provided

17. Advanced Features • Auto-run using the /a option • Using a storage device with low memory devices • Running tests that need to reboot or run “Off-line”

18. CD/DVD-ROM File System Driver Test File System Driver Test Winsock 2.0 Test (v4/v6) Keyboard Mapping Test Keyboard Test Mouse Test Touch Panel Test Storage Device Block Driver Read / Write Test Storage Device Block Driver Benchmark Test Smart Card Resource Manager Test Serial Communications Test Serial Port Driver Test Parallel Port Test IR Port Test (Winsock 1.1) IR Port Test (Winsock 2.0) Telephony API Server Test Telephony API Client Test Two-Card Network Card Miniport Driver Test One-Card Network Card Miniport Driver Test Bluetooth Test GDI Display Test Wave Audio Sound Card Test DirectDraw Display Test list Storage Device Block Driver API Test CD/DVD-ROM Block Driver Test Partition Driver Test RDP User Scenario Performance Test List Of Included Tests

19. Winsock Performance Test Wireless Network Card Test Notification Test MSMQ Test ICMP API Test Credential Manager Test Local Authentication Plug-in Test RAS Server IOCTL Test RAS Authentication Test RTC Provisioning Test RTC Client Test RTC Presence Test RTC PC to PC Session Test RTC PC to Phone Session Test RTC Instant Messaging Session Test RTC Group and Buddy Test (Part 1) RTC Group and Buddy Test (Part 2) RTC ACL Test VoIP Test OAL Cache Test OAL Timer Test OAL IOCTL Test Kernel Scheduler Test User Interface Test Graphics Device Interface Printer Test PC Card 32-Bit CardBus Host Controller Driver Test PC Card 16-Bit PCMCIA Host Controller Driver Test PC Card Legacy PCMCIA Driver Test USB Host Controller Driver Test Flash Memory Read/Write and Performance Test NDIS Performance Test Audio CD Driver Test Imaging Codec Test Direct3D Mobile Driver Comparison Test Direct3D Mobile Driver Verification Test Direct3D Mobile Interface Test New Tests To CE 5.0

20. New Tests For Later Versions Of Windows CE • New tests will be announced at a later date

21. WMCETK CETK LTK Test Kits • There is some overlap between the three test kits • CETK (CE Test Kit) • LTK (Logo Test Kit) • WMCETK (Windows Mobile CETK) • General guidelines • CETK • Suite of tests to validate Windows CE drivers • Includes certain tests that are irrelevant to Windows Mobile • WMCETK • Exposes CETK tests that are relevant to Windows Mobile (omits others) • Also adds some additional Windows Mobile-specific tests; e.g. radio, battery • LTK • A database harness that exposes tests required to validate the requirements in the ‘Hardware and Software Requirements Document’ • Includes manual, automated, and partial-automated tests • Utilizes a subset of the tests from the WMCETK • Adds some additional tests; e.g. a subset of the PlaysForSure test kit

22. WMCETK CETK LTK When To Use These Tests? Primarily needed for BSP bring up Repeat throughout Development of Mobile Device Checkpoint occasionally during development and before final release

23. The Windows CE Test Kit James Zwygart

24. WMCETK CETK LTK Test Suite Drill-DownOAL Tests • Timers • Confirm that drift is minimal • Check for valid resolution • Caches • Cache Parameters • Correctness • IOCTLs • IOCTL_HAL_GET_DEVICEID • IOCTL_HAL_GET_UUID

25. WMCETK CETK LTK Test Case 200 Test Case 210 Test Suite Drill-DownGraphics Device Interface Test • Verifies drawing of shapes • Lines, Polygons, Ellipses, • Verifies surface manipulation • Raster operations • Compares ‘known-good’ output to driver under test • Examples of test scenarios • Draw bitmap, line, shape, or text • Perform a raster operation (e.g. merge two bitmaps)

26. WMCETK CETK LTK Test Suite Drill-DownRadioMetrics Test • Scenarios • Voice Call • 1000 Incoming and Outgoing Calls • General Functionality • Mixture of Voice, Data, RIL, and SIM tests • Call Hold and Conference • Testing Call Waiting, Call Hold, and Call Conferencing • SMS • 100 outgoing and incoming SMS Messages of varying length • Supplementary Services • RIL Tests • GPRS Performance Test • 20 MB of Data received in 4x256 KB sections x 20 sections • GPRS Stability / GPRS Sync • Data testing mimicking Active sync like scenario for 24 hrs • Transfer Data, Sleep, Transfer Data, repeat. • General Stress • Random testing for 24 hrs. to make sure that no crashes occur • RIL Test Kit (RILTK) in the making • ETA  EOY 2005

27. Key Resources • Windows Mobile Device Maker Portal • http://msftmobile.com • Access to the latest Adaptation Kits, Documentation, Logo Test Program information • Helpdesk for Logo Test Program issues • OEM Documentation • Design Requirements (Software, Hardware, and Branding) • Everything Windows Mobile 5.0 • PlaysForSure • http://www.microsoft.com/getplaysforsure

28. Call To Action • Start testing early • Share what you have available with your InROM ISVs • Extend what we provide and add your own tests

29. © 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.