Instart Logic Safari Test Automation: Navigating Through the

1. SAFARI TEST AUTOMATION: NAVIGATING THROUGH THE JUNGLE BY KARAN KUMAR AND JAMES CHUONG

2. At Instart Logic, our focus is on delivering high performance, engaging web experiences to users across devices with our Software-Defined Application Delivery(SDAD) approach, with a particular emphasis on optimizing for mobile devices. To this end, it is important for us to be able to test and quantify the performance improvements in application delivery that our service brings to the various web platforms.

3. With iOS being a dominant platform for e-commerceit is critical for us to optimize for this platform. In this post we’d like to focus on the particular challenges of Apple’s Safari browser on desktop and iOS, and our approach to automate testing of Safari and Mobile Safari, thus making it possible to find both functional and performance bottlenecks with websites on this key platform. SAFARI CHALLENGES Unlike other browsers, Safari does not offer much to automate the browser itself. To this effect it is important for us to identify issues listed in my colleague Rajaram Gaunker’s recent blog post “Holiday Wish List for Browser Makers – Requirements for an Open Web” in an automated and scalable way. In terms of performance, Apple recently added navigation timings as part of Safari 8 for OS X and iOS8, but unfortunately these were removed in iOS 8.1.1. This makes it difficult to reliably measure performance on Safari. In addition, these issues generally tend to be non-trivial and can affect a website’s Quality of Experience, both in terms of functionality and performance.

4. To this effect, we have developed a way to automate testing of Safari and Mobile Safari, enabling us to find both functional and performance bottlenecks with websites on the Safari platform. Now more on our approach and testing methodology. OUR APPROACH: ARCHITECTURE Our automated testing setup is currently a simple client-server model. The client starts a test and sends it to the server to run. InSelenium Gridterms, the client would be the user, the server is the hub, and currently the hub is its own node.

5. We selected Selenium as the core driver for our automation testing, since it is the leading standard on several platforms, with many language bindings. Selenium provides the Safari driver with a JavaScript extension that allows it to control the Safari browser. It also provides the JSON Wire Protocol, which defines how implementations of the web drivers should communicate with browsers. Thus, all web drivers will have the same API and can easily drive any browser with the same code. One such implementation is Appium, the open source project that uses Selenium bindings to drive both Android and iOS applications, on real and simulated devices. In fact, as Selenium deprecated its own iPhone driver, it recommended that users use Appium or iOS-driver as a replacement. On the Appium side, there is an additional application being used — ios-webkit-debug-proxyby Google. We use Appium, which is a node.js server to get commands from Selenium. The node would need to be able to start and manage the Appium server to ensure that it is running and ready to drive the mobile device. Apple has JavaScript running through Instruments, a tool within XCode. Appium has Ruby and Java bindings to translate the Selenium API into Webkit’s Remote Debugging Protocol, and thus allows Selenium to automate iOS. Furthermore, the ios-webkit-debug-proxy is used to translate Webkit’s Remote Debugging Protocol into Apple’s iOS Webkit Debugging Protocol. In our case, we are starting a test remotely with our test configuration. The server receives this test configuration, applies the settings and runs the test. Using Selenium, it starts up the browser and routes web traffic through it. For mobile, the service starts up Appium and ios-webkit-debug-proxy, hooks onto Appium and starts Safari. Then, using Selenium to drive Safari, the server navigates to the given URLs.

6. MEASURING PERFORMANCE Selenium itself can only perform very basic functional tests by driving the browser, without any way to measure performance. So in addition to Selenium, we use BrowserMob Proxy, which creates a local proxy that can route browser traffic. The proxy allows us to shape the network so that we can test under different network conditions. Furthermore, the proxy allows us to capture the network traffic into HTTP Archive Report (HAR) files so that we can see the time it takes to load elements on the page. It can be controlled via its RESTful APIs. Now that we are driving the mobile device, we just need to route the traffic from the device through the machine. Then, we can measure performance by routing that traffic through Browsermob. The traffic is received by the service, and recorded by Browsermob Proxy. The recorded data is written out as an HTTP Archive Report (HAR), and can be sent to a reporting software for details and aggregation.

7. Here’s a visual depiction of the automated testing: (https://youtu.be/NoY_PGxroxg)

8. CONCLUSION We hope you’ve found our take-aways on how to launch both functional and performance tests on Safari Desktop and Safari on iOS helpful. This has given us at Instart Logic a way to conduct a large battery of tests and enable us to deliver the best possible Quality of Experience on Safari with our SDAD platform. If you have thoughts on this and other possible approaches, we encourage you to share your comments with the community here.

9. Visitour Blog for more information