The State of Mobile Quality

A lire sur:  http://www.keynote.com/benchmark/mobile_wireless/interview-jean-pascal-duniau.php

Have you cursed your smartphone lately? Screamed at it? Thrown it at a wall? It’s OK, you’re not alone.

Nearly a quarter of mobile consumers attempting a transaction experience frustration serious enough to make them curse. More than one in ten scream at their phones. And four percent throw their phones (probably while screaming curses).1

In what seems like the blink of an eye, the smartphone has become the most ubiquitous, most used, and at least some of the time, most frustrating technology in the developed world. The majority of consumers now carries smartphones and depends on them in almost every aspect of their lives. Businesses are finding more and more of their revenues are mobile-dependent in one way or another. And yet a stunning 84 percent of consumers — that’s virtually everybody — have experienced problems conducting a mobile transaction.2

That volume of failed experiences is not surprising, given that not even one-third of businesses have an effective mobile application testing programs in place, according to the 2012-13 World Quality Report. Published by Capgemini, Sogeti, and HP, the report is based on a large worldwide survey of financial, IT and quality assurance executives.  The picture it paints of mobile quality testing today is alarming. One can almost see the sheepish expression on respondents’ faces as they acknowledge the embarrassing nature of the problem — specifically the lack of rigorous application testing programs — here more than six years into the smartphone revolution. At the same time, though, one can see solution paths in the data that many companies say they’re eager to embrace.

The World Quality Report summarizes the current quality state, testing practices, and future plans for companies worldwide, based on live interviews with more than 1,500 executives. Visit the Capgemini site to view and download the report for free.

A smartphone wake-up call for business

“We are seeing a lot of companies having that ‘a-ha’ moment, or in many cases more of an ‘uh-oh’ moment,” says Rachel Obstler, senior director of product management for Keynote DeviceAnywhere. “This is the moment when they see the numbers and suddenly realize that their mobile site is as important or even more important than their desktop site. And they have to figure out how to support it with the same care and quality.”

They may be having mobile testing epiphanies now, but companies have been slow to reach that point and are scrambling to catch up. World Quality Report stats indicate that just 31 percent of companies test their mobile applications. Among the reasons that number is so low is that nearly two-thirds report they don’t have the proper tools for testing, and over half do not have access to appropriate devices to test. These are not challenges that companies can easily solve internally.3

Cloud testing is ‘the new norm’

Rather than deal with the expensive and daunting task of setting up in-house mobile testing capabilities, more and more companies are turning to the cloud. Cloud service providers such as Keynote DeviceAnywhere provide immediate access to robust toolkits and, critically, a broad pool of real devices that can be tested live on carrier networks. 

Currently, 28 percent of WQR respondents say they do their testing in the cloud; 39 percent report that they will be doing cloud testing by 2015. This rapid increase in adoption prompted the WQR authors to declare that “testing in the cloud is becoming the new norm.”4 Device selection, speed, lower costs, and on-demand availability are some of the reasons companies are opting for cloud services instead of attempting to handle mobile testing in-house.

“It seems like just about everyone is interested in mobile testing services at this point,” Obstler says. “We see certain industries leading the way. Financial — look at how popular and powerful mobile check deposit has quickly become — and then healthcare, and insurance. Companies in these industries are focusing on both internal and external apps. It becomes very important, very quickly to have a process and platform in place to test these apps, or else it’s just overwhelming.”

Looking beyond the numbers in the World Quality Report

The World Quality Report presents a wealth of data about online quality assurance by companies worldwide — how companies are or aren’t testing, their challenges and plans. Benchmark had a conversation with Jean-Pascal Duniau, mobile testing global lead for Sogeti, one of the report’s publishers, to gain some insight into the current state of mobile application testing. Here is what he had to say.

Benchmark: What has changed since the last World Quality Report was released? Weren’t enterprises way behind in mobile testing then? Have they not caught up at all?

Jean-Pascal Duniau:  The situation has been changing over the past year or so. Increasingly, enterprises are looking for structured ways to test their mobile solutions. Many more are now actively looking for services and solution providers able to tackle the specific challenges of mobile testing.

Benchmark: Looking at the 2012-13 data specifically for mobile, what strikes you the most? What’s your number one take-away?

Jean-Pascal Duniau:  I was surprised that user experience testing was ranked as priority number four in the executive interviews. For me, it’s well-known that user experience is paramount where mobile apps are concerned. Based on my experience, I would have seen it in second position, after performance testing. On the other hand, a common best practice is to validate the user interface, and the flow and story line of the application, before the actual development starts. The respondents then may have considered user experience as outside mobile testing activities, which would explain its low-priority ranking here.

Benchmark: The report indicates that performance is the main focus for enterprises that are doing testing. What type of performance? And why performance over other aspects, especially functional testing?

Jean-Pascal Duniau:  When we say performance testing, we mean performance linked to the network — at what speed the information will be displayed on various devices by the application, and what latency will be perceived by the user.

Performance directly impacts the user experience and studies show that people’s expectations in this area are incredibly high. If one second is added — increasing the time from five seconds to six seconds, for example — when displaying a page of a mobile application, studies show that up to eight percent of users will close this application.

According to Harris Interactive research, the majority of the users expect better performance on their mobile than home broadband.

Benchmark: Given that mobile has been growing so quickly for years now, why is it that the majority of firms in every sector still lack the appropriate tools for testing? And likewise, most lack access to sufficient devices for testing?

Jean-Pascal Duniau: Several reasons. During the early days, mobile solutions were often not managed by IT.  In many cases, IT was not involved or did not want to integrate mobile apps into the company network. In addition, despite well-understood potential, for many enterprises the revenues linked to mobile apps were very difficult to measure and were often very small.

But things have changed.  Now that mobile applications are business-critical, enterprises are increasing their investments to tackle development and mobile testing needs.

That said, enterprises have difficulty addressing some mobile testing challenges. Your example is a very good one: devices. Usually the first two questions from our customers, whatever the size of their organization, are, ‘How many devices do you have?’ and ‘What is your tooling?’  So why is purchasing devices so difficult? In most cases the procurement or purchasing department is set up to handle large contracts, volumes or values of goods, supported by stepped processes. Managing the purchasing of single mobile devices from time to time does not fit with their current processes.

Benchmark: What are your thoughts about testing on mobile emulators? Is it enough? Why do we need real device testing?

Jean-Pascal Duniau: One factor which is often overlooked when testing —especially in the case of mobile apps —is ‘the context of use.’  It’s obvious to say that a mobile app is designed to run on very specific hardware — smartphones, tablets — and a mobile app is mobile, connecting not only by Wi-Fi but also by GSM. So with emulators alone, you can be distant from real ‘context of use.’ Also, features on emulated devices are not the same as real devices unlike, for example, in aerospace or other life-critical industries where emulation accuracy is second to none.

I can appreciate that on some occasions, emulators are useful, but personally, I will not let an app go on a store without testing it on real devices and over a production GSM network. So having real devices through a solution such as the one provided by Keynote DeviceAnywhere is crucial.

Benchmark: What is the danger of sub-par mobile testing? What are the risks?

Jean-Pascal Duniau: The greatest risk is to launch an app with known errors or faults, or an app which, from a quality point of view, does not meet user expectations or business requirements. Any of these can risk damaging the organization’s reputation. As said previously, user expectations nowadays are very high, and in this age of the social network, IM, online and in-store user reviews, even a very few disappointed users can pose a significant risk.

Only a few years ago, some mobile developers considered their customers to be beta-testers. That time is over.

Benchmark: What advice would you give someone who doesn’t have a viable mobile testing strategy in place? What should be their main concern?

Jean-Pascal Duniau:  First, they need to define a test strategy.  It’s always my first advice, and it’s also the first milestone described within the Sogeti & Capgemini TMap testing methodology. Without a test strategy, you do not know where you need to go and how to get there.

About Jean-Pascal Duniau

Jean-Pascal Duniau is mobile testing global lead for Sogeti, a subsidiary of Cap Gemini S.A. and co-publisher of the World Quality Report.

- See more at: http://www.keynote.com/benchmark/mobile_wireless/interview-jean-pascal-duniau.php#sthash.FhzB7y76.dpuf