Advice on the economics of app testing: Skip it and costs add up

Recent research shows the fallout from poorly performing apps, but can indie consumer developers bring an enterprise-level rigor to what they do?

Flossing your teeth. Doing your taxes. Testing your app. They may all be chores developers hate, but it's becoming clearer than ever that the last one could really cost them. 

Keynote, a provider of cloud-based mobile testing surveys, recently released the results of The State of Mobile Software Quality Survey, which reinforced the lack of patience consumers have with apps that don't perform. For example, only 16 percent of those surveyed said they would give a malfunctioning app or mobile game more than one attempt. On average, they also expect an app to respond within three short seconds. Yet according to Keynote's research, testing is still not a standard process, even on the enterprise app side. In fact, 7.5 percent of companies across all verticals told Keynote they do no testing for their apps.

If that's the case, can users really expect much better from consumer app developers?

appurify app performance chart

A majority of developers have seen a direct connection between the performance of their apps and the reviews and ratings they get in app stores, according to Appurify.

"Everybody recognizes they need to test, but they're looking for different ways to do so," said Josh Galde, senior marketing manager at Keynote. "An enterprirse will have funds for a large test via the cloud, versus a smaller shop doing a test on an emulator... Overall, the need for testing on real devices was the No. 1 challenge they were having."

Don't wait for a crash
Appurify, a more recent entrant in this space, has been doing some number crunching of its own. For instance, its The State of Mobile Development & Performance in 2014 report showed that nearly 90 percent of developers have seen a direct connection between the performance of their apps and the reviews and ratings they receive in app stores. On the other hand, 60 percent also said they were scanning user reviews at least twice daily, suggesting they are over-reliant on this approach as a means of monitoring their apps' quality.

Waiting for a crash is getting things backward, said Appurify co-founder and CEO Jay Srinivasan. "I would wager almost anyone out there is using reporting tools like Crashlytics," he said. "Finding everything [that could go wrong] before launch, though, is something they're not as focused on. When it comes to network performance, it's a much larger gap."

Like Keynote, Appurify is offering a service which it calls the Appurify Mobile Platform (AMP) that can automate testing on real devices and create real user conditions. Though there are a number of players in this space already, Srinivasan said he sees it as a fertile area in part because, according to Appurify's research, app updates have increased 62 percent over the past year. This makes manual testing too onerous, even for enterprise app developers. 

Jay Srinivasan Appurify


"The Facebooks, Googles, Yahoos of the world--these are large, tech-savvy development teams who get the need for testing and put time and resources against it. That's the gold standard," he said. "There's a second group of enterprise companies and you would find that the quality of their apps are lower than your typical Silicon Valley startup. They tend to struggle with getting the right mobile developer teams from a quality perspective. Consumer app companies are going to get to it before your traditional bank, because they know it's a problem based on their [app store] ratings."

Skills gap needs to be addressed
SOASTA, a Mountain View, Calif.-based provider of cloud-based mobile testing services, doesn't really differentiate between enterprise and consumer mobile developers, according to its CEO Tom Lounibos. That said, he has noticed a bit of a skills gap on the consumer developer side that still needs to be addressed. 

Tom Lounibos Soasta


"Enterprises have built brands over the years, usually through bricks and mortar. They recognize the fragility around customer retention that is related to their brand," he said. "A consumer mobile developer is just trying to build something really cool. They haven't really quite factored in quality as the key point." 

SOASTA's services allow simulation of how apps will perform across devices and various user experience scenarios, Lounibos said. In many cases, the industry has been focused on using data from app testing to offer predictive analytics of what could happen, but Lounibos said developers are increasingly concerned with what is happening in real time, so they can react accordingly. 

Even if the products and services to ease testing are available, developers will still need to factor in how much time and attention they devote to it as part of the process of launching and managing their apps. Appurify's Srinivasan suggests developers aim to be "feature complete" two weeks before launch, then devote the rest of that time to testing. Lounibos goes even further, estimating that 40 to 50 percent of development time be spent on testing--and not just before launch. 

"It's not like you do two weeks of developing and one week of testing," he said. "The development industry has been talking a lot lately about the concept of continuous integration, and we think we should also be talking about continuous testing."