The ABCs of A/B testing on mobile apps
The key to winning at Wordsplosion, like so many other games, is speed. You are given a five-letter word to guess and are even provided one letter as a head start. If you fail to get the answer within five guesses before the clock runs out, then everything on screen blows up. The question is, how much time is enough to make winning possible but still challenging?
The staff at Concrete Software, an Eden Prairie, Minn.-based developer that created Wordplosion, could have created their own subjective time limits to settle the issue. Instead, they tried an experiment, whereby different sets of players were given slightly different versions of Wordsplosion to play with.
"We thought maybe it was too easy," said Mike Lehne, Concrete's CTO. "We tried increasing the difficulty by offering a shorter length on the game."
This is A/B testing, a concept popularized within marketing departments as a way of rationalizing decisions around advertising but which is quickly making its way into the world of mobile apps. Earlier this month Amazon launched the beta version of an SDK designed to assist developers with A/B testing, which is available free of charge.
According to David Hanson, who works on the development team at Concrete, the SDK allows the company to create server calls that serve up the various versions of the game. Amazon (NASDAQ:AMZN) has some statistical data available through the service that can assist with evaluating the results of the A/B tests, Hanson added.
"It's not just a bunch of people talking in a room about things that sound good, but actually getting data about changes that you could make," he said. "You can try [ideas] both ways and see with the data [and] what's better and hopefully back up what we think we know."
In an e-mail to FierceDeveloper, Amazon Director of Games Mike Frazzini said the company was focusing first on making the tests easy to use and accessible. "Amazon will continue to simplify the math and statistics," he said, adding that, as Amazon adds more features, "we expect that developers will become more adept at instrumenting their games and making more informed decisions based on data."
Amazon is not alone in the app A/B testing space. Startup PathMapp allows developers to push multiple designs and features to different sets of consumers.
According to Salil Malkan, PathMapp's lead developer, the service will be complemented by analytics that give details on how users touch, swipe and otherwise interact with various versions of an app.
"One of the motivations behind PathMapp was that we had a lot of analytics for apps we were working on, but the sheer amount of data available was often overwhelming and time consuming to analyze," he said. "We wanted to make a platform which simplifies data down to its most important components."
The PathMapp technology will create what it calls "Heatmapps" that show the common paths users take through an app, allowing developers to fine-tune the user experience.
Swrve New Media, another startup, offers its own A/B testing service, which provides various packages of testing and analytics for $1,500 to $1,700. Its CEO, Hugh Reynolds, said it's only a matter of time before A/B testing is a standard part of the app development process.
"If I pitched a major commercial website without any ability to test specific components and put the right messages in front of the right customers, I would be laughed out of the room," he said. "We'll be in the same place in the app space within 18 months, and if you're not setting up a testing structure now, you won't be in business in two years' time."
That being said, developers may need to figure out a few steps first, including what to test and how many consumers they need to get any kind of meaningful results.
"We'll at least have shown each version of the test to 1,000 people and obviously the closer [the results] are, you would need to show it to more people," said Concrete Software's Hanson. "If it was very even, maybe it'll take a long time to reach statistical significance."
PathMap measures the differences among test cases.
Reynolds said developers should be creative in thinking about what elements of an app could benefit from A/B testing, taking their cue from the early days of e-commerce development on the Web. Mobile games may offer more specific opportunities, he added.
"For example, start position is incredibly important in terms of early retention and long-term monetization, and it is a straightforward place to run A/B tests to find out what works best for you," he said. "Whatever you can define with data--which is pretty much everything--you can test."
Malkan said PathMapp is focusing primarily on services to help developers test app design and that the emphasis should be on creating as painless a decision-making process as possible. "We feel like A/B testing and usability testing in general on mobile platforms is still very much in its infancy," he said.
What may become more important than an A/B testing service is creating the right kind of attitude among a developer's team.
"A decent testing platform will take users most of the way when it comes to interpreting results. What matters is creating a testing culture and building in a data-driven way that enables testing to take place without engineering involvement," Reynolds said. "That then supports a rapid test/iterate cycle that can lead to huge gains in short time periods."