App Testing In The Wild

If it’s true that 77% of app downloads are never used within 72 hours after installing, you have to wonder about lost dollars in development.
Consider that the average cost for developing and deploying an app is $5o-$300k. Multiply that times the current rate of new apps appearing in the app store alone; roughly 252 per day.
We’re talking $12m – $75m spent every day with $9 – $58m of that investment yielding $0 in ROI. With that probability of loss, why would anyone deploy an app without testing the concept with a target audience?
Mobile product testing does not need to be overly complex or costly. It does not need highly sophisticated software or a massive sample of participants. Mobile product testing does needs the following:

  • Real world scenarios and tasks to evaluate
  • Participants who might experience those scenarios
  • An interactive prototype installed on a device
  • Portable recording set up
  • A skilled moderator

Test real world scenarios

Your app must have a purpose, hence the investment to develop it. That purpose can be anything. It can be as simple as depositing a check with your phone. Determine what the enabling technology is and draft a scenario around that purpose. Your app may require a series of tasks required to complete your scenario. Be sure to include all necessary tasks.

Test with participants from your target audience

Asking your friends, siblings or colleagues for feedback on your app doesn’t count. You must identify a target customer with a real world need and use case for your app. You may use friends and colleagues to find participants but they must be screened to best match your target customer.

Test interactive prototypes

In order to understand if your app will be an enabling technology for your target audience, you must simulate as real an experience as possible. Remember, you won’t be there to answer questions when they download your app and decide if your app will end up in the 77% never-open-again column of app usage.

Test in the wild (on location)

Testing apps in a lab is like test driving a car by turning the ignition on and off. Sure the engine works but how does it feel when I’m rolling down the road? Apps are personal, mobile and highly contextual to an individual’s lifestyle. Meet with your participants onsite to simulate where they might use your app. Utilize a portable recording set up like a laptop with Quicktime. You want to capture every nuance of the participant’s reaction to your app as they interact with it. (see sample video)

Work with a skilled moderator

Even with a great scenario, the perfect participant, and interactive prototype you may miss key insights without a skilled moderator. A skilled moderator knows how to present the scenario and engage the participant. A skilled moderator knows how to develop an engaging 2 way conversation without leading. And most of all, the skilled moderator knows how to get to the “why” a participant does or does not do something. When you get to the “why”, you have something tangible that you can address in the design.

Realize a Return on Your Investment

The bottom line is simple, put aside 15% of your budget to test your app with your target customers. The return on that investment can ensure your app doesn’t end up in the 77% column of customers who never use an app after it is downloaded and that is money well spent.