10 questions to ask when testing a mobile app
The number one thing to pay attention to when testing a mobile app is whatever is detailed in the test case scope and instructions.
If you have a specific set of parameters or use cases, by all means, stick to those. But if you’re doing exploratory testing what should you be looking for? Besides trying to break the app, consider these questions (inspired by the Applause attributes).
How easy is it to use and figure out the app?
Usability testing is a typical facet of mobile app testing, keep it in mind while you’re testing. Is the app easy to navigate? Is it intuitive? Did you have trouble finding a specific feature? Did you happen to stumble upon a feature you didn’t expect (this could be a pleasant surprise or it could mean that that feature isn’t really that usable)?
Do the login and data collection features seem up to par?
Do the password requirements seem less rigid than industry norms? If you exit the app while logged in are you still logged in when you reopen the app (convenient but not very secure)? What does being logged in give you access to (can you see full credit card numbers, for example)?
Unless you’re doing full blown security testing you won’t necessarily come across any major security bugs, but either will end users. You want to look at the app’s security from a layman’s perspective. Does the app’s security seem about right or does it lack compared to other apps or what you’d like?
Did you ever think, “Why do they need that information?”
If an app is asking for too much personal information or info that doesn’t make sense for the app to need, especially if that info is necessary to use the app, it will drive users away. Does the app’s policies adequately explain how your data and information is being stored, for how long, who has access to it and how it is being protected? Is that explanation easy to find, read and understand?
This question goes hand-in-hand with the above question. You need to be comfortable with the data that is being collected and with how it is stored and protected. Users value their privacy.
Is the app fast enough?
This is a simple one. You’re just one tester so don’t worry about load testing right now (though developers should certainly do load testing before launching).
Instead, ask yourself the simple question: If I were using this app in real life, is it loading fast enough or would I have given up by now? If you would have already walked away, the app’s performance needs some upgrades.
Does the app crash, hang or freeze?
Answering this one is a two-step process. First, figure out how well the app works under normal use. If it passes that initial test, go to town on it. Do weird things, click around a lot quickly, change pages without letting them fully load, perform a weird sequence of actions.
While a crash in this situation isn’t necessarily a major problem, it might be helpful for the developers to know. Be sure to keep track of what you’re doing before the app crashes. Simply telling developers that you “clicked a lot” isn’t very helpful. What was the sequence of clicks? How many clicks? How fast?
Was it worth it?
Would you pay for this app? Would you be happy with what you got if you did? Consider the app store description (if it’s available) when you’re considering this question. Compare how the app is advertised to what you actually get, what it actually does and how well it actually works. If the app is free do you get enough features to make it worth the download? Are the in-app purchases reasonable? Go through this set of questions again for every in-app purchase you make.
Does it work with the features on your device?
Another common sense one but one that is hard for developers to test in-house. Each mobile device is a little bit different. So how does the app work with your screen, speakers, buttons, etc? Does the audio work right? Are the touch features responding? Does it connect to the third party plug ins, apps and data that it’s supposed to? This is what in-the-wild testing is all about.
Do you want to come back for more?
App abandonment is a big problem for developers and publishers. If you downloaded this app, would you continuously use it or would it be one of those forgotten apps that sits untouched on your phone. This question serves best as feedback since it doesn’t uncover any hard – or even fixable – issues. It could still be helpful to developers, though. Try to be specific about why you wouldn’t return or what you loved about the app – it’ll help developers focus future versions.
Does the app look cool?
This is another feedback question. People (probably including you) love things that look cool and work in interesting ways. Would you say to your buddies, “Hey, you have to see this?!” Supply this feedback if the answer is yes – positive re-enforcement is good! Conversely, if the app is unusually ugly, find a way to diplomatically mention this as feedback.
Be specific and cite examples, don’t be overly vague, rude or blunt. If the app is average, don’t bother offering this feedback, it will just clutter the stream without offering any value – developers aren’t going to spend time redesigning an app just because its appearance is average.
Is the app accurate?
If you do a search does it return accurate results? Does the GPS pinpoint your location correctly? Is the content of the app true to what’s advertised and does it make sense for the app’s target audience (both in terms of localization and demographic)? If you find yourself thinking “that doesn’t make sense” or “that’s not right” then something is wrong and the developers need to know.
Including specific examples is especially important when reporting this type of issue. Share what you did, why you did it, the results you expected to got and the results you actually got.
During your testing, pay attention to your general mood while using the app. Are you happy? Frustrated? Not feeling anything in particular? Do you find yourself wanting to abandon the app? Meditate on whatever feeling(s) you’re experiencing and see what they’re tied to.
Were you excited about the app’s awesome design but started feeling frustrated when it didn’t return the right results? Pinpointing the source of each feeling will help you figure out which aspects of the app need more work.
Some of these questions are pretty obvious (clearly you’re going to test if an app crashes), but hopefully this post gave you a couple other questions, angles and perspectives to consider while you go about your exploratory testing. For more ideas, look at real reviews in the app stores to get a sense of the actual issues users are complaining about. Search for an iOS or Android app in Applause and check out the attribute scores and signals. This will give you important insights into what you need to test for by showing you what users actually care about.