Is crowdsourced usability testing worth a try?

The feedback and the insight you can gain from usability testing is invaluable. To actually see people kicking your tires, using your website or software in the real world can bring to light a host of issues that would never crop up in the QA stage.

The good news is that usability testing no longer needs to break the bank. Usability testing services exist that follow the crowdsourcing model. These can provide quick feedback of how your software works in the real world.

Old school labs

Usability testing has always been a useful but cumbersome exercise reserved for those with deep pockets. In the past it might have involved engaging the services of a usability lab.

A group of testers would be gathered, given instructions, and then filmed with a camera or observed through a one-way mirror to see how they interacted with your product. This was usually followed by extensive interviews, analysis and recommendations that pushed the product back to the beginning of its design cycle. 

The whole process was time-consuming, contrived in an artificial setting, and very expensive.

Today, usability testing services are offered online and for cheap. These online companies sign up potential testers and match your testing request with testers who meet your criteria. For each testing engagement, you create a set of test instructions which are then forwarded to the selected testers. 

What you get back is a 20 minute video of the testers actually performing your instructions with audio of their running commentary as they perform the assigned tasks. The videos you get back are segmented, so you can analyse each step of the instructions you provided.

Intrigued by this new usability testing model, their cost, and our urgent need to validate a new account registration process, we tried out two services, UserTesting and TryMyUI.

We were quite impressed and pleased with the results. By providing a deliberately vague set of instructions to visit our website and download our qTrace screenshot and bug capture tool we were able to identify a number of issues and make immediate changes to make the process easier for visitors.

One of the advantages is that the testers are sitting comfy working on their own computers. They come from all walks of life and they interact with your product in their home or office setting, so you know they are not under too much pressure, unlike a monitored lab setting.  

You get a stream of consciousness, close-up view on what your end users are doing and how they approach your product. You can see which instructions are not clear, what steps in a process create confusion and frustration, and which steps point to things that require improvement.

Best of all, usability testing exposed unexpected behaviors which may not have been uncovered or predicted in a more controlled environment.

Besides the low cost, we were also impressed with the speed of these services. Within a few hours of placing the request for testing, we started receiving test results. After 24 hours, we received all of the feedback and could perform our analysis. That allowed us to make some tweaks, retest and improve our usability in a matter of days.

What are the drawbacks to crowdsourced usability?

For starters, you are working with a generic user base and there is no screening for demographics.

If your product is aimed at a very specific segment of the market this might not be so useful for you. The majority of testers are presumably work-at-home types or students looking to earn extra cash.

Because suggested run times are limited to around 15 minutes with a hard limit of 20 minutes, you can’t make the tests too complicated or in-depth. This can be a good thing, though, because you can garner extremely focused results on specific areas, and you aren’t asking testers to concentrate for too long.

Overall, we found crowdsourcing for usability testing to be an exciting new concept that is well worth exploring. It produced actionable results that helped improve our end user experience, it was fast, and the fees are low. With the services we used, we engaged 10 testers for $200. We found this a great way to bolster traditional QA and get some valuable insights.

Related Stories

Leave a comment


This will only be used to quickly provide signup information and will not allow us to post to your account or appear on your timeline.

7 Nov 2012, 6:54 p.m.

Hi Vu, there are a number of platforms out there that conduct non-generic participant recruitment and screening, for example,, including highly specific B2B and B2C, as well as customized client specific participant feedback community set up and management


12 Nov 2012, 7:13 p.m.

Thanks for the tip. I'll certainly look at other sources if we need more than just a generic user profile.


8 Nov 2012, 2:59 a.m.

Crowd Sourced testing could also be useful to get some external perspective. Most of the time in-house QE teams are inherently biased and limited simply because of the fact that the QE have been talking to the developers and believe somethings to be simply "wrong usage", or "out of scope" etc. Sometimes these do not even get filed as bugs. Most importantly, these things happen below the radar of product management, so such issues will not be discovered until public release.
So getting an outside testing is really useful. There is some other similar service called which offers a crowd sourced testing service too.


12 Nov 2012, 7:11 p.m.

We used as well for even earlier feedback of qTrace. But in the earlier round, we were more concerned with features and function definition rather than usability.