The Two Sides of Crowd-Sourced Testing

I was a huge fan of crowd-sourced testing.

I joined a smaller crowdsource testing company as a tester 9 years ago. I fell in love with the idea to have projects tested by a crowd with different backgrounds, experiences, and skills.

I immediately saw the advantages of crowd-sourced testing for companies:

  • valuable feedback to the development about the product within one or two days
  • a great diversity of devices and software versions
  • reveal bandwidth issues based on location and real network providers
  • localization testing by native speakers
  • low testing costs

and I saw the advantages of crowd-source testing for testers:

  • individual growth in different domains
  • learn about new testing tools
  • learn by example: reading others’ bug report is a great inspiration

Raised issues have to be unique and well written, otherwise, they would be rejected, and one has to know the tricks of the trade to succeed in a project. Duplications get closed, hence the quickest and the best gets the bug bounty.

There might be projects that require a survey to be filled in beforehand, or you might get personal invites based on your profile.

Each crowd-sourced testing company has its own platform, which is evolving over time, and the UX is getting better and better at each of the crowd-sourced testing companies where I’m registered to.


Years later I joined a company that decided to have a project crowd-sourced tested and I had the privilege to lead this initiative.

So I did a more exhaustive research on companies available. I wanted to see how the platforms of these companies look like for both testers and project managers. I wanted to know which information is asked from a registered tester, how they invite testers to the projects and if there is a forum in which they can raise questions or have discussions. I found it important. All the experience I gained as a crowd-source tester helped me in the preparation.

In order to prepare a project for crowd-sourced testing, you will have to think over:

  • if you need a project manager(s) to manage the crowd and to review the reported issues
  • if you need someone to create test plans for you
  • how long do you want the project to be open
  • is there a location limitation for the testers
  • how many test cycles do you need
  • what deliverables you need
  • communication channels and availability

and for the testers you will have to give out:

  • precise instructions about the scope
  • specific descriptions of software versions/devices: which software versions/devices are supported, which ones are not. Analytics can help in defining these.
  • testing type. You would be surprised if one starts a performance test, while you were expecting functional testing
  • known issues list, because you don’t want to pay for issues that is already found in-house again.
  • well written test plans, if you want the crowd to follow a specific test script.
  • documentations, like user guides, if the product desires.


What I learned from this initiative:

  • not every project can be crow-sourced tested. The too complex applications are doomed in crowd-source testing.
  • reported issues might lack the significance. Even though you might get a list of accepted issues, it can happen that those have slow values and doesn’t involve any business risk.
  • even though there is a manager on the project who communicates with the testers, coordination takes time. You would need a dedicated person for that. You will have questions to be answered in time, analyzing the results, alter or change strategy based on the feedbacks, prepare the next cycles.
  • there is no guarantee on having the most experienced testers and if you have more cycles, it can happen that different crowd members participate in them.
  • crowd-source testing is easily mixed up with outsourced testing activities. Yes, there is a difference between the two.


Why is my first sentence in past tense?

Nevertheless, I’m grateful for what I have learned by being a crowdsourced tester, nowadays whenever I get an email notification on a paid testing project I either move the mail to trash without reading it, or I go to the site and delete my account for good.

Why I tend to ignore invitation letters for paid testing projects?

Working on a project where the application has clearly never been tested in-house just makes me sad. Yes, I can raise many issues, many of them get accepted and paid, but these projects are rarely well paid. These projects send me the message that this crowd-sourced testing is not about having cross-browser tested the application, but to have the application tested at least before it interacts with real users. The only good light in this is that at least they care about their users. I feel exploited.

Then there are invitations to test complex applications. I realized that I spend way more time on a project like this than I wanted to. As a crowdsourced tester, I dedicate some of my free time on a project to do testing. Considering the bug bounty values, I say that it doesn’t worth it for me.

Would I suggest to be a crowdsourced tester?

Yes. It is a great platform to get into a community where you can expand your knowledge and skills as a tester. Mark your boundaries, though.

One thought on “The Two Sides of Crowd-Sourced Testing”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s