Forums » Ideas » Can Turkers help requesters find spammers? Requesters and Turkers are hurt by spammers alike, what can we do about it?

What if we could for example have requesters upload their results (without worker and assignment IDs of course) and we could help them find cases that look like someone is obviously spamming. It would then be up to the requester to decide if they want to reject that work or maybe add that worker ID to a qualification so they can’t do their HITs anymore. Remember that requesters can reject work anyway, but helping them improve the quality of their results will both get more people interested in using MTurk, and it will raise the wages for those who are actually doing honest work. We could even probably figure out a way for the requester to pay for us helping them.

This is a half baked idea that came out of a recent problem we had on a project I was collaborating on. For the requester it’s extremely hard to handle spammers, but Turkers are more experienced at spotting spammy work. I’m posting this idea here to put it to discussion and vote! Thanks for your thoughts.

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

Luis von Ahn, a professor of computer science at Carnegie Mellon University and a crowdsourcing expert, says Irani is one of the few people trying to stop abuse on the part of the employers. Much more frequently, the abuse that takes place is on the part of the workers who try to game the system, hoping to get paid for little or no effort, he observes. "Imagine a crowdsource task that pays people to look at images and tag each one with a description," he explains. "A worker can just, say, hit the 'F' key a few times and hope to get paid for that useless input."

from: Software Aims to Ensure Fairness in Crowdsourcing Projects

Do turkers agree with this? What can we do about it?

Sign in/Register to reply to this post.

Requesters can simply put in spam questions with known answers and when those are wrong It would flag their work for further inspection. If that user gets too many rejections they would soon not have much of a choice of hits they qualify for.

Someone is already checking people’s work. There is a mysterious requester by the name AR-WA (Amazon Requester-Washington?). No one really knows what they are up to but it is assumed they are reviewing work to hand out Masters. Doesn’t really apply to this situation but maybe follow their lead and do the same. Maybe there could be a widely adapted alternative Masters type qualification.

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

Thanks for your comment funny_giant_panda

Requesters can simply put in spam questions with known answers and when those are wrong It would flag their work for further inspection. If that user gets too many rejections they would soon not have much of a choice of hits they qualify for.

This isn’t really easy to do in most cases, it’s usually done in two ways:

  • Either the test questions are obvious, that is you ask each new Turker to do a few test questions first and prevent people from continuing your HITs if they get them wrong. This is easy for spammers to get past or even to share the answers for.
  • Or you ask random test questions every other real HIT. For one thing, this costs money. For another, you have to have enough test questions so that every turker who works for you get’s asked at least one test question. In this case, what if someone who’s done a lot of HITs just makes an honest mistake?

I’m not really sure how popular these methods are and what problems Turkers face when they are used, I’m sure others in the forum know much more than I do. I’m just sharing the problems I’ve faced as a requester on MTurk.

Someone is already checking people’s work. There is a mysterious requester by the name AR-WA (Amazon Requester-Washington?). No one really knows what they are up to but it is assumed they are reviewing work to hand out Masters. Doesn’t really apply to this situation but maybe follow their lead and do the same. Maybe there could be a widely adapted alternative Masters type qualification.

I may be wrong, but my thought on this is that automated methods for creating qualifications for Turkers have not been the most successful. Whereas a lot of the times when things are left to Turkers to take care of, they do an amazingly good job. Turkers may decide that they want to use Dynamo to create a qualification on MTurk, if this idea gets up-voted we are more than happy to build the software to support it. But what if in addition to (or instead, I really don’t know) qualifying Turkers, you could help identify spammers. If there was a way that spammers could get quickly identified and removed from the worker pool, wouldn’t that make life a whole lot better for everyone? My experience is that this is very hard to do programmatically, but what if Turkers could collaborate to do it?

If Turkers can collaborate to author an amazingly well written guideline for requesters, why can’t they become the leading force for removing spammers from MTurk?

Sign in/Register to reply to this post.

This is work. I can’t imagine attracting qualified people unless they’re paid. My funny_giant sibling is on the right track by making an analogy to AR-WA. Whether AR-WA work is done under contract for the individual requester, or whether it’s for some mysterious Amazon internal purpose, it typically pays as much as the original HIT itself.

The CACM article is behind a paywall. With a line like “Much more frequently, the abuse that takes place is on the part of the workers who try to game the system,” that is perhaps for the best. One could just as accurately – and with just as little meaning – say that workers perform precise and accurate work much more frequently than requesters do. It’s the nature of crowdsourcing to have a many-to-one relationship. (I don’t really think Paul Hyman missed the memo on that; he’s just being sensationalistic).

“clickhappier” collected and collated a frightful amount of information about the mysterious AR-WA: http://www.mturkgrind.com/threads/74-AR-WA-The-Truth-Is-Out-There?p=254714&viewfull=1#post254714

Sign in/Register to reply to this post.

light_dragonfly and clear_mountain_lion

say thanks for this post

I agree that this should be payed for. Right now, I can’t really think of a good way to distribute the payment between the people who do the work though. It would also probably be much easier to find a spammer if you have all their work in a list in front of you, e.g. this person is just clicking on the first option all the time, or these results are obviously coming out of a script, etc.

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

I interpreted funny_giant_panda’s suggestion (for an “alternative masters type qualification”) as being a proposed implementation of light_dragonfly’s idea, not an alternative to it.

It’s unclear to me how “turkers” could be helpful unless a prequalification like funny_giant_panda suggests is applied to the workers who review this work that, for whatever reason, the original requester elected not to prequalify. You can move the demand for qualification from one pile of work to another, but I don’t see how it can be avoided.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

Yes, this is basically what I was getting at. A Masters alternative granted by a trusted group of turkers that can be applied to hits by any requester at no additional cost. Perhaps not just a simple qualification but a quality score % or spam score %. Requesters and turkers alike should be aware of the quality of work needed to attain and maintain this qualification (unlike Masters). Anyone who has the qualification should be audited at random. The main problem would be how to choose these auditors and who would pay them.

Sign in/Register to reply to this post.

cheerful_panda and light_dragonfly

say thanks for this post

I think the first thing that would need to be looked into about this is whether requesters can actually set a requirement for a custom qual that was created by a different requester, and how. I don’t think I’ve seen any examples of this being done yet, beyond the Amazon-created quals (stats quals, Adult Content, and the three Masters variants). Perhaps a qual ID number would just need to be publicized, but it’s also possible that Amazon might prevent cross-account use of custom quals. Some people with requester accounts could experiment with this. If it works, then proceed with the idea of how to manage it. :-)

Sign in/Register to reply to this post.

cheerful_panda and light_dragonfly

say thanks for this post

I think the first thing that would need to be looked into about this is whether requesters can actually set a requirement for a custom qual that was created by a different requester, and how. I don’t think I’ve seen any examples of this being done yet, beyond the Amazon-created quals (stats quals, Adult Content, and the three Masters variants). Perhaps a qual ID number would just need to be publicized, but it’s also possible that Amazon might prevent cross-account use of custom quals. Some people with requester accounts could experiment with this. If it works, then proceed with the idea of how to manage it. :-)

I know only one example of what appears to be cross-account reference of custom qualifications, apparently without knowledge or consent of the requesters (both commercial and academic) who originally granted them.

That isn't possible using the web interface exclusively. If that's really what happened, part of it has to be API stuff.

But it isn't necessary. Different requesters can just refer to specific worker IDs, granting and revoking their own individual qualifications (supported by elements of a shared database). I haven't encountered any problems "rolling my own" common elements between two requesters, using only the web interface and spreadsheets. (I find it less confusing for each requester to have differently-named qualifications, even when they are functional analogs of each other).

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

I think the first thing that would need to be looked into about this is whether requesters can actually set a requirement for a custom qual that was created by a different requester, and how.

100% impossible, I have looked into it myself. I wanted to create a list of good workers to make available to Requesters, but they could not use a qual that wasn’t theirs. I even checked with Amazon directly, no dice.

The only way to share the qualified group would be to share the list of Worker IDs themselves.

Sign in/Register to reply to this post.

light_dragonfly, gorgeous_monarch_butterfly, and cheerful_panda

say thanks for this post

I wasn’t really referring to a qualification when I created this idea, but since the discussion has gone that direction, I have a question: assuming there was a way to share a qualification between requesters, and Turkers could somehow decide who gets in on it. Would you be for or against the use of it by requesters?

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

There would only be two possibilities for this with the current mTurk setup:

1) Sharing a list of trusted Worker IDs - it’s anonymous, so that should be fine, it’s unlikely to be abused (e.g. someone gets it and blocks everyone on it multiple times to have their accounts suspended) - the only issue is that we would all have to be notified that Requester name is planning to add us as we don’t always get an email when a qualification is granted

2) Sharing of a single Requester account for the sole purpose of having work double checked - not a good idea to give someone the login to a Requester account

So, I think the first option would be okay.

Sign in/Register to reply to this post.

cheerful_panda and light_dragonfly

say thanks for this post

Thanks dark_bird_of_paradise (BTW that’s the best username ever!)

I can think of some ways to do 1 without giving out worker IDs, for example we could have a system that requester’s HITs could issue API calls to and check to see if a worker ID is in a list or not. We could even charge requesters for every 100 API calls they make to raise money for maintaining the list of good workers.

IMO what is most difficult with this is, how would you create and maintain the list? How would you ensure that spammers don’t get in but honest Turkers do? If we implement something like a peer review system, how can we ensure that people will be honest in their assessments?

Sign in/Register to reply to this post.

I can think of some ways to do 1 without giving out worker IDs, for example we could have a system that requester’s HITs could issue API calls to and check to see if a worker ID is in a list or not. We could even charge requesters for every 100 API calls they make to raise money for maintaining the list of good workers.

There is nothing more annoying as a worker than to accept a HIT only to have it tell you you can’t do it because you’re on their list of workers who have already done the HIT (or, in this case, you AREN’T on their list of good workers.) I want to know in the search results whether or not I can do the HIT to ensure I waste no time I could be making money.

IMO what is most difficult with this is, how would you create and maintain the list? How would you ensure that spammers don’t get in but honest Turkers do? If we implement something like a peer review system, how can we ensure that people will be honest in their assessments?

AR-WA is hand-picked individuals who showed an aptitude for work and adhere to strict codes of high quality submissions. I’m not sure how any of us could determine that any worker is at these standards as we can neither see the work they submit nor their current statistics. Even screenshots can be doctored. Also, to determine who was worthy of this list, anonymity would be impossible. I also see politics coming into play strongly. I can see huge backlash as well when someone who is recommended is denied, even if it is for cause.

Sign in/Register to reply to this post.

cheerful_panda, gorgeous_monarch_butterfly, and light_dragonfly

say thanks for this post

There are many reasons why requesters do not share qualifications or want to use someone else’s qualifications. If you post a lot of work, you do not want to give out your best workers to someone else who could steal your workforce of quality workers. Also, some requesters have paid a high price for finding the right workers. They will not give them up and they always pay a fair wage.

If you are spending money, would you want to trust some unknown entity to give you a list of worker ID’s? I wouldn’t. I know of two very large requesters that were handed a list of time tested workers and they refused to use them. They wanted to do it themselves and ended up getting burned pretty badly. (They also did not listen to advice on how to do their own qualifications, insert “gold” questions, design their hits so they could not be scripted and communicate changes in instructions.) What about new turkers, when would they be allowed to be added to the list? No matter how good the qualification is, time is the only way to eliminate bad workers. Even then, when you are adding workers, you are essentially watering down your list with unproven workers.

To remove spammers, it would have to be tiered. Tier 1 with new workers, tier 2 with workers who preformed well on x number of hits, and maybe tier 3 with completely trustworthy workers. This system has caused major problems with requesters in the past because the review process is not transparent and there are no set guidelines on how to gain or lose a tier.

dark_bird_of_paradise makes some very valid points about politics being involved and removal of anonymity being a necessity.

Just maintaining this list would require hours of unpaid work.

Sign in/Register to reply to this post.

cheerful_panda, gorgeous_monarch_butterfly, and light_dragonfly

say thanks for this post

I’m not really sure the line between “good: and “bad” workers is as clear as it’s sometimes being made to sound.

I don’t think of myself as a bad worker, but two of the rejections on my record were not ‘unfair’ (although I won’t work for the requester again, either). I also fairly ‘earned’ another ten rejections which were reversed as a courtesy. Good workers can have bad days, absent moments, or a stuttering computer mouse.

At first glance, the line between “spammer” and “honest worker” seems more clear, but I don’t think this can be the universal case for all kinds of AMT work.

I agree with everything dark_bird_of_paradise and vast_anaconda wrote in the two posts above, too.

edit: meant to reply directly to light_dragonfly, but close enough for government work

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly and light_dragonfly

say thanks for this post