Forums » Campaigns » Call for comments: Turker-authored guidelines for research on AMT

I would also support a revision of the guidelines to version 2.0, to address these changes and also the issue of fair pay. There have been many instances of researchers saying they will only pay $6/hour because that is what the Dynamo guidelines say. They’re using this as a hard fact and are not open to discussion of anything close to minimum wage because “Dynamo says $6/hour in it, so that’s all we’ll pay”. I also know some workers who will not send the Dynamo guidelines link to researchers because they don’t want them to see the $6/hour figure. I’d really like to remove this $6/hour part and replace it with a variation of the current points about things to consider when deciding how to set pay.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

I’ve also heard Turkers talk about how the number itself could be problematic. I personally think it was a good first step, before this most researchers didn’t even consider fair payment. But now might be the right time to take the next step in thinking about what a fair payment really is and how in many cases it should be more than $6/hour.

Sign in/Register to reply to this post.

I’ve also received an email from a researcher at another institution asking about the ethics of doing deception studies on MTurk. They do have IRB approval but they’re not sure if it would be appropriate in the MTurk environment. For example, is it OK to give people phone numbers and ask them to call in?

I think we should create an FAQ and gradually fill it with what we think is ethical. This would be much better than having no benchmark at all.

Sign in/Register to reply to this post.

A couple people have expressed interest in moving forward with V2.0 of the guidelines. I think one of the main topics that people are interested in changing is the “fair payment” section. We could start an “all our ideas” poll to get input on this and circulate it among Turkers. The way it works is that anyone can submit a new idea and it lets others vote on those ideas. What do you think? (allourideas.org)

Sign in/Register to reply to this post.

I think it’s important that we move forward and see what changes need to be made now that the Guidelines have been in use for a while. A poll is a good idea, but one thing I worry about is that people can’t really sign up for Dynamo anymore, so how can we get them involved in the discussion? We need a new way to validate that people are workers so they can join in.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

I think it depends on how certain we want to be. For example, we could ask them to screenshot their dashboard and email it somewhere, and we do that bit manually? It could be doctored, but many workers have multiple accounts anyway, so it’s not like it’s at a much huger risk.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

I think it is a great and essential idea to move forward with v2 guidelines since things have evolved since the initial guidelines were in place. I also agree the pay section is a good place to start.

I would also love to see these guidelines directed at all requesters, and not specific to academic requesters. The current pay by requesters is decreasing over time when it should be increasing - I believe that having a page directed at all requesters would be a good resource for every requester to gather updated information, and stress that minimal pay constitutes coercion.

I do send the current links to requesters I see paying unfairly; however, when it boldly states “for academic requesters” I think it’s easier for the standard requester to justify to themselves that the pay can be lower - even though it shouldn’t be.

I added this suggestion to the guidelines 2.0 staging URL.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

Great points!

I’m not sure how we want to proceed. Some parts of the guidelines are specific to academics. We could make the whole document general and then have a “research” subsection where we can move all of the research related parts. On the other hand I’m not sure where we would put the academics’ signatures and how we would convince new academics (and possibly IRBs) to sign. Maybe the ethical research guidelines could have a subsection or peer wiki for more general requesters… What do you think?

I think the screenshot idea is great. For now I changed the instructions on the sign up page, let me know if you want me to change anything: http://www.wearedynamo.org/users/sign_up

Sign in/Register to reply to this post.

faithful_fly

says thanks for this post

Also, to the point of academic vs. general, it might make sense to keep the entire document as a Guideline for Academic Requesters and then have a single page of text that is sort of distillation of the important points that are applicable to general requesters. That might make it more approachable for general requesters to read yet still preserve the academic signatures and document outline as it currently stands.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

Hmm, are you thinking that there would be like a single page that’s relevant to all requesters regardless of who they are - and then two pages, one for academics and one for industry?

I really like that idea, and it’s very doable to break down the guidelines sections and flag things that we think are just good practice whether the requester is an academic or not.

Sign in/Register to reply to this post.

light_dragonfly and faithful_fly

say thanks for this post

That could work!

In my mind, I was thinking a Guideline for Academic Requesters document like we have right now (with the added/modified stuff we want for the new version). And then secondary, an approachable one page document that points out the important things for general requesters to understand with links and references to the full Guidelines for Academic Requesters in case they want to go read that, too.

Sign in/Register to reply to this post.

tired_cricket, faithful_fly, and light_dragonfly

say thanks for this post

careful_owl’s suggestion sounds good to me!

I also have a suggestion about consent forms. Right now it’s not really clear how to use them on MTurk, many people just leave them out and that makes it harder for researchers who want to use them. I had a short discussion with Stanford’s IRB manager about the consent form necessary for online research and it seems that we could come up with a template for MTurk (and other online) studies. If you are onboard with this idea I could schedule a meeting with him and see if we can create a template that we could host on Dynamo.

It could be something like: fill out this form about your study and when you submit it Dynamo will give you a Dynamo approved consent box to copy past to the top of your HIT. We could put in information that both Turkers care about and IRBs cares about. Such as: how long do you expect this task to take, how much are you paying, (we could use that to calculate an hourly rate and put that in automatically), what is the email address and phone number of your university’s IRB (we could also pull that in automatically based on their institution), etc.

What do you think?

Sign in/Register to reply to this post.