Forums » Campaigns » Call for comments: Turker-authored guidelines for research on AMT

dark_bird_of_paradise took 1 minute of their time to prepare this

The guidelines are frozen and being signed. I didn’t see any proposed changes that were worth my input. A new thread was established for new content and proposals, so this thread is essentially closed.

Where is the new thread? I can’t find it on the forum.

Sign in/Register to reply to this post.

light_dragonfly made a comment on page three, quoted below edit, Sorry, not a new thread but a page was created

Sure, I created a page here: Academic guidelines brainstorm and staging. Everyone is welcome to use this page to author new content or make proposals for future changes.

My suggestion is that unless something urgent or necessary comes up, content on this page be looked at and debated every once in a while (maybe leave months or even a year in between). We should remember that going over and debating these changes takes up a lot of people’s time and energy, we should be considerate of that.

Sign in/Register to reply to this post.

light_dragonfly and dark_bird_of_paradise

say thanks for this post

Hi all -

Sorry for the radio silence. I wish Dynamo had a way to send a message to everyone involved in the guidelines so far to say “here is where we are at!” When I proposed the guidelines project on Turkopticon-discuss about a month ago, I proposed that once we have some guidelines, we announce them on crowdresarch. I know excited_iguana and tired_cricket have taken a stab at beginning the drafting process it in a wiki page.

I’ll go take a look too. You all are also welcome to.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

The Basics of how to be a good requester discussion page pointed me here, so I guess here is where I will post about the changes I’d like to see.

Avoid duplicates/retakes in fair ways - it needs to have the “DoesNotExist” method added and explain why it’s the most efficient (doesn’t waste as much time as accepting, having your ID checked, returning)

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

Hi everyone, in case you haven’t seen this yet the guidelines have been published here: crowdresearch.org/blog/?p=9039! :-)

So far 12 researchers and 55 Turkers have signed the guidelines. You can help spread the word by sharing this blog post in mturk forums and reddits that you know. If there’s anywhere else that you would like us to post about it, let us know! Thanks.

DBOP, I believe your suggestion has been moved here: Potential ideas to consider in future guidelines revisions. These suggestions will be considered by the community and implemented every once in a while.

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly

says thanks for this post

DBOP, I believe your suggestion has been moved here: Potential ideas to consider in future guidelines revisions. These suggestions will be considered by the community and implemented every once in a while.

I tried to log in to the wiki above to propose some changes, but it will not recognize my username.

There seems to be a lot of material that is out-of-scope, and an extensive amount of conjecture. Respectfully, I think this document needs to be drastically reduced. I think it would appear quite daunting if it it were sent to a requester. It could be made considerably more concise and impactful.

I can not put my support behind this document in its current iteration.

Sign in/Register to reply to this post.

Dr David Brake, Professor of Journalism, at Humber College has signed the guidelines and made the following suggestion: I suggest modifying the guidelines to increase the minimum suggested payment level. If (in the US) you said you were paying f2f research assistants that wouldn’t you be told it was too low? It is below even the US minimum wage!

Sign in/Register to reply to this post.

DV, thanks for your comment. The guidelines have been authored and debated for over three weeks, Turkers have taken care to address every issue and make sure there are no loopholes for problematic behavior. There has also been a lot of work to make the document well-organized in clearly-identified sections and subsections.

You are welcome to add your suggestion to the appropriate wiki page here. To do so you need to activate your Wiki account by clicking on “Activate” in the Wiki account box here: www.wearedynamo.org (if you have trouble, please email us at: info@wearedynamo.org). We are collecting suggestions that will be reviewed by the community and appropriate changes will be made to the next version.

We hope you can join us in supporting these guidelines!

Sign in/Register to reply to this post.

tense_ringworm and dangerous_viper

say thanks for this post

Prof. Don Norman of UCSD has written to us to sign the guidelines, he says: I write in support of the Guidelines for Academic Requesters​. Sensible, clear, and fair to all sides. It would be excellent if Amazon would require (or at least recommend) them.

Thanks!

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly and cheerful_panda

say thanks for this post

Maxine Eskenazi of CMU has written to us to sign the guidelines:

I think this is wonderful. I have a few questions: First, although you may get this information to the workers, it is essential to get it to as many requesters as possible. How can we do this? Second, although I have seen a fixed number as a reasonable per-minute pay, I would suggest a variable so that as the cost of living goes up, the wage does also and requesters a few years from now are not citing a very old number. Finally, I have my students calculate payment according to the equation that I provide. My problem is getting them to correctly estimate the amount of time that it takes to do something. I tell them not to do it themselves, but to observe friends doing a task. Any better suggestions?
Sign in/Register to reply to this post.

Something I feel the guidelines should promote is the use of participatory action research when studying turkers on forums, TO, wherever. Just wanted to note that here :)

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

This seemed to be the most relevant thread to post this in, but let me know if it should be elsewhere.

Turk forum moderator Zingy saw these guidelines refrenced in a mturk study today :D

“Wow, this Isis survey’s debrief page has recommendations from Dynamo. Nice one turk community.” (Isis is a typo, the requester was ISS)

Not all turkers are on forums or aware of this campaign, so I was glad to see it referenced in a HIT.

Sign in/Register to reply to this post.

light_dragonfly, gorgeous_monarch_butterfly, excited_iguana, and cheerful_panda

say thanks for this post

I didn’t happen to catch that HIT, just saw it talked about on the worker forum. I’ll keep my eyes peeled and my print-screen key ready!

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

Requester account “IIS” appears seasoned (two years on AMT) and well-regarded on a light TO http://turkopticon.ucsd.edu/reports?id=A2WDDWA4KZC5Y6 I saw the HIT, but I was wrestling alligators, and don’t recall the individual researcher’s name. It contained an external survey hosted on Harvard’s Qualtrics servers.

(By the time I got to a machine where I could poke around without causing trouble, the HIT was gone).

edit: Here’s an image of a consent form (from what may be a different HIT) by the requester: http://i.imgur.com/aH46Vej.png

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly and light_dragonfly

say thanks for this post