Forums » Campaigns » Call for comments: Turker-authored guidelines for research on AMT

Maybe someone could email Pao (paopow@seas.harvard.edu) thank her, and invite her to sign the guidelines. I’m happy to do it from the Dynamo email if you want me to.

Also, a researcher contacted me and said they didn’t understand what the adjective_animal usernames meant at the bottom of the guidelines. Once they found out they were very impressed and suggested that we add a short explanation in the Signatories section. I assume this counts as a minor edit. I could add something like:

The following users have signed the guidelines with their Dynamo accounts, Dynamo auto generates usernames in the form of adjective_animal to protect Turker’s identity. To create an account Turkers have to accept a HIT on AMT which is only open once to each person and requires more than a hundred approved HITs. So each of these signatures represents an individual, active Turker.

Let me know if you don’t agree with this, I will make the edit in the next few days.

Sign in/Register to reply to this post.

Pao has signed the guidelines. She also pointed me to Ken Arnold who is a researcher at Harvard. He has signed the guidelines and also built a template for academic requesters on MTurk: template

The html code for this template is here. Maybe we can point to this template for other requesters? I added this suggestion to the brainstorming and staging page for future consideration: http://wiki.wearedynamo.org/index.php/Academic_guidelines_brainstorm_and_staging

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly

says thanks for this post

“The following users have signed the guidelines with their Dynamo accounts, Dynamo auto generates usernames in the form of adjective_animal to protect Turker’s identity. To create an account Turkers have to accept a HIT on AMT which is only open once to each person and requires more than a hundred approved HITs. So each of these signatures represents an individual, active Turker.

This is already covered more concisely in the first part of the Signatories section, under the 5.0 heading…

“Workers, to lend the strength of your support to these guidelines, please click on the “Sign” button in the Dynamo signing campaign, to sign with a pseudonym. (You will need to submit a HIT on MTurk to receive a code to register for a Dynamo account, ensuring there is one Dynamo account per worker.) “

I don’t think we need to repeat that information just a few lines down, but if you think it’s necessary, you could add the following brief clarification under the 5.1 heading, before the list begins:

“The following pseudonyms each represent one individual Turker who has completed at least 100 approved HITs.”

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

“While active Turkers have struggled to deal with these issues, they’ve worked to find common ground: Earlier this year, an online community for active Turkers, Dynamo, created a set of worker guidelines specifically for the academic community, which include standards for ethical pay. So far, 28 researchers from schools such as Harvard, Carnegie Mellon, and Stanford University have agreed to follow the standards.”

http://associationsnow.com/2014/10/mechanical-turk-putting-humans-behind-machines/

Sign in/Register to reply to this post.

Also, tense_ringworm posted this in another thread but not many people are subscribed to that thread so I’m copying it here:

Hello all!

It has been a month since we launched the guidelines and they have been viewed over 12,000 times! Thanks to all who have signed.

We had a suggestion from a commenter on the crowdresearch.org blog post to get the word out about the guidelines in more venues:

Maybe it would be possible to get this circulated via some official channels. Off the top of my head: CHI Tech News (email newsletter), CACM, XRDS, ACM daily newsletter, Stanford press release. You could also ask researchers you communicate with to broadcast it to their department. There should be some good outlets for communicating in the AI/ML research community (I would imagine they’re heavy Turk users) but I don’t know what publication channels would be appropriate there.

To translate some of that: - ACM = Association for Computing Machinery - CACM = Communications of the ACM (the print and online magazine of the whole ACM) - XRDS = Crossroads, the student magazine of the ACM

What do you all think? One option is we could draft a press release of sorts and send it out to the editors of these various publications.

Once word about the guidelines is out, we could even reach out to organizers of the big conferences like SIGCHI (Computer Human Interaction), HCOMP (Human Computation), AAAI (Association for Advancement of Artificial Intelligence) – places where researchers who use MTurk often publish – and ask them to disseminate the guidelines in their communities’ announcement lists.

Are any Turk workers willing to lead up writing such a press release? I am a researcher and I think it is important for a document like this to be from workers.

Sign in/Register to reply to this post.

Hi everyone!

In the last 24 hours we have had 27 new signatures on the guidelines. The total number of Turkers who have signed on is now 109! :-) Thanks for all your effort!

electorlyte from MTG has started a post to promote researchers who have signed the guidelines: http://www.mturkgrind.com/threads/26926-Trusted-Academic-Requesters-and-Researchers

They also have a post promoting the guidelines and encouraging others to join in by signing it, and sending emails to researchers inviting them to sign too: http://www.mturkgrind.com/threads/26933-We-Are-Dynamo-Guidelines-for-Academic-Requesters-on-Mechanical-Turk?p=403042

careful_owl has posted a new idea to brainstorm ways that we can further promote the guidelines here: http://www.wearedynamo.org/forum/ideas/topics/how-can-we-further-promote-the-dynamo-guidelines-for-academic-requesters-more-academic-requesters-need-to-be-informed-about-it

Sign in/Register to reply to this post.

careful_owl

says thanks for this post

We’ve also seen a research paper reporting the use of MTurk and pointing out that their HITs are in compliance with the Dynamo guidelines (I’ll provide a link as soon as possible). This got us thinking, maybe the guidelines could have a section that gives recommendations about how to report the use of MTurk in academic papers. This is already standardized for how you report in-lab participants. I’ll add this idea to the meta-section of the guidelines tomorrow.

Sign in/Register to reply to this post.

careful_owl

says thanks for this post

A worker made a post on Reddit last night regarding a change in the MTurk policies that neither workers or requesters were notified of. This change now means some parts of the Dynamo Guidelines for Academic Requesters are outdated and need changing.

The AMT Policies page gives a list of examples of types of HITs that violate the Terms of Service. One of these examples was “HITs that require Workers to download software”. At some point after the last Archive.org capture on March 17, 2015, Amazon changed this sentence to say “HITs that require Workers to download software that contains any malware, spyware, viruses, or other harmful code.” https://www.mturk.com/mturk/help?helpPage=policies

Original Reddit post: http://www.reddit.com/r/mturk/comments/3598st/can_hits_ask_you_to_download_software/

Discussion on MTurkGrind: http://www.mturkgrind.com/threads/major-change-to-amazon-tos-policies-hits-now-can-require-software-downloads.27827/

Section 3.5 of the Guidelines for Academic Requesters says “Note that requiring users to download software is against AMT’s Terms of Service. Some workers are willing to download software, but others will refuse as it can be a security risk to their systems.” http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters#Abide_by_AMT_Terms_of_Service

The Guidelines also say: “Don’t require workers to download software programs or apps to complete your HITs (this includes Java programs and plugins such as Inquisit). This can be a major security risk for workers, particularly if the program comes from an unofficial source set up just for the HIT. It became known in 2014 that an academic researcher had performed a study on MTurk intended to see how low of pay levels would still convince workers to download and install a program that pretends to be malware, so many workers who are aware of this study are now even more hesitant to go along with download-requiring HITs even from seemingly legitimate requesters.” http://wiki.wearedynamo.org/index.php?title=Basics_of_how_to_be_a_good_requester#Don.27t_violate_workers.27_trust_and_the_MTurk_Terms_of_Service

These sections will need to be updated to reflect the new changes on the AMT Policies page.

Sign in/Register to reply to this post.

excited_iguana, cheerful_panda, and light_dragonfly

say thanks for this post

Hi everyone!

I just received an email from the manager of non-medical IRBs at Stanford. He has a question about the guidelines that I am pasting here for your input. This is very exciting! I hope we get more attention from IRBs!

I’ve recently come across Dynamo and the guidelines for ethical academic research using mTurk, and they are great. I am referring Stanford researchers to these guidelines all the time now that I’ve found them, because a lot of research here uses mTurk. I’m coming across a problem I wonder if you might be able to help me with. A few studies are recruiting people via mTurk and asking for contact information for follow-up studies (presumably also to be conducted via mTurk). Since collecting identifiable information violates the mTurk Terms of Service, is it even possible to recruit people for follow-up studies? Is there a better way to follow up with mTurk users to recruit them for follow-up surveys than simply asking for email addresses? I know there is an internal mTurk messaging system, but I don’t know much about how it works.

My hunch here is that it would be OK to collect emails if that part of the task is optional. For other tasks they can use the MTurk messaging service. Am I right?

Thanks, Niloufar

Sign in/Register to reply to this post.

According to the Terms of Service, requesters can’t ask workers for email addresses at all. It says in the policies that a HIT that violates the TOS is: “HITs requiring disclosure of the Worker’s identity or e-mail address, either directly or indirectly”. (https://www.mturk.com/mturk/help?helpPage=policies)

In the past, I’ve recommended that requesters use the internal system to contact workers (http://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_NotifyWorkersOperation.html).

But another way which is probably the easiest is to send all workers that you want to follow up with a bonus in the amount of $0.01. You can attach a message to the bonus so the worker receives an email with the $0.01 along with whatever text the requester wants to include. It can be a link to a follow up study or information about the follow up study.

Hope that helps! :)

Sign in/Register to reply to this post.

tired_cricket and light_dragonfly

say thanks for this post

Thanks careful_owl! I will communicate this. I have a follow up question, so if a HIT doesn’t “require” email addresses it would be OK? For example: If you would like to receive email updates on our HITs give us your email address (optional)

Sign in/Register to reply to this post.

The TOS says that HITs can’t “require” disclosure of information. My personal opinion is that it would be acceptable to ask as long as it’s made very clear that it’s completely optional.

Sign in/Register to reply to this post.

The TOS says that HITs can’t “require” disclosure of information. My personal opinion is that it would be acceptable to ask as long as it’s made very clear that it’s completely optional.

Even if it says optional, if someone reports it, it can/will be pulled.

Sign in/Register to reply to this post.

My personal recommendation would be to either do the bonus route recommended above, or have a checkbox in the HIT that is checked by default and says “Contact me about future HITs for this research project.” If they leave it checked, I’d think it’s totally kosher to send a message through the AMT messaging system to invite them to future tasks.

Sign in/Register to reply to this post.

Just a quick note: the ethical research guidelines still reference out of date TOS policies regarding downloading software: http://wiki.wearedynamo.org/index.php?title=Basics_of_how_to_be_a_good_requester#Don.27t_violate_workers.27_trust_and_the_MTurk_Terms_of_Service

We might want to update the relevant portions of the document.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post