Forums » Campaigns » Call for comments: Turker-authored guidelines for research on AMT

I’m going to freeze the guidelines tonight at 11:59 PM (PDT). If anyone objects to this please say so, if not it would be nice if you thank this post so we get a sense of who’s onboard. Thanks!

Sign in/Register to reply to this post.

Sounds good, except I’d think saying ‘a private Dynamo thread’ (one per requester/situation), rather than one thread being ‘the’ thread for everything, would be preferable long-term; if you ever have more than one situation going on simultaneously, it could get pretty confusing otherwise.

Sure, I guess this is one of the things we’ll figure out as we go forward :-)

Sign in/Register to reply to this post.

As the document nears freeze point:

  • I want to remove the phrase “politely.” I don’t want these ethics guidelines to be seen as a small group trying to discipline workers. I worry it could be read as condescension. Since it is small, I will do that now.
  • I’m worried that when we go around to get signatures, this idea of cc:ing a Dynamo thread is a bit unclear to me still. I’m not in a position where I can standup and say “here is why I think this is a really good idea and here are the people lining up to volunteer to do it.” I’m happy to leave it in if there is someone who is willing to stand up and advocate for it as we try to get signatures. But if this is something that feels like it is coming from researchers rather than from workers, I would like to scrap it as I’m uncomfortable with the idea of researchers overseeing workers’ activities. Happy to help but uncomfortable suggesting it is my place to oversee or supervise.
Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

Does this (and in conjunction with an edit earlier tonight in a similar direction) make its optional nature clearer?

“Encourage the worker to email the requester, and to CC ethics@wearedynamo.org if the worker would like copies to be posted in a private Dynamo thread for ongoing feedback.”

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

I am happy with this, or one other option: instead of a CC address, we link to a private Dynamo forum thread where people can post their emails as they send them and receive the responses.

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly, light_dragonfly, and cheerful_panda

say thanks for this post

The idea of the CC address is that anything CC’d to it will be posted (whether automatically, or manually by the admins) in a private Dynamo thread, which is linked on the resources page. Manual posts to the thread are also encouraged.

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

I agree with gorgeous_monarch_butterfly. I feel like providing the CC address both lowers the bar for Turkers who want to share their coordinations with other Turkers and it will also result in requesters taking the email more seriously because they may feel like a bigger group of Turkers are waiting for their response.

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly

says thanks for this post

http://wiki.wearedynamo.org/index.php/Communication_with_requesters

“CC: ethics@wearedynamo.org (emails sent to this address will be posted to a private Dynamo thread, so Dynamo users and Dynamo creators can help monitor and step in; you can also post emails manually in the thread if preferred)”

I’m really uncomfortable with the wording “step in”, and I asked on a Turking forum and they are also uncomfortable with it. Why does someone need to step in? You can give suggestions for the template emails, but to suggest you’ll need to “step in” can be read as insulting.

I also worry about the same template email being sent to a Requester multiple times. Won’t this make it look like we’re just a group of robots spitting out spam emails?

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

I didn’t write this but I think it was in case a Turker wanted to ask others to step in, there was a place to do so.

gorgeous_monarch_butterfly said:

Another thought I had, which does have some pros and cons, is what if cc’ing to a particular Dynamo email address triggered a copy of the email to be posted in a private thread in a special section of the Dynamo forum? Then both the creators and any individual Dynamo users who happen to have an interest in a particular case could keep up with it and be ‘ad-hoc volunteers’ temporarily as interested.

If you feel against this or you have an alternative way of saying it better, please make a suggestion. If others agree, you can change it.

Sign in/Register to reply to this post.

“CC: ethics@wearedynamo.org (emails sent to this address will be posted to a private Dynamo thread, so Dynamo users and Dynamo creators can help monitor and step in, if you ask them to; you can also post emails manually in the thread if preferred)”

I think that simple change gives the “power” back to the original actor.

Sign in/Register to reply to this post.

light_dragonfly, gorgeous_monarch_butterfly, and cheerful_panda

say thanks for this post

Thanks! Since this is a small change and I don’t think any one would be against it, I’m going to edit right now. If any one is against this please say so :-)

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly and cheerful_panda

say thanks for this post

I was shown this today, I thought it might help guide future changes to the guidelines:

http://www.apa.org/monitor/2014/05/online-research.aspx

Sign in/Register to reply to this post.

cheerful_panda and light_dragonfly

say thanks for this post

I shared the guidelines with a prominent academic, and he had two interesting observations: 1) The doc makes no distinction between hiring Turkers as a workforce to get work done, vs. running a study to understand humans. IRBs will make this distinction when they consider whether something is under their purview.

2) In terms of IP: how should the crowd’s contribution to the research be recognized?

Conversation copypasta’ed below in its entirety. Do you have thoughts? Should we change anything about the guideline?

~~~~Original email:

I was glad to see the guidelines.  I think that the crowdsourcing industry has been a bit caviler at times about the way that they conceive and treat their workers.  

In reading this document, I was struck by a couple of things.  For the most part, it treats the Turkers as workers and posits its ideas as a refined form of workers rights.  Indeed, it states that the Turkers are "a workforce not a voluntary study population." and argues that the origin of the document should hold researchers to a higher standard.  Yet, it misses two key elements that I anticipated that I would find.  The document makes no distinction between using AMT as source of service to support research and researching the nature of crowdwork, between using AMT to clean a large dataset and using an AMT session to study the nature of cleaning data with microtasks.  IRB's would generally make this distinction.  They would approve the former projects with little debate and would tend to balk at the latter.  (Our IRB approved an undergraduate project to clean a 18Million point data set with AMT without taking it to the board on the grounds that the crowd was doing work that could have been done by research assistants.  They would have had a very different approach to it if the student stated that she was trying to understand how the crowd resolved the conflicts in her data set.)

The second thing that I expected to see was a discussion of IP, of when it is necessary to recognize the intellectual contributions of the crowd to the research.  I am aware that in many AMT tasks, the crowd has little opportunity to make and communicate a substantial contribution to the investigator.  Again, the line is fuzzy but the problem is still real.   I can point to a number of discoveries made by crowd computing groups that were incorporated in to final reports.  Most were ultimately acknowledged (which is how I know about most of them), albeit a bit reluctantly and condescendingly.  Still, they were acknowledged.  

~~~My response: I actually tried to articulate the difference between service and crowd research, but others disagreed with me and said that how Turkers should be treated ought to be the same for both. I’d be curious for your thoughts on that.

The IP one is interesting! I’d like to raise that to the group.

~~~His response: Sure. Go ahead.
I think that the labor/research issue is tricky. While I understand that Turker’s may not see a difference, IRBs will, Hence, they might want to anticipate the approach IRBs will have to research. When they get a request to use AMT or any other site as a source of labor for processing data, they will approve it with little question provided the researcher says that they are following the labor standards of the site. I had an undergraduate make a request to use Crowdflower to categorize a large data set (18 Million as I recall but I suspect that number is growing.) They did not even take it to a faculty reviewer but approved it on the spot with the notation that it was unusually for an undergraduate to have so large a data set. If she was doing research on the crowd or on crowdsourcing or the crowdmarket, they will immediately start with the issue of informed consent and start the review from there. (This is, of course, the problem that caught Facebook recently.) What kind of notice must to give to the workers? What constitutes consent? and When can they remove themselves from the study? There will soon be long consent forms that go with any AMT study and require the approval of each worker. We may have to deliver them back to the IRB, but my IRB may just be touchy because it ran into a few problems in the early 00s.

The IP problem is tough.  Historically, (and you know that history is important to me) workers got no credit for their contributions if they followed the instructions given to them by the researcher and returned data that they did not interpret.  They got credit if they created or modified the original instructions - Gertrude Blanch got on Bethe's paper on the operation of the sun this way - or if they saw something unusual in the data - something that fell outside the requirements of the task.  This is the contentious bit.  There was the recent case with Boinc or Zooniverse where a crowdworker - I believe she was a high school biology teacher - saw a pattern in her cases and communicated her observation to the researchers.  The pattern proved to be interesting enough to warrant a paper.  After initially balking, the researchers did the honorable thing and put her name in the middle of a 50 or so person author list.  

Post if you want.
Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

I think that the labor/research issue is tricky. While I understand that Turker’s may not see a difference, IRBs will, Hence, they might want to anticipate the approach IRBs will have to research.

I was struck by this working with the IRB at my school earlier this year. They care about subject populations, but they have never had to think about the labor conditions by which which the research gets done. I am assuming this is because the IRBs were formed in a mid-century America when far more people were in unions and the minimum wage had not been broadly circumvented by “contracting” relationships. As the universities have faced funding crises and budget crises, there has been a growing spectrum of academic exploitation of cheap labor – from grad students as instructors to hiring adjunct lecturers for a fifth of what a professor would earn (e.g. say $12k a year for four classes). Crowdlabor powering labs also fits with this trajectory, though in many cases researchers are doing it for convenience too. I would imagine the cash strapped social scientists might be more moved by expense concerns.

My personal opinion (and I’m a researcher, not a worker, so my 1 cent) is that it would be cool if this document included a guideline that Turk labor on projects get at leased the normative Turker minimum wage. That would really push the IRBs into some new, important, and interesting territory of having to consider the ethics of data processing labor, not just data elicitation labor. An easy rebuttal from researchers would be that this would throw grad student research assistants into a tizzy since grad students are paid salary for unknown numbers of hours that may result in less then minimum wage. However, the grad student is basically like a salaried worker or even an apprentice, not an hourly. If the lab hired an hourly worker, they would clearly be bound by minimum wage laws.

But yikes, we don’t really have a process for amending the guidelines, particularly so early in the process. How to proceed? Perhaps we can kick off a thread to discuss adding a research processing labor clause to the document?

Sign in/Register to reply to this post.

cheerful_panda and light_dragonfly

say thanks for this post

gorgeous_monarch_butterfly took 45 minutes of their time to prepare this

The guidelines were already intentionally largely written to be universal to any type of HIT, survey or not, and some points specifically mention non-survey academic HITs (batch ‘data processing’). Such as “Example: When a large batch of HITs was posted by a new requester with no Turkopticon reviews …”, and “Consider the estimated duration and difficulty of your task when deciding about payment.”, and “Even if your university doesn’t force you to, it’s always a good idea to use a consent/intro page at the beginning of a survey, and/or paragraphs in the HIT content text for non-survey tasks.”, and “Make sure your instructions are written very clearly and comprehensively, particularly for batch HIT groups …” Any researcher agreeing to the guidelines should be doing so with the intent of following them for any HITs they post on MTurk, not just surveys, and not just those that their university requires certain forms of IRB approval for.

The process for amending the guidelines has a subpage devoted to it. But I think it’s important that this be used sparingly, so researchers can feel like they’re committing/agreeing to something that won’t keep getting changed on them frequently, which was the point of the ‘freeze’ after several weeks of incorporating input. Many people are uncomfortable agreeing to something unfinished/unstable, and if it seems like it’s never finished/stable, will never be comfortable agreeing to it.

Sign in/Register to reply to this post.

cheerful_panda and light_dragonfly

say thanks for this post

light_dragonfly took 10 minutes of their time to prepare this

Thanks! I understand where excited_iguana, his researcher friend, and tense_ringworm are coming from. It might be a good idea to make additions to the guidelines about using MTurk for labor since sometimes IRBs don’t review that work like they review the use of MTurk for research and experiments on Turkers.

Gorgeous_monarch_butterfly raises a fair concern though. We might not be at a state where making this change right now is our best move forward. My suggestion is that we keep this issue in mind and think about ways to incorporate it into the guidelines. We can then give the guidelines some time for people to become familiar and start using them. Later on as we see more examples of the guidelines in action and have more conversations with IRBs we can come back to this issue and think about ways to solve it.

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

Can we have an unlocked section of the wiki somewhere that becomes a staging area or brainstorming space for the next version?

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

light_dragonfly took 20 minutes of their time to prepare this

Sure, I created a page here: Academic guidelines brainstorm and staging. Everyone is welcome to use this page to author new content or make proposals for future changes.

My suggestion is that unless something urgent or necessary comes up, content on this page be looked at and debated every once in a while (maybe leave months or even a year in between). We should remember that going over and debating these changes takes up a lot of people’s time and energy, we should be considerate of that.

Sign in/Register to reply to this post.

gorgeous_monarch_butterfly

says thanks for this post

dark_bird_of_paradise took 1 minute of their time to prepare this

Where did all the Turkers go? I just noticed that the activity here seems to be limited to researchers lately…

Sign in/Register to reply to this post.