Forums » Campaigns » Call for comments: Turker-authored guidelines for research on AMT

Sorry: I don’t propose that we remove all reference to technical issues. (The ToS is nothing but technicalities).

I need time to catch up with the ideas, and maybe correspond with people smarter than I am.

Sign in/Register to reply to this post.

A forum colleague suggests:

Our code of ethics shall mandate that AMT academic researchers provide their own IRB with a copy of the platform’s ToS, as a requisite part of submitting their project for IRB approval.

IMO it’s an elegant and effective way to ensure that ‘clever’ researchers do not perform an end run around a technologically ignorant IRB. (The IRB might not know technology, but it is in the business of reading whatever is attached to the project application).

It forecloses a lot of opportunities for “I didn’t know the gun was loaded” sorts of mischief.

To consider my favorite whipping boys: It could not be apparent to workers that HITs published by Minnesota researchers violated AMT ToS.

An IRB, aware of the project's details, might have considered ToS clauses like:

[ ] disrupting or degrading the operation of any website or internet service

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

This is a good point cheerful_panda, I believe we can expand on this section to cover these details: Abide by AMT TOS, you are more than welcome to edit the section.

Thanks to all for your feedback and comments. I believe we’ve reached a point where everyone knows what they like and don’t like about the guideline, and we face the danger of getting lost in back and forth discussions. A lot of time and energy has been put on authoring these guidelines and you have done an amazing job at creating and structuring a very well written document, thanks! The most important point now is concentrating on what we all agree on and pushing for it. I’m inviting everyone to think about their last concrete proposals to make changes (if any) before freezing the document and calling out to requesters and Turkers to sign it. Thanks!

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

cheerful_panda said: “Let’s focus on the ethical core ideas, with a view to an audience of academic IRBs as much as researchers. The spinoff project can deal more directly with the gritty technical details appropriate to an audience of researchers with assorted, highly specific, needs.”

I agree with this as well. There are lots of “how to be a skilled requester” documents on the web. The ethical guidelines should be concise and clear. However, I do like that someone can click to the other longer page on “basics” to get more discussions on this topic. How to treat the guidelines as shared and stable while also including explanatory context? What if we very clearly publish the guidelines page as an uneditable HTML page but leave the “Basics” as an editable wiki that is living and contested (and clearly declare it as such)?

Would that be suitable?

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

I favor that idea; but I am eager to hear from gorgeous_monarch_butterfly, whose contributions have a much more positive tone than mine.

I disfavor treating Inquisit as any kind of special case ethically. (In a technical document, it surely deserves attention as its very own kind of special snowflake). If gorgeous_monarch_butterfly were agreeable, I’d like the downloads section to include something like (this includes Java programs and plugins such as Inquisit).

Sign in/Register to reply to this post.

light_dragonfly and gorgeous_monarch_butterfly

say thanks for this post

If gorgeous_monarch_butterfly were agreeable, I’d like the downloads section to include something like (this includes Java programs and plugins such as Inquisit).

Done. I hope this will now be satisfactory.

Regarding what pages end up where and in what format, I have been discussing the potential logistics of this with N in other mediums, and would prefer not to have to repeat everything and get into a debate about it here too at this time. I am not presently in favor of the proposal by tense_ringworm, as cheerful_panda should be aware of from some of our own discussion this morning. The proposal of treating part of the document differently from other parts had only arisen due to the specific consternation about how Inquisit was covered, which has hopefully now been resolved above, leaving no reason it can’t move forward as a whole as it was intended.

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

If gorgeous_monarch_butterfly were agreeable, I’d like the downloads section to include something like (this includes Java programs and plugins such as Inquisit).

Done. I hope this will now be satisfactory.

Regarding what pages end up where and in what format, I have been discussing the potential logistics of this with N in other mediums, and would prefer not to have to repeat everything and get into a debate about it here too at this time. I am not presently in favor of the proposal by tense_ringworm, as cheerful_panda should be aware of from some of our own discussion this morning. The proposal of treating part of the document differently from other parts had only arisen due to the specific consternation about how Inquisit was covered, which has hopefully now been resolved above, leaving no reason it can’t move forward as a whole as it was intended.

Thank you, gorgeous_monarch_butterfly! I agree with this.
I anticipate additions or deletions in “Links to other resources” as the document is circulated.
I hope for augmentation of the current section “Communication with requesters.”

Some individuals already signed off on the entire document.

I find that more instructive than encouraging.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

That’s why I’m sorry to see the core document get excessively mired in technical detail in what purports to be a high-level document on ethics. I certainly hope this document sees fewer revisions than it will if it ties any of its own legs directly to the five-year-old “beta test” which Amazon calls “policy.”

That’s also why I’m pleased to see our core document raise such issues as “personally identifiable information” as our own concern; because, Yes, Amazon might in theory suddenly say it’s okay, and No, for us it still isn’t.

Woo. So let me transform this into a concrete proposal. Simply state in the document that we expect requesters to abide by the Terms of Service of the platform they are using. (Doesn’t matter the platform, really.) Remind them of the basic requirements. Then we focus the document only on issues that are ethical issues we would hold on to regardless of what the TOS says.

(Sometimes a partial solution to a problem changes the problem. I’ve been racing to catch up, and I don’t want to leave one leg up in the air with excited_iguana).

I’m satisfied that the current level of detail is appropriate. Until such time as Amazon Inc. may be ready to launch a real “beta test,” a lot of AMT’s landscape is built on shifting sand. I think this document does a very good job of focusing on essentials; its touchstones are likely to be permanent.

Sign in/Register to reply to this post.

light_dragonfly and gorgeous_monarch_butterfly

say thanks for this post

cheerful_panda said: “Let’s focus on the ethical core ideas, with a view to an audience of academic IRBs as much as researchers. The spinoff project can deal more directly with the gritty technical details appropriate to an audience of researchers with assorted, highly specific, needs.”

I agree with this as well. There are lots of “how to be a skilled requester” documents on the web. The ethical guidelines should be concise and clear. However, I do like that someone can click to the other longer page on “basics” to get more discussions on this topic. How to treat the guidelines as shared and stable while also including explanatory context? What if we very clearly publish the guidelines page as an uneditable HTML page but leave the “Basics” as an editable wiki that is living and contested (and clearly declare it as such)?

Would that be suitable?

(still playing catch up)

I think we must consider individual “links to other resources” as impermanent anyway; things on the Net have a way of moving around. I am happy (if you are) simply using that section to reflect environmental changes and other living content.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

I’m satisfied that the current level of detail is appropriate. Until such time as Amazon Inc. may be ready to launch a real “beta test,” a lot of AMT’s landscape is built on shifting sand. I think this document does a very good job of focusing on essentials; its touchstones are likely to be permanent.

Also, the MTurk sands have historically shifted quite slowly and minimally. ;-) The API documentation has had 8 fairly-minor changes in the 5 years it has existed as a separate document, and the CLT documentation has had only 5 changes. Somewhere in one of the MTurk-related papers/posts I’ve read, a researcher said something along the lines of that MTurk updates/changes so little, they can keep reusing old slides/screenshots for years and nobody can tell the difference.

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

You’re correct about the technical interface from a requester’s point of view. Of course, we can probably avoid API questions today.

But Amazon policy changes are sometimes the bastard child of bureaucratic decree and slipped deadline.

As a worker, I crave the raw numbers for my returned and abandoned HITs, which appeared on my dashboard as recently as April.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

Thanks everyone, I’m so glad that we got to a point where we all agree with the guidelines we have! :-) This is a major accomplishment.

I have a proposal for taking the next step on this. If there are no objections, I will freeze the whole guideline on Wednesday 8/20 at 11:59 PM (PST). We will call this version 1.0. Any further changes happen by:

  • If the change is a minor edit (e.g. fixing a broken link or adding a signature), the admins will temporarily open the document and make the change.
  • If the change is a major edit, a proposal has to be submitted to the Dynamo thread. After discussing the change the Wiki will be opened and edited, after which the version number will change and all those who have signed the guidelines will be informed by email of this change.

I have added this information to the wiki here, so IRBs, requesters, or other Turkers who find the document can learn how it is maintained. It might be worth adding a few sentences about how it was created. Feel free to edit this page. Thanks!

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

So, I did a quick Google search for MTurk IRB and I stumbled upon this: http://www.behind-the-enemy-lines.com/2009/08/get-consent-form-for-irb-on-mturk-using.html

This is a sample IRB proposal for using MTurk. It might be worth looking at it to get a better sense of what people usually include in these proposals and make sure we are addressing all the things that we would like a researcher and their IRB to pay attention to. I know you guys are much better than me at doing this so I’m posting here. Thanks!

Sign in/Register to reply to this post.

cheerful_panda

says thanks for this post

A couple of us took a whack at one of the only remaining sections: the template emails to send to requesters who are in violation. http://dynamowiki.herokuapp.com/index.php/Communication_with_requesters

One open question — we had discussed allowing workers to cc some sort of Dynamo volunteer list for moral support or backup. What do we do about that?

Feedback, please! - E.I. and T.R.

Sign in/Register to reply to this post.

light_dragonfly and cheerful_panda

say thanks for this post

A couple of us took a whack at one of the only remaining sections: the template emails to send to requesters who are in violation. http://dynamowiki.herokuapp.com/index.php/Communication_with_requesters

One open question — we had discussed allowing workers to cc some sort of Dynamo volunteer list for moral support or backup. What do we do about that?

Feedback, please! - E.I. and T.R.

tense_ringworm has access to a couple letters I wrote today as an experiment. (The originals went to real live IRBs). I tried to use the Talk page to post them as examples of what we don’t want, but my attempts to format them came to grief.

I think the current examples – like the emails I sent – are ‘way too slow for a worker dealing with a past or imminent rejection. It takes too long for things to percolate through the bureaucracy and for the correct fingers to be placed on the right buttons. There will be too many times when on Day 25 a requester agrees to reverse some rejections, but then takes a week to get the details correct. Oops. (That’s why gorgeous_monarch_butterfly wrote “Know how to undo a rejection before you do [reject]”).

Speaking for myself: if the 30-day clock is already ticking on a rejection, I won’t give three warnings. I won’t allow the requester more than one unanswered email, or let the non-answer persist for more than 48 hours, without some sort of escalation. Yet my last rejection came from a guy from University of Illinois/Urbana-Champaign who deleted his own data in a week and still took thirty-five days to reply. (The IRB was much more responsive than he was, but I still have the rejection).

Outside the immediate din of battle, I think the only kind of AMT workers who will come to Dynamo with ethical questions is the kind that’s here already. There will be a few. But they won’t be a majority of complainants. The people of my class and tribe are a pragmatic people. We get excited mostly about money, rejections, warning blocks, and loss of time. But if our problem is loss of time, we’re not going to spend a lot of our time to redeem it.

If I ever have time to address an issue which is also addressed by Amazon’s requester ToS, I expect I’ll go to the IRBs directly (as I did with my sample). I have no duty to a requester to remind him of duties of his own. There’s probably also a lot to be said for the patient education of such requesters, but I’m not eager to be one of the people saying it.

In that sense, a Dynamo volunteer board would be useful to me only if the “ethical violation” were not also a violation of Amazon ToS, and did not involve an actual or pending rejection or warning block against me or a tribesman. Other people would be more willing to come to Dynamo with ToS issues, but they also might expect things to move at an unrealistically brisk pace.

Sign in/Register to reply to this post.

tense_ringworm

says thanks for this post

I can see how the courtesy email can seem slow. A requester once called the IRB because he was angry about a Turkopticon review. The IRB contacted the Turkopticon team, established that TO data was not research data (no generalizations were being made from it; it wasn’t being published in research), and they moved on. We also saw how the Minnesota IRB went down. TO mod Anne calls IRBs all the time to get issues resolved quickly when requesters don’t respond. I have yet to hear of any examples of someone losing their career because of IRB issues (except maybe medical researchers that defraud patients or something). I support workers getting their issues resolved as quickly as possible. (Emailing a requester and giving them 12 hours might actually be quicker than getting IRB involved, but waiting several days for a requester might not.)

I’d be open to scrapping the ethical review board bit. I think it raises a lot of questions about governance that I’m personally not prepared to figure out and I don’t see anyone else stepping up to commit to figuring it out. My original hope for these guidelines was that they would be a tool for individual Turkers to use as necessary. If others thing the review board is really valuable, I’m happy to be convinced too (stepping up to work on it is most convincing ;) ).

Sign in/Register to reply to this post.

So I think we should have twin goals: 1) Give the requester the benefit of the doubt 2) Make clear that time is important to workers, so things are fixed expediently

Here’s my proposal: 1) Email the requester directly. Point out the problem. Ask for a response in 48 hours. If it’s a bad rejection (as opposed to other problems), the email points them at the un-reject guide and gives them one week to fix it. 2) If they don’t respond in 48 hours, send a reminder and let them know that you might contact the IRB. Likewise, if they don’t fix the rejection in 1 week, send a similar email. 3) If they’re being unresponsive or unhelpful, contact the IRB.

Emailing an IRB about every piece of misconduct will in aggregate make IRBs gunshy about letting any researchers use AMT. So I favor letting personal communication be the first round of defense.

Turkers should not be bound to this recipe, but the goal would be to provide a useful guide if you’re feeling lost.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

TR wrote: “… Emailing a requester and giving them 12 hours …”

This may have just been an example, but wanted to point out that I think a 12-hour turnaround limit is a bit much to expect for email responses; many do/should have no problem meeting that, but it is probably more reasonable to give at least 24 hours on anything like this, and probably at least 3 days if there’s a weekend included.


TR wrote: “I’d be open to scrapping the ethical review board bit. I think it raises a lot of questions about governance that I’m personally not prepared to figure out and I don’t see anyone else stepping up to commit to figuring it out. …”

wiki currently says: “… CC: [ethics team email], so the Dynamo creators can help monitor and step in …”

I’m not clear on exactly what is being talked about here or where everything stands in regard to the communicating-with-problem-requesters-and-IRBs plans, but if you the academic researchers who created Dynamo are willing to monitor such correspondences if a worker wants to cc them to you, I don’t know that this would really need to take the form of a formal ‘ethical review board’ with governance and additional volunteers and whatnot, at least not unless/until things grow to a high volume…

Another thought I had, which does have some pros and cons, is what if cc’ing to a particular Dynamo email address triggered a copy of the email to be posted in a private thread in a special section of the Dynamo forum? Then both the creators and any individual Dynamo users who happen to have an interest in a particular case could keep up with it and be ‘ad-hoc volunteers’ temporarily as interested.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post

Sounds good, except I’d think saying ‘a private Dynamo thread’ (one per requester/situation), rather than one thread being ‘the’ thread for everything, would be preferable long-term; if you ever have more than one situation going on simultaneously, it could get pretty confusing otherwise.

Sign in/Register to reply to this post.

light_dragonfly

says thanks for this post