In mTurk, how can I use participation in a previous HIT (or series of HITs) as a qualification?

Question

I am using mTurk for surveys, and I need a way of making sure that people who have participated in a previous survey / HIT do not participate in certain future surveys / HITs. I am not sure whether I should do this as a qualification or in some other way.

I know there is some way to do this, but I have no idea how. I have very limited programming experience and would greatly, greatly appreciate specific instructions on how I might do this. My understanding is that I might need to use AWS? Many thanks!

Answer 1

Mass rejections as suggested above are a really, really bad idea in terms of your reputation as a requester. You are much better off creating a Qualification for the new HIT, which automatically grants a score of 100 (or whatever) to anyone who takes it, and assigning scores of zero to everyone who has done the previous surveys. This prevents repeats but doesn't annoy any of your workers.

The easiest way to create a Qualification is at https://requester.mturk.com/qualification_types.

If you download the csv of workers from here https://requester.mturk.com/workers, you can assign scores to workers who have done the previous HIT(s).

To make the qualification grant scores to new workers automatically requires the API, though.

Answer 2

Here's a hacky way to do it:

• When you accept HITs for surveys, save every participating worker's ID.
• In the writeup, note that "if you've done previous surveys w/ us, then you can't do this one (IE, you can, but we won't approve it)".
• When you approve HITs, cross-reference the worker ids with anybody who participated in a previous survey, and reject the hits of any that match.

If you're doing enough surveys, then yes, you probably want to use AWS API for at least the approval part. otherwise, most things appear to be do-able from the requester interface.

Answer 3

Amazon Mechanical Turk service has this option for requesters to grant their workers by Qualification_Type. In this way by connecting your HITs to a qualification_type naming "A", then granting workers exactly the same qualification_type, only workers who have that qualification can see and work on HITs.

First, creating desired qualification types through mturk web UI.(it is only name and description) requester.mturk.com > manage > QualificationTypes. It will give you a qualification id after generating it. (you will need it soon)

Second, in HIT creation loop, you have to use QualificationRequirement class. (I am using java code and it looks like the below-mentioned code):

QualificationRequirement[] qualReq = new QualificationRequirement[1];
qualReq[0] = new QualificationRequirement();
qualReq[0].setQualificationTypeId(qualID);
qualReq[0].setComparator(Comparator.EqualTo);
qualReq[0].setIntegerValue(100);
qualReq[0].setRequiredToPreview(false);

then in HIT creation loop, I will use this:

try {
            hit = this.service.createHIT(null,
                    props.getTitle(),
                    props.getDescription(), 
                    props.getKeywords(),
                    question.getQuestion(),
                    new Double(props.getRewardAmount()), 
                    new Long(props.getAssignmentDuration()),
                    new Long(props.getAutoApprovalDelay()), 
                    new Long(props.getLifetime()),
                    new Integer(props.getMaxAssignments()),
                    props.getAnnotation(),
                    qualReq,
                    null);

Third is assigning the qualification type to the workers that you want them to work on your HITs. It is very straightforward, I usually use mturk UI to do it. https://requester.mturk.com/ > manage tab > Workers. You should download the CSV file if you want to assign this qualification to a bunch of workers. (Workers are who worked with you in the past)

you could notify workers by sending them an email after qualifying them

Notice: Some workers are very slow in answering your new HITs after qualifying them; so keep in mind that you should have some backup plan and time if you will not receive enough response in a certain amount of time.