Comments on 3 Geeks and a Law Blog: Crowdsourcing Part 3 - An Art and a Science

Marina, MTurk allows you to "accept" or "reject" ...

2009-05-12T20:27:00.000-05:00

Marina,

MTurk allows you to "accept" or "reject" the work done by the worker. So, in our case, there were answers where they either didn't follow instructions, or they pointed to the wrong person entirely. So, we rejected that work, and did not pay them for it.

I also learned the hard way that if you reject the work, there is not "un-reject". One of the poor workers answered the question in a "technically" correct way, and I should have approved the answer. However, at first the answer looked wrong, so I rejected. The worker sent an email (via MTurk's network, so I still remained anonymous) pointing out my mistake. I replied to him apologizing for the error on my part.

I'm not sure if MTurk allows you to rate the worker beyond "accept/reject", but there are ways to pay bonuses for some types of work. I'll have to look more closely at the rewards system to see if there are additional types of recognition that can be earned by the worker.

As for SmartSheet, that is a good looking 3rd-party app for MTurk. I've taken a look at it, but haven't used it yet because I wanted to test what it is like to use MTurk in its native format. The next run will probably be tested using something like SmartSheet, as it looks like it saves a lot of time on my end, and looks very easy to use and set up.

Thanks for the comments!!

I'm very intrigued by your experiment. I work for...

2009-05-12T19:47:00.000-05:00

I'm very intrigued by your experiment.

I work for Elastic Lab, which does a different type of crowd-sourcing -- we pre-select filmmakers to join our network, and then hire multiple filmmakers from within the network to shoot footage for various commercial video projects. This allows us to cover diverse perspectives and geographies and spread various components (b-roll, interview) across filmmakers of varying degrees of experience with a smaller budget than a typical production. (I'd also like to note that we pay every filmmaker for commissioned submissions, even if we don't end up using them; it sounds like MTurk is similar in that you still pay even if the answer was incorrect, yes?)

Does MTurk offer an ability to rate answers, so contributors could earn a reputation over time? I wonder if that would help self-manage the crowd-sourcing process. Our business model wouldn't work if we put out our projects to just anyone, but because we have a screening process (looking at demo reels and existing online content/feedback/reputation), it works extremely well. What if you could specify that an MTurk project was only open to people who had completed X projects and/or had a minimum feedback of Y?

Also, have you seen or heard of Smartsheet? (http://www.smartsheet.com) I haven't used it myself, but they're a local company here in Seattle. Would that make projects like yours easier?

I'll point out a couple of other things that I not...

2009-05-12T08:13:00.000-05:00

I'll point out a couple of other things that I noticed on this first round of testing.

1) The workers do tend to "think on their feet" when you give them a question. Sometimes they will go off the given instructions and find the answer to your question elsewhere. A lot of the time, it still doesn't give you a good answer, but sometimes it can help you find an alternative approach that you hadn't though of before.

2) I can't prove this, but I have a feeling that some of the "double-blind" entries were handled by the same person. I only say this because there were a couple of very wrong answers that were identical, but performed by different workers. This type of low wage, anonymous work will bring out some very ingenious people that can figure out ways to increase their revenues by either streamlining their process through technology (completely legit!) OR, by figuring out ways to beat the system (not so legit).

Toby's point on not automatically running a double-blind task, but rather putting the question out there once, putting the answer in one of the three categories, then aborting, rerunning, or modifying the next round is a much better approach for these types of tasks.