5/12/09

Crowdsourcing Part 3 - An Art and a Science

Carolyn Elefant posted an interesting comment to Part 2 of our Crowdsourcing dialogue. She noted that “managing a crowd sourcing project can be difficult.” This brings us to explore our methodology and the art of crowdsourcing. In a nutshell what we’ve learned is that managing crowdsourcing staff is unique. In crowdsourcing you pay for very discreet tasks performed by anonymous workers. We envision the emergence of crowdsourcing staffing companies to meet this challenge. Their role will be packaging and structuring projects in the most effective and efficient ways. In this new environment ‘employers’ will have to be quick on their feet to adjust to new worker behaviors and keep projects profitable. Adjusting our Methodology Ron Friedmann in his comment to Part 1 suggests “recognition” as a motivator for our crowdsourced staff. Unfortunately, we don’t have much opportunity for that with an anonymous staff (although we’re open to creative ideas). However, we can make adjustments to our compensation method. In requesting researched information, we had three main categories of responses: 1) Obviously right, 2) Possibly right (or wrong), and 3) Obviously wrong. Our new approach will allow us to differentiate between these and pay the right amount for each type. Instead of just a blanket ‘double-blind’ approach where we pay for two responses to all requests, we will only allow one response per request. Category 1 responses will be accepted and paid (almost half from our 1st experiment). Category 2 responses can be accepted, paid and re-posted for verification. Category 3 responses can be rejected and re-posted as many times as needed. This revised approach will give us better returns on our investment (such as it is). Under Consideration We think this adjustment to our compensation model will produce better results at lower costs. The next layer of modifications to our method will come through how we design our requests (tasks). One thought is to take unanswered requests and reformulate the question for re-posting. Perhaps campaigns of follow-on, modified requests will bring us our desired results. In our current experiment the request for a contact with the title GC could be modified to include broader terms or even out-sourced options. We might have to increase the payment amount in this situation (reflecting greater effort or knowledge), but that would make sense if the response information carried enough value. Whichever direction this experiment takes it is proving quite intriguing. We feel we’re on to something and will keep exploring the crowdsourcing idea to see what we learn.

Bookmark and Share

3 comments:

Greg Lambert said...

I'll point out a couple of other things that I noticed on this first round of testing.

1) The workers do tend to "think on their feet" when you give them a question. Sometimes they will go off the given instructions and find the answer to your question elsewhere. A lot of the time, it still doesn't give you a good answer, but sometimes it can help you find an alternative approach that you hadn't though of before.

2) I can't prove this, but I have a feeling that some of the "double-blind" entries were handled by the same person. I only say this because there were a couple of very wrong answers that were identical, but performed by different workers. This type of low wage, anonymous work will bring out some very ingenious people that can figure out ways to increase their revenues by either streamlining their process through technology (completely legit!) OR, by figuring out ways to beat the system (not so legit).

Toby's point on not automatically running a double-blind task, but rather putting the question out there once, putting the answer in one of the three categories, then aborting, rerunning, or modifying the next round is a much better approach for these types of tasks.

Marina Martin said...

I'm very intrigued by your experiment.

I work for Elastic Lab, which does a different type of crowd-sourcing -- we pre-select filmmakers to join our network, and then hire multiple filmmakers from within the network to shoot footage for various commercial video projects. This allows us to cover diverse perspectives and geographies and spread various components (b-roll, interview) across filmmakers of varying degrees of experience with a smaller budget than a typical production. (I'd also like to note that we pay every filmmaker for commissioned submissions, even if we don't end up using them; it sounds like MTurk is similar in that you still pay even if the answer was incorrect, yes?)

Does MTurk offer an ability to rate answers, so contributors could earn a reputation over time? I wonder if that would help self-manage the crowd-sourcing process. Our business model wouldn't work if we put out our projects to just anyone, but because we have a screening process (looking at demo reels and existing online content/feedback/reputation), it works extremely well. What if you could specify that an MTurk project was only open to people who had completed X projects and/or had a minimum feedback of Y?

Also, have you seen or heard of Smartsheet? (http://www.smartsheet.com) I haven't used it myself, but they're a local company here in Seattle. Would that make projects like yours easier?

Greg Lambert said...

Marina,

MTurk allows you to "accept" or "reject" the work done by the worker. So, in our case, there were answers where they either didn't follow instructions, or they pointed to the wrong person entirely. So, we rejected that work, and did not pay them for it.

I also learned the hard way that if you reject the work, there is not "un-reject". One of the poor workers answered the question in a "technically" correct way, and I should have approved the answer. However, at first the answer looked wrong, so I rejected. The worker sent an email (via MTurk's network, so I still remained anonymous) pointing out my mistake. I replied to him apologizing for the error on my part.

I'm not sure if MTurk allows you to rate the worker beyond "accept/reject", but there are ways to pay bonuses for some types of work. I'll have to look more closely at the rewards system to see if there are additional types of recognition that can be earned by the worker.

As for SmartSheet, that is a good looking 3rd-party app for MTurk. I've taken a look at it, but haven't used it yet because I wanted to test what it is like to use MTurk in its native format. The next run will probably be tested using something like SmartSheet, as it looks like it saves a lot of time on my end, and looks very easy to use and set up.

Thanks for the comments!!

 

© 2014, All Rights Reserved.