5/21/09

Crowdsourcing Part 4 - Time, Money and Quality

Toby and I had some left over money in our MTurk Crowdsourcing account, so we thought we run a few more tests to see what kind of results we could get from the vast amount of potential workers. In the process of blowing the remaining $14.00 we had in the account, we learned a thing or two and thought we'd share our new found knowledge with you.
We ran 3 tests where we asked the following:
Test 1: Bullet-Point Reviews
Review the following legal article and give a synopsis of the article 5 bullet point of approximately 15 words each.
Test 2: Paragraph Reviews
Review the following legal article and give a synopsis of the article in approximately 100 words.
Test 3: Copy & Paste Article Title, Date and Intro Paragraph
Open the following URL and copy the Title, Date, and First Paragraph of the legal article found at that site.
For tests one and two, we took the same 10 legal articles and submitted them on Saturday and offered to pay 25¢ for each correct answer. The articles that remained on Monday afternoon were resubmitted and we then offered 50¢ for each correct answer. For test number three, we submitted 30 URL's on Monday and offered 5¢ per answer.
Lessons Learned:
  1. Don't submit anything on a Saturday morning.
  2. Two-Bits can still buy some things
  3. The more "mechanical" the work, the better the results
  4. MTurkers are people... some of them pretty smart people! (we even had one comment on this blog about her experiences)
For this blog entry, I'm going to show all of the results of the "Bullet-Point" MTurk project and discuss the what we went through and learned in the process.
First of all, we took 10 Health Care articles that were written by large law firms as either client alerts or general publications. We submitted the MTurk project asking the workers the following:

Write 4-5 Bullet-Point Reviews of the Following Law Articles

  • Each Bullet-Point should be approximately 15 words in length.
  • Do not repeat the title of the article or the author name(s) in your summary.
  • Summary must be in your own words, do not cut-and-paste sentences from the article.
  • Try to mention approximately 4-7 keywords relating to the article topic.

Law Firm: [Law Firm Name]

Article Title: [Name of Article]

Website: [URL Link]

Bullet-Point 1:

Bullet-Point 2:

Bullet-Point 3:

Bullet-Point 4:

Bullet-Point 5:

Please provide any comments you may have below, we appreciate your input!

The instructions were pretty specific, and we submitted this on Saturday and paid 25¢ for each answer. By Monday, we had 4 of the 10 answered and decided to "kill" the project at that amount and resubmit the final six at 50¢ per answer. Here are the four answers we received at the lower 25¢ rate:

Firm Name: Holland & Hart
Article Title: Internet Medicine Part IV: `Health 2.0`
  • Health 2.0 represents a new paradigm in the relationship between doctors and their patients
  • Many websites are evolving into the role of information providers, acting as intermediaries between patients and their physicians and care providers
  • Websites are key information providing sources to patients because U.S, physicians don't receive compensation for providing medical advice to patients
  • Electronic medical records offer promise for an advertising-funded model for online information dissemination to patients
  • Privacy issues are highly likely to continue to be an impediment to personal health records online because of confidentiality concerns

Firm Name: Baker & Daniels
Article Title: EEOC issues reminder on ADA compliance when dealing with the H1N1 flu virus
  • Employers are limited in medical examination and disability information requests for current employees
  • Employers may not request disability requests or demands to prospective employees
  • Employers may inquire of new employees disability related inquiries, but these inquiries must be standard practice and may not segregate
  • Disability information acquired from employees must be carefully guarded and kept confidential
  • Recommends employers identify standard best practices related to medical outbreaks and emergencies, including options for telework, as long as these do not discriminate against those with disabilities

Firm Name: McDermott Will & Emery
Article Title: Senate Health Care Reform Policy Options: Medicare Advantage
  • There has been released of three particpated health reform option paper from the Senate which was voted April 29,2009.
  • With the new laws that have been passed enfornced the four most important things from the Medicare Advantage.
  • -
  • -
  • -

Firm Name: Dinsmore & Shohl
Article Title: Medicare DMEPOS competitive bidding program becomes effective April 18, 2009
  • Will affect DME suppliers in ten selected cities (Cincinnati, Cleveland, Pittsburgh) and surrounding areas
  • Replaces existing fee for service model with bids for competitively priced items in ten product categories
  • No effect on current Durable Medical Equipment (DMEPOS) suppliers
  • Medicare is interested in input from DMEPOS suppliers impacted by the program
  • More information in presentation by Mark A. McAndrews at Ohio Pharmacists Association Annual Conference

When we saw that only 4 of the 10 were answered by Monday morning, we began to question our logic of starting a project like this on a Saturday. So, we decided to stop this project on Monday afternoon and resubmit the remaining six articles at 50¢. It still took us another 24+ hours to get the remaining six answered, even after doubling the amount we were paying.
Tomorrow's post will show the final six results and talk about whether more money + less time really equals better results.

Bookmark and Share

0 comments:

 

© 2014, All Rights Reserved.