Although I cannot disclose much detail about this new creation, I can honestly say this is probably the most exciting web project I’ve worked on in a very long time.
As part of the development process, I am currently directing a series of usability tests to see how the online property behaves once it comes in contact with real users.
Enter Amazon.com’s Mechanical Turk
To perform these tests, I’m using a combination of live testers and also enlisting the help of Amazon.com’s Mechanical Turk.
If you are not familiar with Mechanical Turk, it’s an online crowdsourcing “artificial, artificial intelligence” platform developed by Amazon which gives businesses and developers access to a virtual workforce, available On-Demand and at a very low cost.
Mechanical Turk Workers are people who sign up to perform HITs (Human Intelligence Tasks) which are essentially tasks which require a human being, and usually cannot be done by a computer or at least done in a cost effective manner.
Examples of HITs include basic research tasks, comparing two photos to see similarities, moderating content and photos, transcribing a recording, data entry, etc.
There is a wide selection of applications built on top of Mechanical Turk, including several which focus on Web Usability. It is those Web Usability applications which I am currently “employing” on these tests.
Crowdsourced Web Usability Applications
As a model for the tests I am conducting, I have been following the recommendations of Steve Krug on his Web usability classic “Don’t Make Me Think: A Common Sense Approach to Web Usability.”
I’ve followed Steve’s guidelines previously in other web design projects, and I am now applying his line of thinking and suggested questions onto the Mechanical Turk platform… By using MTurk (that’s how it’s affectionately called, hence the URL MTurk.com) instead of exclusively live testers – I am able to conduct multiple simultaneous tests running concurrently and at a fraction of what it would cost to do it using in-person human testers.
Both TryMyUi.com and EasyUsability have been delivering some very complete answers which have been very revealing and are having a definite impact on improving the navigation and design of the project.
An Eye Opening Experience…
As a personal note, it’s really quite an interesting experience to see a web property through the eyes of remote users who have not been as close to the design as the developing team has been – and then seeing how things that seem obvious – are simply “not seen” by a completely new user first exposed to your design.
Believe me – it’s a wake up call to wind up literally yelling at the video which is playing the usability test… Things like “No!”, “That’s not how it’s used!”, “Are you blind?” “Wait! What are you doing?”, “Look closely – it’s there!” and things of that sort…
Sometimes you can be too close to something and totally miss the big picture…
Anyway – I find usability testing one of the most valuable parts of any web project and by using MTurk, I’m being able to conduct some excellent tests in a very short time (few hours from start to finish) and at a great price (TryMyUi charges approx. $35/ Recorded Video Test and EasyUsability $15/ Written Test.)
Both of the services I’ve listed allow for filtering the Testers down to specific demographics to refine to the target audience of the type of user who you would expect to regularly use your website (male/female, age, professional profile, income, location, etc.)
Other Services You Can Use
There is also a wide selection of other Human powered services that you can use for your own business processes for things like moderating content and photos, classifying items, etc. (Many companies are actually choosing to embed MTurk workers in their regular business processes just as you could embed a technology piece… Think of it as “plug and play virtual employees for simple tasks at a very low cost.”)
In conclusion, I am very pleased with the type, honesty, and quality level of the usability feedback I have been collecting with these services, especially at the price being offered
Considering the massive impact that a well designed website can have on business results (one of my favorite examples is “How Changing a Button Increased a Site’s Annual Revenues by $300 Million“) – I am definitely taking all this valuable feedback and working with the development team to refine and optimize the new web property.
How about you?
Have you had any experience using Mechanical Turk?
Do you regularly conduct usability tests on your websites?
I’d love to hear your experiences with both topics… Do you have any suggestions?