With the screen-sharing technology I use, I can quickly usability-test your user-interface designs with users located anywhere in the world.


I've worked on projects where friends and family are suitable test-participants.

On other projects, I needed to recruit dozens of busy, highly-specialized professionals located around the globe.

A good usability review will find many of the usability problems that would be found in a usability test.

Much of the preparation time and cost for testing is reduced if I was involved in earlier research or design phases of the project.

For an Internet banking site, a key usage scenario could be:

"Find which credit card best suits your needs"

 


Testing is a usability lab, Toronto.

 

There is no guarantee that the usability and user-experience of your new application will be perfect, or even acceptable, despite your insightful user-research, the skills and opinions of your user-interface designers, and ongoing design reviews by usability experts.

The performance and feelings of real users in their real world are the measures of a usable application.

The earlier that you properly test whether your application's interactive design is actually usable, the easier, quicker, and cheaper it is to fix problems your find. And by continuing to test your design as it progresses, the more you can ensure that your have identified and fixed potential problems before they are initially coded or actually released in the application.



Usability Testing- Why and When
  • Often, my clients need usability testing to identify and fix problems in their current applications that are currently "causing grief" for its end users.

  • In major development projects, I conduct usability tests of low- and mid-fidelity user-interface prototypes to validate high-level design concepts, discover and fix usability problems and learn more about users and their tasks. The goal of this "formative usability testing" is gathering proper feedback to improve design before development is so far along that it becomes too expensive to change.

  • On occasion, I run usability tests specifically designed to measure usability ("summative testing") on pre-release (QA/UAT/Beta) applications to determine whether it has met stated usability targets, and to identify and fix any serious problems before release. User satisfaction, as well as objective performance measures such as successful task completion, time to complete the task, and number of errors, are the primary measures. This testing can involve competitive benchmarking as well.



Overview of a typical testing project

Below are the steps I complete to enable and conduct a usability test. A typical project can be completed in six-weeks after you first contact me, although quick tests can be completed in as little as a week or two.

1. Plan the test

Establish the purpose of the test and develop a strategy to deliver quality results given available budget, staff, and time.
    • I explore your application or prototype to understand how it works and to identify potential usability issues to watch for in testing. I also step through the application / prototype while working through a set of key usage-scenarios This link opens another browser window that have been identified by your application's business stakeholders and from prior user research. If a prototype is being tested, I assess its fidelity to an actual interactive application and determine how to gain reasonable feedback from user testing.

    Determine which types of users to test, how many of each type to test, and how to get them.

    • The type of users and the number of each type are determined from your project's personas (if they were written), or by consulting your application's marketing, support, or business functions.

    • Your type of application will determine the most effective strategy to recruit suitable end-users for the usability test. For public Internet sites I usually write a detailed recruiting screener and engage a professional recruiting agency to locate and schedule qualified users. Participants are usually paid for their participation.

    • Where to run the test also depends on your particular project requirements. The only real requirements are that the location provides easy access to test-participants, does not present distractions, and will allow the test-application to be run. An office (free), a hotel meeting room (~$300/day), a focus group facility (~$1,500/day), or a usability lab (~$1,500) are all options. Testing can occur in multiple locations if required.

      With Internet
      screen-sharing technology (GoToMeeting, WebEx), a live usability test can occur with the user sitting in one location and the usability analyst observing thousands of miles away. Reasonable usability tests can even be conducted with automated tools (such as Keynote This link opens another browser window), without requiring the usability analyst to attend each session live.

    Determine how to conduct the usability test sessions.

    • Develop a testing protocol to run each usability test. The protocol includes initial interview questions, instructions to the user, a set of key user-scenarios that each test-user will attempt to complete with the application / prototype, notes about design issues for me to watch for during the test, and wrap-up interview questions and satisfaction /user-experience rating scales.

    • Typically, 6 to 10 key user-scenarios are tested in each usability test. Six users per target market segment are tested, so a typical project involves a total of 12 to 24 test participants. Group-based test sessions are also possible if appropriate for your project requirements.

2. Enable testing

  • Reserve appropriate testing facilities for scheduled dates.

  • Determine characteristics of target market segments. If necessary, write the participant recruiting screener. Execute participant recruiting strategy.

  • Dry-run testing protocol with suitable participants; revise as necessary.

  • Set-up test environment at testing facility. Your project's executive sponsors, business stakeholders, and development team are encouraged to observe test sessions from the facilities observation suite or by watching the user's screen and face using my remote viewing technology.

3. Conduct test sessions

    • Actual user-testing is conducted over a one or two week period.

      One user is tested at a time in a 1 to 2 hour test session. The test sessions run any time between 10:00am and 10:00pm.
Usability testing in the IBM Rochester Minnesota Lab, 1992.
    • In most projects, the test participant is encouraged to think-aloud as they use the application / prototype to complete each user-scenario. I watch and listen closely while the user works. I try to understand their moment-by-moment motivations and decision processes and note when (and possibly why) they become confused and wander off the path from completing the task or can't proceed at all. If they haven't already complained to me, I ask for their feedback, but only when appropriate.

      To expedite analysis, I make time-stamped notes during each usability test.

    • At the end of each session, the participant rates their experience using the application on a variety of key usability and user-experience attributes, and I interview them to ensure that the feedback I observed is accurate.

    • I record all audio and user's screen-activity during the test session for detailed analysis. I provide your with all recordings, and can also produce a "Highlights" DVD that isolates particular issues.

4. Review test session data to identify issues

    • When all usability tests are complete I consider all the testing data and review recordings in detail as necessary to identify all user-problems, rate their impact, diagnose the underlying design problem, and determine how to fix it.

5. Produce Deliverables

Deliverables are tailored for your own project's objectives: to identify and fix problems in a current release, plan for future releases, or convince executives to fund redesign projects.

  • List and prioritize high-level issues that compromise usability and user-experience. Provide recommendations for each issue. Typically 10 to 15 high-level issues are identified.

  • Document detailed usability issues, rate their impact, and provide recommendations for each. Typically, hundreds of detailed issues are identified. Issues can be documented for easy integration to your bug-tracking system, if appropriate.

  • Provide DVD-video recordings of all test session activities. If necessary, produce a DVD of session highlights.

  • Present findings to project team, product management, and executive stakeholders.

6. Take action on results

If my role continues beyond the test itself, I work with your team on design solutions to the usability problems identified in testing.
    • Turning a large set of test results into a more usable application requires an intimate understanding of the causes and user-impact (frequency+severity) of each of the identified usability problems. But there may be several ways to address any one usability problem. And resources to address the problems are finite: resource should be assigned to make design changes that will bring the biggest improvements.

      To accomplish this, each problem needs to be prioritized not only in terms of its user-impact, but each potential design-solution must be evaluated in terms of technical feasibility, its time and difficulty (cost) to implement, and its compatibility to the future application roadmap.

      • It may be costly to fully address a particular problem, but a quick fix could lessen the severity of the problem until it can be properly addressed in a future release.

      • It could be that fixing several low/medium-impact (but easy-to-fix) problems encountered in a single user-scenario, such as completing a form, will add-up to significantly improving its overall usability.

    • When every one of the usability problems identified in testing are considered in terms of their user-impact and cost to address, then your project team can focus in a way that will most benefit the end-user and your business goals given available resources. Keeping me involved will ensure that your application's end-users are properly represented in this process.

    • I can also remain involved in your project to fully design the solution for identified usability problems and/or to review or usability-test proposed design solutions to ensure that they solve the problem.


Pricing of a usability test

I have conducted so many user-testing projects that I can provide a fixed-price quote for your project requirements. Quick tests of a simple application may be as little as $5,000 to $7,500. Typical tests that involve a moderately complex application, rigorous end-user definition and recruiting, and 12-24 user-participants: they generally range from $15,000 to $45,000.

Pricing variables

  • Test-application - Pricing depends on the operational complexity of your application, the type and completeness of the design artifacts available for testing, the number of key-usage scenarios selected for testing, the ability of these artifacts to support the selected test scenarios, and the ease of creating reasonable test scenarios.

    Design artifacts for testing range from early static prototypes to full operational applications. A good test requires that the test-scenarios feel realistic to test-participants or their confusion with the testing situation itself will overshadow their experience of the "application" of interest. A static prototype may need additional pieces of the task-flow represented. Or an operational application may need a set of non-production data to provide a realistic situation.

  • Test-participant recruiting - Pricing also depends on the number of target markets to be recruited and tested (typically, 6 test-participants per each identified target market, user-role, or persona), and the complexity and expense of the necessary recruiting effort. I engage professional recruiting agencies when required.

  • Deliverables - The style and scope of the test-deliverables can vary considerably, which also affects cost. I optimize test-deliverables to address the primary purpose of your testing: identify problems for development to fix in a maintenance-release, help executives and product management plan for a major redesign, explore an early prototype (formative testing) in a design project, or validate a design prior to development (summative testing). In addition, nimble and lean startup companies rely on team working-sessions more than detailed documentation, whereas projects at large institutions must adhere to corporate requirements.

Basic expenses included in the fixed-price cost:

  • Recruiting costs using a professional recruiting agency are typically $75 per attendee.
  • Incentive paid to each end-user to participate typically ranges from $0 to $50 per hour but can be more depending on characteristics of your target markets (e.g., college students vs. CEOs).
  • Test facility costs range from $0 (your office) to $500 (hotel conference room) or up to $1500 per day (usability lab or focus group facility).
  • Travel to remote locations.


My background in user testing

I've lived in user-testing labs for nearly 30 years and tested well over a thousand real people using technology in dozens of software development projects.

I know which methods are needed to get the job done for your own design and development project. I know through experience all of the bells-and-whistles of "usability labs" that are indeed technically impressive, and I've built several myself...but I also know that much of the capability is usually not necessary.

  • I began working in human-testing labs in 1978, first as a lab assistant in a university "Perception and Cognition" lab for a future Nobel Prize winner This link opens another browser window. I finished my academic career with 8 years of user-testing and user-research in aviation and air-traffic control applications This link opens another browser window.

  • After I was awarded my Ph.D. in 1990 I joined the IBM Toronto Software Development Lab This link opens another browser window. For 8 years at IBM I usability-tested enterprise-level database and application development products, IBM's first GUI (Graphical User Interface) applications, first client-server applications, and even IBM's first designs for public Internet ventures (1994).

  • In 1998 I left IBM to lead the Usability Research and Analysis discipline at Bowne Internet/Immersant This link opens another browser window an international Internet consultancy. We made a significant investment building our own state-of-the-art usability labs in Toronto, Seattle, Ann Arbor and Parsippany. I usability tested Internet applications for several Fortune 500 clients in the financial services industry.

  • Since 2001 I have conducted usability testing on dozens of projects through my own Canadian corporation located in greater Toronto. I am contracted directly by project sponsors or subcontracted by design and development companies I have partnered with.

  • I know the difference between good and bad testing practices, and data versus opinion. In addition to my work as a professional in usability-testing, for 17 years I was also a professor (contract faculty) at York University, Toronto where I taught full-year psychology courses in Advanced Research Design and Statistics (See what some of my students thought on RateMyProfessors.com This link opens another browser window).

Usability Reviews

Most user-interfaces that were not designed by an expert user-interface designer will present obvious usability problems that your don't have to do formal usability testing to find. And any user-interface design will benefit from a usability review by a usability expert who can give a fresh perspective, at arm's length from the original design and development project.

In a usability review I work with your design specifications, existing prototypes, or your complete application, as if I were real end-users (i.e., your application's personas) using the application to complete key usage scenarios in the real world.

I apply usability review (inspection) This link opens another browser window methods (heuristic evaluation, cognitive walkthrough, pluralistic walkthrough) to determine whether and why end-users would become confused, annoyed, or frustrated at each step in the usage scenarios.

For many Internet sites I also evaluate the more elusive attribute of your design's ability to engage and build a personal relationship with its visitors by conveying value, building trust, by promoting attributes of your brand online and through integrated touch-points (e.g., customer support). The ultimate goal of the relationship is to encourage visitors to act in a way that supports your business goals.

For each identified issue, I describe the context-of-use at the time it is encountered, describe how users would likely become annoyed, frustrated, or confused, rate its impact on usability or user-experience, and recommend how the issue should be addressed.

Usability Reviews vs. Usability Testing

  • Reviews provide a quick and inexpensive way to identify obvious usability problems. Many problems that are identified in testing are the same issues that would have been found in a review.

  • Usability reviews are useful when early designs concepts are not in a form that would be useful for usability testing.

  • Even with the most "expert" of reviews, real end-users always encounter usability problems in testing that were not anticipated in design reviews. Most of these problems make sense in hindsight. This new-found knowledge makes the analyst and design team more informed about their application's end-users and their individual usability requirements.

  • When planning the test protocol for a usability test I always conduct an informal usability review of the application. Sometimes, early designs are flawed to the point that redesign should be undertaken before usability testing is conducted.

Quick usability checkup reviews

It's always discouraging to see how many usability problems creep into a user-interface during its development phase. Usually, problems arise because the UI design spec wasn't followed accurately, or the spec was incomplete and poor design decisions were made by a busy developer, or because of unforeseen technical challenges in the spec, or because the user-interface does not scale well to the actual operational environment (e.g., network load and performance, large or sparse data sets).

I can do quick usability reviews of components during development to ensure that your user-interface remains coherent as it is translated from specs and built into an operational application.

Pricing of a usability review

The time and cost of a design review depends on:

  • the scope of the review: a navigation framework, a small component, or the complete application;
  • the completeness and fidelity of your design spec, prototype, or QA/UAT-level or released application;
  • the complexity of your application and its operational environment, and my prior knowledge of both;
  • the number and complexity of usage scenarios to be covered in the review.

The work is billed on a time & materials or retainer basis. Typical costs range from $500 for ongoing reviews during design and development to $20,000 or more for a complete review of a large or complex application.

Last update: June 2012
© 2002-2012 Don Hameluck Usability Consulting Inc.

OVERVIEW : User ResearchUser Interface DesignUsability Testing905-868-8145

Me testing in one of the usability labs I designed for Bowne Internet / Immersant.

Toronto, 1999


I hold a Canadian and US Patent The orange link opens another browser tab. on UCDCam The orange link opens another browser tab. a software tool for remote usability testing.

I developed UCDCam when I was a usability analyst / User Centered Design specialist at IBM.

 

 

Download the PDF file.