There is no
guarantee that the usability and user-experience
of your new application will be perfect, or even acceptable, despite
your insightful user-research, the skills and opinions of your user-interface
designers, and ongoing design reviews by usability experts.
The performance and feelings of real
users in their real world are the measures of a usable application.
The earlier that you
properly test whether your application's interactive design is actually
usable, the easier, quicker, and cheaper
it is to fix problems your find. And by continuing to test your design
as it progresses, the more you can ensure that your have identified
and fixed potential problems before they are initially coded or actually
released in the application.
Testing- Why and When
- Often, my clients
need usability testing to identify
and fix problems in their current applications that are currently
"causing grief" for its end users.
- In major development
projects, I conduct usability tests of low- and mid-fidelity
user-interface prototypes to validate
high-level design concepts, discover and fix usability problems
and learn more about users and their tasks. The goal of this "formative
usability testing" is gathering proper feedback to improve
design before development is so far along that it becomes
too expensive to change.
- On occasion, I run usability tests specifically designed to measure
usability ("summative testing") on pre-release
(QA/UAT/Beta) applications to determine whether it has met stated
usability targets, and to identify and fix
any serious problems before release. User satisfaction, as
well as objective performance measures such as successful task completion,
time to complete the task, and number of errors, are the primary
measures. This testing can involve competitive
benchmarking as well.
Overview of a typical testing project
Below are the steps I complete to enable and conduct
a usability test. A typical project can be completed in six-weeks
after you first contact me, although quick tests can be completed
in as little as a week or two.
1. Plan the test
||Establish the purpose
of the test and develop a strategy to deliver quality results
given available budget, staff, and time.
- I explore your application or prototype to understand how it works
and to identify potential usability issues to watch for in testing.
I also step through the application / prototype while working through
a set of key
that have been identified by your application's business stakeholders
and from prior user research. If a prototype is being tested, I
assess its fidelity to an actual interactive application and determine
how to gain reasonable feedback from user testing.
||Determine which types of users
to test, how many of each type to test, and how to get them.
- The type
of users and the number of each type
are determined from your project's personas
(if they were written), or by consulting your application's marketing,
support, or business functions.
- Your type of application will determine
the most effective strategy to recruit
suitable end-users for the usability
test. For public Internet sites I usually write a detailed recruiting
screener and engage a professional
recruiting agency to locate and schedule
qualified users. Participants are usually paid for their participation.
- Where to run the test
also depends on your particular project requirements. The only real
requirements are that the location provides easy access to test-participants,
does not present distractions, and will allow the test-application
to be run. An office (free), a hotel meeting room (~$300/day), a
focus group facility (~$1,500/day), or a usability lab (~$1,500)
are all options. Testing can occur in multiple locations if required.
With Internet screen-sharing
technology (GoToMeeting, WebEx), a live usability test can occur
with the user sitting in one location and the usability analyst
observing thousands of miles away. Reasonable usability tests can
even be conducted with automated tools (such as Keynote
without requiring the usability analyst to attend each session live.
to conduct the usability test sessions.
- Develop a testing protocol to run
each usability test. The protocol includes initial interview questions,
instructions to the user, a set of key user-scenarios
that each test-user will attempt to complete with the application
/ prototype, notes about design issues for me to watch for during
the test, and wrap-up interview questions and satisfaction /user-experience
- Typically, 6
to 10 key user-scenarios are tested in each usability test. Six
users per target market segment are tested, so a typical project
involves a total of 12 to 24 test participants. Group-based
test sessions are also possible if appropriate for your project
2. Enable testing
- Reserve appropriate testing facilities for scheduled dates.
- Determine characteristics of target market segments. If necessary,
write the participant recruiting screener. Execute participant recruiting
- Dry-run testing protocol with suitable participants; revise as
- Set-up test environment at testing facility. Your
project's executive sponsors, business stakeholders, and development
team are encouraged to observe test sessions
from the facilities observation suite or by watching the user's
screen and face using my remote viewing technology.
3. Conduct test sessions
- Actual user-testing is conducted
over a one or two week period.
One user is tested at a time in a 1 to 2 hour test session.
The test sessions run any time between 10:00am and 10:00pm.
- In most projects, the test
participant is encouraged to think-aloud
as they use the application / prototype to complete each user-scenario.
I watch and listen closely while the user works. I try to understand
their moment-by-moment motivations and decision processes and note
when (and possibly why) they become confused and wander off
the path from completing the task or can't proceed at all. If they
haven't already complained to me, I ask for their feedback, but
only when appropriate.
To expedite analysis, I make time-stamped notes during each usability
- At the end of each session, the participant
rates their experience using the application
on a variety of key usability and user-experience attributes, and
I interview them to ensure that the feedback I observed is accurate.
- I record all audio
and user's screen-activity during the test session for detailed
analysis. I provide your with all recordings, and can also produce
a "Highlights" DVD that isolates particular issues.
4. Review test session data to identify issues
- When all usability tests are complete I
consider all the testing data and review recordings in detail as
necessary to identify all user-problems, rate their impact, diagnose
the underlying design problem, and determine how to fix it.
5. Produce Deliverables
are tailored for your own project's objectives: to identify and
fix problems in a current release, plan for future releases, or
convince executives to fund redesign projects.
- List and prioritize high-level issues
that compromise usability and user-experience. Provide recommendations
for each issue. Typically 10 to 15 high-level issues are identified.
- Document detailed
usability issues, rate their impact, and provide recommendations
for each. Typically, hundreds of detailed issues are identified.
Issues can be documented for easy integration to your bug-tracking
system, if appropriate.
- Provide DVD-video
recordings of all test session activities. If necessary,
produce a DVD of session highlights.
- Present findings
to project team, product management, and executive stakeholders.
6. Take action on results
||If my role continues beyond
the test itself, I work with your team on design solutions to
the usability problems identified in testing.
- Turning a large set of test results
into a more usable application requires an intimate understanding
of the causes and
of each of the identified usability problems. But there may be several
ways to address any one usability problem. And resources to address
the problems are finite: resource should be assigned to make design
changes that will bring the biggest improvements.
To accomplish this, each problem needs to be prioritized
not only in terms of its user-impact, but each potential design-solution
must be evaluated in terms of technical feasibility, its time and
difficulty (cost) to implement, and its compatibility to the future
- It may be costly to fully address a particular problem, but
a quick fix could lessen the severity of the problem until it
can be properly addressed in a future release.
- It could be that fixing several low/medium-impact (but easy-to-fix)
problems encountered in a single user-scenario, such as completing
a form, will add-up to significantly improving its overall usability.
- When every one of the usability
problems identified in testing are considered in terms of their
user-impact and cost to address, then your project team can focus
in a way that will most benefit the
end-user and your business goals given available resources.
Keeping me involved will ensure that your application's end-users
are properly represented in this process.
- I can also remain involved in your project to fully design
the solution for identified usability problems and/or
proposed design solutions to ensure that they solve the problem.
Pricing of a usability test
I have conducted so many user-testing
projects that I can provide a fixed-price
quote for your project requirements.
Quick tests of a simple application may be as little as $5,000
to $7,500. Typical tests that involve
a moderately complex application, rigorous end-user definition and
recruiting, and 12-24 user-participants: they generally range from
$15,000 to $45,000.
- Test-application -
Pricing depends on the operational complexity of your application,
the type and completeness of the design artifacts available for
testing, the number of key-usage scenarios selected for testing,
the ability of these artifacts to support the selected test scenarios,
and the ease of creating reasonable test scenarios.
Design artifacts for testing range from early static prototypes
to full operational applications. A good test requires that the
test-scenarios feel realistic to test-participants or their confusion
with the testing situation itself will overshadow their experience
of the "application" of interest. A static prototype may
need additional pieces of the task-flow represented. Or an operational
application may need a set of non-production data to provide a realistic
- Test-participant recruiting - Pricing
also depends on the number of target markets
to be recruited and tested (typically, 6 test-participants per each
identified target market, user-role, or persona),
and the complexity and expense of the necessary recruiting effort.
I engage professional recruiting agencies
- Deliverables -
The style and scope of the test-deliverables can vary considerably,
which also affects cost. I optimize test-deliverables to address
the primary purpose of your testing: identify problems for development
to fix in a maintenance-release, help executives and product management
plan for a major redesign, explore an early prototype (formative
testing) in a design project, or validate a design prior to development
(summative testing). In addition, nimble and lean startup companies
rely on team working-sessions more than detailed documentation,
whereas projects at large institutions must adhere to corporate
expenses included in the fixed-price cost:
- Recruiting costs using a professional recruiting
agency are typically $75 per attendee.
- Incentive paid to each end-user to participate
typically ranges from $0 to $50 per hour but can be more depending
on characteristics of your target markets (e.g., college students
- Test facility costs range from $0 (your
office) to $500 (hotel conference room) or up to $1500 per day (usability
lab or focus group facility).
- Travel to remote locations.
background in user testing
I've lived in user-testing labs for nearly 30
years and tested well over a thousand
real people using technology in dozens of software development
I know which methods are needed to get the job done
for your own design and development project. I know through experience
all of the bells-and-whistles of "usability
labs" that are indeed technically impressive, and I've
built several myself...but I also know that much of the capability
is usually not necessary.
- I began working in human-testing labs in
1978, first as a lab assistant in a university "Perception
and Cognition" lab for a future Nobel
Prize winner .
I finished my academic career with 8 years of user-testing and user-research
and air-traffic control applications .
- After I was awarded my Ph.D.
in 1990 I joined the IBM
Toronto Software Development Lab .
For 8 years at IBM I usability-tested
enterprise-level database and application development products,
IBM's first GUI (Graphical User Interface) applications, first client-server
applications, and even IBM's first designs for public Internet ventures
- In 1998 I left IBM to lead the Usability
Research and Analysis discipline at Bowne
an international Internet consultancy. We made a significant investment
building our own state-of-the-art usability
labs in Toronto, Seattle, Ann Arbor and Parsippany. I usability
tested Internet applications for several Fortune
500 clients in the financial services industry.
- Since 2001 I have conducted usability
testing on dozens
of projects through my own Canadian corporation located
in greater Toronto. I am contracted directly by project sponsors
or subcontracted by design and development companies I have partnered
- I know the difference between good and bad
testing practices, and data versus opinion. In addition to my work
as a professional in usability-testing,
for 17 years I was also a professor (contract faculty) at York University,
Toronto where I taught full-year psychology courses in Advanced
Research Design and Statistics
(See what some of my students thought on RateMyProfessors.com
Most user-interfaces that were not designed by an
expert user-interface designer will present obvious
usability problems that your don't have to do formal usability
testing to find. And any user-interface design will benefit from a
usability review by a usability expert who can give a fresh perspective,
at arm's length from the original design and development project.
In a usability review I work with your design specifications,
existing prototypes, or your complete application, as if I were real
end-users (i.e., your application's personas)
using the application to complete key usage scenarios in the real
I apply usability
methods (heuristic evaluation, cognitive walkthrough, pluralistic
walkthrough) to determine whether and why
end-users would become confused, annoyed, or
frustrated at each step in the usage scenarios.
For many Internet sites I also evaluate the more
elusive attribute of your design's ability to engage
and build a personal relationship with its visitors by conveying
value, building trust, by promoting attributes of your brand online
and through integrated touch-points (e.g., customer support). The
ultimate goal of the relationship is to encourage visitors to act
in a way that supports your business goals.
For each identified issue, I describe the context-of-use
at the time it is encountered, describe how users would likely become
annoyed, frustrated, or confused, rate its impact on usability or
user-experience, and recommend how the issue should be addressed.
Reviews vs. Usability Testing
- Reviews provide a quick and inexpensive
way to identify obvious usability problems. Many problems that are
identified in testing are the same issues that would have been found
in a review.
- Usability reviews are useful when early designs concepts are not
in a form that would be useful for usability testing.
- Even with the most "expert" of reviews, real end-users
always encounter usability problems in testing that were not anticipated
in design reviews. Most of these problems make sense in hindsight.
This new-found knowledge makes the analyst and design team more
informed about their application's end-users and their individual
- When planning the test protocol for a usability test I always
conduct an informal usability review of the application. Sometimes,
early designs are flawed to the point that redesign should be undertaken
before usability testing is conducted.
It's always discouraging to see how many usability
problems creep into a user-interface during its development phase.
Usually, problems arise because the UI design spec wasn't followed
accurately, or the spec was incomplete and poor design decisions
were made by a busy developer, or because of unforeseen technical
challenges in the spec, or because the user-interface does not scale
well to the actual operational environment (e.g., network load and
performance, large or sparse data sets).
I can do quick usability reviews of components
during development to ensure that your user-interface remains coherent
as it is translated from specs and built into an operational application.
of a usability review
The time and cost of a design review depends on:
- the scope of the review: a navigation framework, a small component,
or the complete application;
- the completeness and fidelity of your design spec, prototype,
or QA/UAT-level or released application;
- the complexity of your application and its operational environment,
and my prior knowledge of both;
- the number and complexity of usage scenarios to be covered in
The work is billed on a time & materials
or retainer basis. Typical costs range from $500
for ongoing reviews during design and development to $20,000
or more for a complete review of a large or complex application.