What is “Usability Testing”?

On a broad scale, usability testing is exactly what it sounds like – the act of testing a product or service for usability. But what does “usability” mean?

In IMD325 (User Centered Design – Web Development), we discussed the planning stages of creating web sites. In this class we will take over what happens on a quality assurance level during and after the project has been deployed. We will also look particularly at encapsulated objects – things like images and Flash animation – and how they also have a quality assurance requirement built into them.

Your clients or customers of your web site or products want service – they demand that what you produce works, that it works properly, and that it delivers the information and interactivity that makes THEM comfortable. And there are innumerable places that this can go awry. Obviously one aspect is the design itself – is the interface intuitive, do all the links work. But there are many, many other things to consider that you might not realize on the surface – is your server capable of taking the load, does the user have the plug-ins or applications needed to run the site, does all of the form input go to the right place, is my code within acceptable standards – and more.

During this quarter we will be exploring how the term “usability testing” can have many meanings, but that all of them apply to the idea of “quality of service”. In any aspect of design, particularly in web site and interactive media production, we as designers and developers must always be cognizant of how the user or consumer experiences our outcome. We must also recognize that the outcome of our design might not always have the same effect that we intended it to.

User Centered Design – Web Design Review

For students who have taken IMD325, this section is a quick review of things you’ve learned in that class but you may want to read this as a refresher.

In developing web sites, designers often overlook the fundamentals of what creates a good design. What happens more often than not is that we tend to attack the actual, physical design of the product without taking into account the basic aspects of why we are building it in the first place.

“User-centered design” is defined (by Jesse James Garrett) as “the practice of creating engaging, efficient user experiences.” To do this, real-world project managers and stakeholders create project plans, detailed documents that define every aspect of the site in a pattern that builds on top of itself so that the developer has no question in his or her mind what the final product should look like or act like.

In creating usable design, we use a five-tiered approach to developing these plans. These five “planes” are:

Strategy where we define:

• the vision of the site,
• the mission statement,
• the historical background of need for the site,
• the target audience, success metrics

Scope where we define:

• the scope of the work to be done,
• the feature set,
• the potential risks

Structure where we define:

• the architectural diagram (site map) of the site,
• the basic content needs for each page

Skeleton where we define:

• the structural layout of each page (the template)

Surface where we define:

• the style guide (a definitive guide to the overall ambience, such as color palettes and typography),
• the actual physical design

Additionally, we discuss the ramifications of “duality” with respect to each plane. Duality in the context of user-centered design proposes that there are two views to each plane – the view of the design as a software interface and the view of the design as a hypertext information source. Super-imposing the duality context on top of the five planes gives us (a visual representation that looks something like) the diagram shown on the right.

These five planes take you from the initial concept phase to the start of the development phase.

Quality Assurance and Maintenance

Where this course picks up is what happens after that – during and after the actual coding and graphic development phase, and after deployment. To our five original planes, we add two additional planes:

Quality Assurance is where we define whether or not the site meets the goals of the original plan:

• Does the site accomplish the mission?
• Does the site cater to the needs of our target audience?
• Does the site have built in means of measuring success?
• Does the site have all of the pages defined in the architectural diagram and the navigational elements to support it?
• Does the site structurally look like the templates?
• Does the site look like the design compositions?
• Is there error-handling in appropriate places?
• …in short, does it do, look like, and contain what it’s supposed to!?!?

Maintenance is where we define and document the modifications made to the site with respect to either quality assurance deficits or user complaints:

• Additional features, changes to existing features,
• Changes to existing layout (both structurally and design oriented),
• Changes to the error-handling,
• Changes to the dynamic data sources,
• Changes to the hosting environment,
• Other customer service needs

As you might guess, we still need to define the new planes in terms of duality because every change made to one side may affect the other, or affect the construct as a whole!

The Testing Matrix

During the quarter, we will be looking at a battery of different tests that can be performed to help you create quality assurance. There is no industry “standard” for testing a site (or any other type of media), so we will be calling on a number of methods to help us determine what the true meaning of “usability” is, and discussing the ramifications of each.

Frankly and sadly, many sites rarely perform even the most basic of these tests. And certainly there are occasions when some of these tests are not even warranted, either because of costs or just basic non-necessity. But as a project manager, it is imperative that you know:

• what types of tests exists,
• how each are performed,
• how to create viable testing documentation,
• what types of results are and should be expected from each test,
• how to evaluate plans of action for implementing changes,
• how to implement changes when deficits occur,
• and how to avoid future problems through proper project planning

Each week we will be completing a section of the document labeled “Testing Matrix” (each one will be notated with a version number so you can see the progression and keep track of what we’ve done). For this week, download “Testing Matrix w.1” (bottom of article).

Qualitative versus Quantitative

As we approach the very large topic of Usability Testing, we have to bear in mind that the results of our tests can be either qualitative or quantitative.

Quantitative results are those which can be measured directly – results that have specific numbers driven by calculations. Some of these results might include usage statistics (unique users, total users, or better yet, total failed responses) and load test results (user capacity, CPU capacity) for example.

Qualitative results are those which can be measured by implication or subjectivity – results which generally are based on perceptions. In general, this can be problematic so when developing tests it is normal to try and quantify qualitative results by using large sets of sampling.

Approach The Task

“Approaching the Task” is the means by which we say “how am I going to wrap my arms around this?” From one side of the picture, you as a tester must put yourself in the seat of the consumer – to be able to perceive how the user sees, feels and interacts with your product. If you are on a development team, this can often be difficult because implicitly you know enough about the INTENDED result that your perceptions are biased.

On the other side of the picture, you as a tester need to have so much carnal knowledge of the SCOPE of the product that you explicitly know when something is not right – you must be the owner of the product. This means that as a quality assurance person, you know and understand exactly what every part of the project plan is and how to measure if something does not meet the plan.

Thus, the means of your task involves:

1. Seeing the picture from both perspectives and being able to think in extremes
2. Knowing enough about the product to select appropriate tests
3. Knowing enough about your target audience to be able to think like that group
4. Knowing what TYPES of tests are needed to avoid costly testing processes

Method Focus: User Acceptance Tests (part 1)

Today we are going to focus on the first battery of user tests – user acceptance tests.  User Acceptance Tests are the means by which we obtain information about how our targeted users perceive the site.  Perception is a subjective issue so typically user acceptance tests require 1) a large sampling (large quantity of respondents) and 2) quantification of the results.  Next week we will discuss more in-depth the issue of quantifying the results.  For now, let’s focus on what the tests specifically accomplish.

Note that many of these tests may have been originally designed for purposes other than web site testing, such as software or systems testing, but the applicability of the testing crosses over into web sites as application interfaces and consequently have relevance.  For purposes of explanation, I will use the term ’site’ to refer to the web site, even if the test was developed outside of the Web environment.

Questionnaire for User Interface Satisfaction (QUIS)
The QUIS test was developed by at the Human Computer Interaction Lab at the University of Maryland in 1988.  There are four primary goals of the QUIS test:

1. guide in the design or redesign of systems,
2. give managers a tool for assessing potential areas of system improvement,
3. provide researchers with a validated instrument for conducting comparative evaluations, and
4. serve as a test instrument in usability labs

The QUIS tells us primarily how easily the user learns to use the site through consistency, task arrangement, and error handling.  Questions are rated on a scale of 0-9 with 9 generally meaning a ‘good’ score.

More information on QUIS can be obtained here.

Perceived Usefulness and Ease of Use
The PUEU was developed at the University of Michigan in 1989.  The PUEU attempts to predict user acceptance of systems and how the system would positively or negatively affect job performance overall.  This test is particularly useful for evaluating CMS systems because it details the ability of the site to improve or speed working capacity.

An online version of the PUEU can be found here.

Nielsen’s Attributes of Usability
Respected usability guru Jakob Nielsen developed this test, the NAU,  in 1993.  The test is extremely short but poignant.  It attempts to provide overall final impressions of a site, the quality, based on 5 subjective areas.  The point of the test is to measure if the site was impressionable enough to make the user return – was the experience and/or utility of it sufficient for the user to want to use it again.

Nielsen’s article describing the test.
An online version of the NAU can be found here.

Nielsen’s Heuristic Evaluation
Nielsen also developed this test in 1993.  Like his NAU test, the NHE is short and succinct.  Essentially it is a “systematic inspection of a user interface design for usability”.  The NHE purportedly is more cost-effective to run because it is heuristically (heuristic: a replicable method or system or approach to directing the learning process) based, and therefore requires smaller samples.

Nielsen’s article describing the test.

---

• Note: As I mentioned last quarter, this class uses Microsoft Excel extensively. If you are (still) not familiar with the basic interface and use of Excel, you will need to get started quickly. We will be doing some in-class tutorial work with it next week.
• Downloads: Please download these materials and store them digitally – save a tree today!
  IMD335 Syllabus for Summer 2007
  Testing Matrix Week 1
  Test: Questionnaire for User Interface Satisfaction (QUIS)
  Test: Perceived Usefulness and Ease of Use
  Test: Nielsen’s Attributes of Usability
  Test: Nielsen’s Heuristic Evaluation
• Reading/Review: Please review all of the download materials. Next week we will be introducing User Acceptance Testing methodologies but because of the quantity of them, we are starting early.