When computer science moved from the realm of scientists programming their own number-crunching systems to everyday people's offices and homes, the bottleneck of "usability" switched from the limitations for the hardware and programming language to how these systems support the goals of (often) untrained users. Today, usability is multidimensional, encompassing learnability, effectiveness, flexibility, and userís attitudes towards the system (Shackel, 1991). Unfortunately, our ability to predict the performance and attitudes of users has not developed as quickly as our ability to produce massive quantities of consumer software, inundating our environment with barely learnable software that never does exactly what we want and leaves us frustrated and unproductive. (For example, I would have liked to put a footnote in here describing the not-quite-WYSIWYG html-editor I am using to create this position paper, which canít quite handle within-document links, but it can't do footnotes either!) A credible case has been made that insufficient evaluation of computer systems with respect to real-world usefulness and usability is a major source of what has been called the "productivity puzzle" (Landauer, 1995).

Several techniques for understanding users' needs and predicting the usability of software systems have developed in the last 15 years. Participatory design techniques (Schuler & Namioka , 1993) purport to identify users' needs. User testing (where typical users are brought into a usability lab, asked to think aloud while performing some typical tasks on a prototype, and questioned about their experience of the software) is probably the most widely-used technique in industry today (Nielsen & Mack, 1994). Heuristic Evaluation (Nielsen, 1994), Cognitive Walkthrough (Wharton, et. al. 1994), GOMS (Card, Moran & Newell, 1983), and other analytic techniques for predicting usability problems have been proposed that do not require a running prototype or actual users. However, there are many open questions about all these techniques. For instance,

• Does the technique accurately predict needs or usability problems that people will encounter in the real world?
• Can the technique make design suggestions as well as identifying needs or problems? And does implementing these suggestions actually fix the problem or satisfy the need?
• When in the development process can the technique be used?
• How effectively does the technique feed into the development process. For instance, can the evidence from the technique in favor of design changes outweigh the costs of changes and the "not-invented-here" syndrome.
• How can techniques be used together to best advantage?
• What are the true costs of learning and applying a technique?
• How can we best train development teams to use different techniques (e.g., become a standard part of undergraduate CS training, continuing education courses, etc.)?
• What modifications to these techniques are necessary for them to be effective? What new techniques might be needed?

Unfortunately, research in HCI has not yet put sufficient monetary, chronological or intellectual resources toward answering these questions. As a field, we need to

• Insist on, and commit resources to, validating the predictive power of techniques. That is, we need to be confident that the techniques we use actually predict real-world behavior. Validation requires controlled empirical studies with many analysts (using the technique), analyzing many different systems, of many different types, intended for many different types of users. Their predictions must be compared to empirically observed problems in the real world (not just in the laboratory with college freshman). (An example of a study that satisfies a few of these requirements appears in Gray, John & Atwood, 1993. )
• Have studies that follow usability problems all the way through the development process. That is, from the technique that identifies it, to the developers' decisions to change the product or not, through the implementation of the change, through re-evaluation with user-tests or other technique, to real-world use of the product.
• Study the training of usability techniques. Longitudinally follow students in HCI classes or professional tutorials. Are they using the techniques as intended? Is the technique as effective in the hands of a trainee as it is in the hands of its developer? How much background in CS, psychology, sociology, design, etc., is sufficient to effectively use a technique?
• We must insist on evaluating the usefulness of software as well as usability. Rigorously assess, with controlled studies, whether the addition of new technology has actually improved the productivity or quality of the real-world domain it was intended to improve. The history of introducing computer systems has been quite dismal when such evaluations have been done (Landauer, 1995). Therefore, we must insist on more such evaluations and be ready to hear (and act on) the bad news that our beloved computers are not improving the situation if indeed that is what the data report.
• Finally, judging from a small amount of initial data already collected on the effectiveness of usability evaluation techniques (e.g., Jeffries, et. al, 1991; John & Mashyna, in press), we will probably require improvement of our current techniques and development of new ones. Especially necessary are techniques that can predict severity of problems, make design suggestions and that address the issue of usefulness as well as usability.

These suggestions require that HCI move to a "bigger science" than has previously framed the field. These questions cannot be answered by academics or industry alone. We need to find ways to build partnerships between academics, government and industry to fund long-term process studies like those suggested above. We need to find ways to routinely collect and share usability data, both in the lab and in the field, without compromising a product's place in the market. We need to borrow the well-defined techniques of other disciplines to assess usefulness and process (e.g., medical research has well-defined techniques for large-scale, real-world assessment of process, as does educational research). In short, when considering systems used by real people, and how we design them, we need to move toward a more empirical science than Computer Science has often embraced.

References

Card, S. K., Moran, T. P., and Newell, A. (1983) The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, NJ.

Gray, W. D., John, B. E., & Atwood, M. E. (1993) Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world task performance. Human-Computer Interaction, 8, pp. 237-309.

Jeffries, R., Miller, J. R., Wharton, C., and Uyeda, K. M., (1991) User interface evaluation in the real world: A comparison of four techniques, In Proceedings of CHI, 1991, New Orleans, LA, ACM, NY.

John, B. E., & Mashyna, M. M. (in press) Evaluating a Multimedia Authoring Tool with Cognitive Walkthrough and Think-Aloud User Studies. Journal of the American Society of Information Systems

Landauer, T. K. (1995). The trouble with computers: Usefulness, usability, and productivity. Cambridge MA: MIT Press.

Nielsen, J. (1994) Heuristic evaluation. In J. Nielsen and R. L. Mack (eds.) Usability Inspection Methods. New York: John Wiley.

Nielsen, J & Mack, R. L. (1994). Usability Inspection Methods. New York: John Wiley.

Shackel, B. (1991). Usability -- Context, framework, definition, design and evaluation. In B. Shackel and S. Richardson (eds.) Human Factors for Informatics Usability. Cambridge UK: Cambridge University Press.

Schuler, D., & Namioka, A. (Eds.). (1993). Participatory design: Principles and practices. Hillsdale, NJ: Lawrence Erlbaum Associates.

Wharton, C., Rieman, J., Lewis, C., & Polson, P. (1994) The Cognitive Walkthrough Method: A practitionerís guide. In J. Nielsen and R. L. Mack (eds.) Usability Inspection Methods. New York: John Wiley.