Commonsense Guidelines for Developing Multimodal User Interfaces

James A. Larson

February 24, 2002

When fonts were first introduced, many messages looked like ransom notes from kidnappers. When color was introduced, many reports looked like they barely survived an explosion in a paint factory. To avoid these annoying user interfaces, developers have adopted widely-accepted guidelines for using fonts and colors.

With the introductions of multiple modes of input—voice, pen, and keys—developers may develop applications reminiscent of Times Square on New Years Eve, The Daytona Raceway during a high speed race, and the Fourth of July fireworks all combined into a loud, confusing, and annoying user interface that result in low user performance and high user discontent. This document attempts to enumerate a collection of commonsense guidelines for developing high performance and high preference multimodal user interfaces.

This set of guidelines present a first attempt to enumerate commonsense heuristics for developing multimodal user interfaces. Each guideline is rated with one to four stars to indicate its importance. Guidelines with four stars should always be followed. Guidelines with one star may be ignored if they conflict with other guidelines for developing multimodal user interfaces. We encourage developers to apply these guidelines when developing multimodal user interfaces and avoid many of the potential usability problems of using modes incorrectly.

Many of these guidelines are just commonsense. The reader might think that no one would ever develop user interfaces that violate these guidelines. But developers have violated commonsense before and will likely do so again. Use these guidelines as a check list when you design every multimodal interface. These guidelines should help you to construct a multimodal user interface that improves user performance and user preference scores, so your application can be used easily and effectively by its intended users.

We have collected suggestions, techniques, and principles from many diverse disciplines to generate the following guidelines for developing multimodal user interfaces.

Task oriented guidelines

Task-oriented guidelines suggest which tasks lend themselves best to data entry using various modes of entry.

New mobile devise will enable users to enter data by pressing the keys on a small keypad, write using a stylus, and speak into a microphone. These input modes can be used to perform four basic manipulation tasks:

• Select objects (e.g., menu options)
• Enter text
• Enter symbols (e.g., part of mathematical equations)
• Enter sketches or illustrations

There are other basic tasks, but the four above are performed most frequently in common applications using handheld computers.

Table 1 summarizes how users perform these tasks using the following popular input modes:

• Voice—The user speaks into a microphone.
• Pen—The user manipulates a pen to write, draw, or point.
• Keys—The user manipulates a keyboard or keypad by pressing keys.

Table 1: Performing the four basic tasks using the three popular input modes, ranked from easiest (1) to most difficult (3)

Select objects. Object selection is easy with a pen—just point to or circle the desired object. When using voice, just say the name of the desired object, as long as the object has a name. With a keyboard, press keys to position the cursor on the desired object and press the select key.

Enter text. Each of the three modes can be used for text entry—the user speaks words into a microphone, handwrites the words using a pen, or presses keys on a keypad to spell the words. Most users can speak and write easily. However, some training and practice may be necessary to use a keyboard efficiently.

Enter symbols. Entering mathematical equations, special characters, and signatures is easy with a pen, awkward and time-consuming with a mouse, and most difficult with speech.

Enter sketches or illustrations. Drawing simple illustrations and maps is easy with a pen, awkward with a mouse, and nearly impossible with speech. When speaking, users must verbally describe the illustration or map.

Each input mode has its strengths and weaknesses. Voice is good for describing attributes. The pen is good for pointing and sketching. Keys are good for entering text, numbers, and symbols. A useful and efficient multimodal system uses the appropriate mode for each entry.

*** Guideline: Use the easiest available mode for each task.

Guideline examples

• To select an icon, use a pen or stylus to point to the icon.
• To enter text, use a keypad.
• To enter the symbols of a mathematical equation, use pen
• To draw a map, use a pen

Different physical devices exhibit different usability characteristics. The device’s size, shape, and weight effect how it may be used. Most important, placement of microphone and speaker, size of keys in a keypad, and size of display and writing surface all affect the ease with which a user can enter information by speaking, writing or pressing keys.

Guidelines for using physical devices

Physical device characteristics may restrict the modes used to enter data.

Table 2: Device usability issues for the three popular modes for information entry

Required use of hands. No hands are needed to speak and listen to a voice user interface. A pen requires one hand to hold the pen. A 12-key keypad requires one hand to enter data, and a QWERTY keypad requires two hands to enter data efficiently. By their nature, handheld devices may require a hand to hold the device. Some users become skilled at holding a small QWERTY keyboard with both hands and using their thumbs to type. So in reality, voice may require one hand and pen and a 12 key keypad requires may require two hands.

**** Guideline: Use speech when the user’s hands are busy.

Guideline examples:

• If the user is driving a car, use speech to dial a cell phone.
• If the user is repairing a machine, use speech to ask for the next repair instruction.
• If the user is preparing a meal, use speech to ask for the next recipe instruction.

Required use of eyes. Usually, users must look at what they are writing with a pen or typing on a keypad. However, the user’s eyes are free to observe his or her environment while speaking

**** Guideline: Use speech when the user’s eyes busy.

Guideline examples:

• If the user is driving a car, use speech to manipulate a radio.
• If a guard is monitoring a TV monitor, use speech or hand controls to manipulate the camera.
• If a scientist is looking into a microscope, use speech to dictate his or her observations.

Portable. Speech and pen devices are very portable. Users may use them while sitting, standing, walking, and sometimes while running. Traditionally, keyboard devices are used only while sitting. Keypads requiring only one hand, like those frequently found on handheld devices and telephones, can be used while sitting or standing.

**** Guideline: Use speech if the user is walking

Guideline examples:

• While walking on the streets of New York, use speech to ask directions to the nearest subway station. (Both voice and a map may be used to present directions to the user).
• While shopping in a department store, use speech to ask for the location of a specific item to purchase.

*** Guideline: Use voice to provide commentary or help . Voice is more immediate and does not obscure screen contents.

Guideline examples:

• While the user is working with a spread sheet application, use speech to notify the user that he or she has mail or that his or her document has jammed the printer.
• While the user is driving a car, use voice to describe visible from the car’s current location.
• If the user asks for help to change the font size of a text string give verbal instructions to guide the user to perform the task.

Environmental Guidelines

Environmental guidelines suggest that some modes can not be effectively used in certain environmental conditions such as low lighting, high noise, extreme cold, and other environmental conditions.

Environmental concerns. Speech recognition systems often make mistakes if the user speaks in a noisy environment.

**** Guideline: Use a pen or keys in a noisy environment.

Guideline examples:

• Use pen or keys to enter a telephone number in a noisy airport.
• Use pen or keys to enter data on a noise shop floor.

Pen and keyboard devices are difficult if the user must wear thick gloves, such as in a cold environment or when protecting their hands from rough objects.

**** Guideline: Use voice if the user must wear heavy gloves.

Guideline example:

• If the user works in cold meat locker, works on a construction site and handles rough material, or works with dangerous chemicals and must wear gloves, then use voice to enter data.

Social Guidelines

Social customs among people suggest guidelines for user interfaces between people and the computer.

Privacy. Speech presented with a speaker is not private. Others in close proximity can hear both the user and the computer. The keyboard/mouse and pen provide greater privacy.

**** Guideline: Make pen or keys available for use if privacy is desired.

Guideline examples:

• Use keys to enter personal identification numbers.
• When using an automatic bank teller, always use a keypad to enter the account number.
• When using a weight management application, enable the user to enter his or her weight using a pen or keypad.

Acceptable in meetings. Pen devices are acceptable in meetings—they replace a pen and pad of paper for taking notes. Keyboards and keypads are becoming acceptable with the widespread use of laptops. Usually, devices that speak or are spoken to are not acceptable in meetings without the use of earphones, and in some cases, earphones may infer that the user is not interested in the current discussion taking place in the meeting.

**** Guideline: Use pen or keys (with the key clicking sound turned off) if the device will be used during a business meeting.

Conversational Guidelines

Principles of conversational discourse suggest guidelines for the nature, content, and format of information exchanged between two humans may be applied to information exchanged between a human and a computer.

Reflexive principle. The reflexive principle states that people tend to respond in the same manner that they are prompted. For example, if users are prompted with long rambling prompts, they will likely respond with long rambling responses.

* Guideline: Users tend to use the same mode that was used to prompt them.

Guideline examples:

• When spoken to, users will use their voices to respond
• When presented with a drawing, users will respond with another drawing.
• When presented with text, users will type their responses .

Reliability guidelines

The backup mode principle. If an input mode fails, the user should be able to use a different mode as backup.

**** Guideline: support at least two input modes so each input mode can be used when the other can not be used.

Guideline examples :

• Enable the user to use a keypad when speaking or using a pen in the event that the speech recognition or handwriting recognition engine fails .
• Enable the user to speak or type if the user looses the pen or input stylus .
• Enable the user to speak if rain or snow will damage a keypad

Users need feedback to determine whether the computer is processing input data, is waiting for input, or is malfunctioning.

**** Guideline: Always indicate whether the computer is busy or idle.

Example Guideline:

• Here are some suggestions for indicating if the computer is idle or busy.

Consistency Guidelines

If prompts are worded inconsistently, then users must pause to decode each wording format. Users must spend additional time and mental effort to respond to questions that are worded differently from one another. When prompts are consistently worded, users are able to supply the answers quickly.

*** Guideline: Phrase all prompts consistently

Guideline example:

• To be consistent and encourage experienced users to barge-in, consider using the following general voice prompt format:

1. Speak the name ofthe menu or form item. The menu name serves as a landmark. A landmark is a speech or non-speech cue that marks a specific location within the dialog structure. By providing a name such as “main menu” or “thermostat,” callers can jump to this menu by speaking the menu name or callers can return to this menu when they get confused or lost. Also, repeating the menu name to the caller confirms that the caller has reached the correct menu. However, if the name is contained within the question and is not needed as a landmark, then skip speaking the name.

2. Ask a question. Often, this can be achieved with two or three words. This should be enough to remind experienced callers to respond without listening to the enumerated options. Novice callers will listen to the enumerated options before speaking their selection.

3. Enumerate options. If there are a small number of valid responses, then list the options so novice callers can hear and select their desired options. However, if the user is likely to know the set of valid responses, then skip the operation.

Average callers can barge-in after they hear the question, while novice callers will respond after they hear the entire menu option list.

• Use the same terms and in all prompts, be they text prompts, voice prompts, or multimedia prompts.

Switching modes can be jarring and sometimes surprising. For example, a user who has just answered three verbal questions will be surprised if a textual question suddenly pops up.

*** Guideline: Do not switch modes without good reason .

Guideline examples:

• If the user is repeatedly experience errors when using voice or handwriting recognition, consider switching to a different a text mode. Text mode often avoids the recognition errors occurring because of poor handwriting or heavily-accented speakers.
• Switch from audio output to text output if the result of a verbal query is and the user is likely to become anxious listening to the result.
• Switch from audio output to graphical output if the result can be structured as a table, graphic, or other illustration .

Guidelines for creating emotions

TV and movie directors set the mood with sets, lighting, and background music. Screen layout, colors, and background music can also create moods in multimodal user interfaces.

** Guideline: Use audio and graphics design to set the mood and convey emotion in games and entertainment applications . However, moods and emotion may not be appropriate in productivity applications.

Example guidelines:

• Use background music to introduce new scenes with the appropriate mood. For example, discordant music indicates trouble lies ahead, cheerful music signals a scene filled with goodwill, and a dirge indicates a depressing scene.

• Use background music to “set the stage.” For example, classical music for an art museum, calliope music for a circus or fun fair, or bagpipes for a lonely scene in a ghost story.

Organizational Guidelines

Grade school teachers always teach that organizing your thoughts before writing a composition will dramatically improve its understanding. The same principle applies to user interfaces. Guidelines for organizing information and transitioning between topics will improve the voice user interface.

* Guideline: Use audio to indicate structure.

Example guidelines: There is no standard assignment of meanings to sounds, so common sense and user testing should guide the dialog designer. Here are some suggestions for items that lend themselves to non-speech sounds:

• Links—Identify words that the user may say to jump to another VoiceXML document by introducing them with a unique sound.

• Turn taking tone —A tone signals to the user that the system has finished talking and that the user may speak.

• Brand earcon —Many businesses have audio icons, such as the distinctive “bong” sound of AT&T, the three tones of NBC, and the four tones for Intel Inside. These audio icons can be presented to the user to announce that the user has arrived at the company’s site.
• Feedback—The user needs to know if the speech application is processing data or waiting for input. A non-speech sound such as a percolating coffee pot is ideal for informing the user that the speech application system is busy processing. It also reassured the user that the application is busy, and has not abnormally terminated. A bell sound is ideal for informing the user that the system is ready for the user’s input.
• Barge-in temporarily disabled —Designers may disable barge-in when presenting advertisements or legal notices. To prevent the user from trying to barge-in—signal the user that barge-in is temporarily disabled by presenting “barge-in disabled” and “barge-in enabled” audio icons.
• Bulleted list—A short sound snippet can be used at the beginning of each item on a list.

** Guideline: Use pauses to divide spoken information into natural “chunks. ”

Guideline examples

• Chunking numbers—Phone numbers, identification numbers, and other number sequences of numbers are frequently clustered into groups of two or three numbers when spoken. A short pause between the pairs of groups helps users comprehend and remember the number easier. For example, in North America, telephone numbers are frequently spoken in three chunks: the three-digit area code, the three-digit exchange number, and the four-digit subscriber number.

• Pause between instructions and options—Placing a pause between instructions and the options of prompt signals the user when the instructions are complete. Experienced users may barge-in after the instructions, but before hearing the list of options.

* Guideline: Use animation and sound to show transitions.

Guideline example

• For example, display a turning page and present an audio pause to indicate the transition between two pages

Normally, human short-term memory holds only to 7±2 items. So replace long verbal presentations by other visual presentations.

*** Guideline: Use the screen to ease stress on the user’s short-term memory.

Guideline examples:

• If a list of options contains more than 7±2 items, display the list of options.
• If possible, display results of a query as a table. For example, display travel schedules as a table instead of presenting a verbal text.
• If text contains more than two sentences, present the text to the user visually rather than verbally

Advertising Guidelines

Techniques from the field of advertising can be applied to user interfaces.

* Guideline: Use animation to attract the user’s attention.

Guideline example:

Animate the delivery of important events and messages so users will notice them. Often this type of animation is accompanied with sound, which also attracts the users’ attention.

Company brands and logos are quickly recognized by users, who unconsciously recognize brands based upon previous experiences with products and services offered by the company. A well-recognized brand name encourages users to purchase associated goods and services. Sound can be an important part of a brand or logo.

* Use audio to create an awareness of company brands and logos.

Guideline example:

• The “bong” heard at the beginning of long distance telephone calls indicates the service is being offered by AT&T. The “Intel Inside” audio logo indicates that Intel supplied the computer chip inside of a computing device.