Our approach assumes a collaborative user who is willing to assist the
system in overcoming recognition errors. Errors are corrected by
providing additional input, either in the same modality, or switching
to other modalities. This multimodal approach to error recovery
attempts to leverage both the fact that input in different modalities
provides redundant information and the fact that switching modalities
itself alleviates user frustration.
We developed a prototypical speech user interface augmented with
repair by respeaking, spelling, handwriting and gestures on a
dictation task. In a typical scenario, the user highlights errors
in the hypothesis displayed, and corrects them by providing input
in a modality of his choice. Preliminary studies have shown that
repair by spelling or handwriting can be very effective - requiring
significantly less effort than repair by choice from N-best
alternatives.
More
back to top
