Human-Machine Collaboration Using Natural Language Interfaces

True collaboration between humans and computers requires the computer to understand the range of interactions that naturally occur in human dialogue. In addition, practical models that address problems such as intention recognition, planning in dynamic situations, and mixed-initiative interaction will improve the interaction of humans and their machines.

IHMC scientists are developing computational systems that use natural human speech as the interface. Recognizing the words themselves is a huge challenge. Researchers are developing linguistic models to assist in recognizing spoken words. These models build on simple word prediction systems by incorporating hierarchical structure into a dynamical system.

This system can then examine the speech at different levels to figure out what was said. Whereas simply recognizing words is useful for dictation systems, knowing what the words mean is critical for conversation systems. Dictionary definitions only go so far; the system must also understand the person’s intentions. IHMC’s intention recognition systems integrate task models that allow the system to figure out the relationship between what a person says and what they do. The system tracks the objectives and planned solutions in the task model, helping it interpret ambiguous sentences. The systems under development have a generic structure into which language specific to the current task is incorporated.

Very complex systems require mixed-initiative interaction. Mixed- initiative interaction allows the system and user to exchange the lead in planning and meeting the goals of the task, exploiting the strengths of both the computer and the human. In controlling multiple robots, for instance, it isn’t feasible for the human to remember and keep track of all the abilities and specifications of the robots. IHMC researchers are developing mixed-initiative systems that allow the human to give high-level commands. The computer then figures out how to execute the command, perhaps returning to the human for permission to perform certain actions.

Humans learn through a variety of modalities, including observation, trial and error, and instruction. We also find that different tasks are more suited to different learning methods. Currently, computer learning of tasks requires many examples of the task being performed. IHMC researchers are adding natural language capabilities to an integrated learner. Human instructors explain the steps while acting them out, dramatically increasing the speed by which the system learns.