Many human-computer-interaction applications require the information where a person is looking and what he or she is paying attention to. Such information can can be obtained from tracking the human gaze. While current approaches to gaze tracking tend to be highly intrusive - the subject must either stay very still or wear a special device such as head mounted camera - we have developed a non-intrusive gaze tracking system that allows the user to move freely in the field of view of a camera.

Where a person is looking at is mainly determined by two components: the orientation of the head, and the orientation of the eyes within their sockets.
In this system we focus on estimating the pose of the head, e. g. the 3-D rotation and translation of the head.
The pose of an object can be described through a rotation matrix R , which maps a 3D object point x onto an image points p: R x = p. The nine parameters of the rotation matrix R can be computed by finding correspondences between model points of the head and corresponding points in a camera image.
In order to estimate the pose of the head, we therefore try to find and track six facial feature points in the image - eyes, nostrils and lip corners.

To track the facial features, first the face of the user is searched and tracked in the camera image, using a statistical color model (see our Face Tracking page).
Then, the eyes, nostrils and lip corners are searched and tracked inside the facial region.
You can read more about the searching and tracking of facial features in our system.

Our current system allows the user to freely move in the view of the camera. It automatically finds and tracks facial features and can recover from tracking failures. The system runs at a frame rate of about 15+ frames / second on a HP 9000 using a framegrabber and a Canon VC-C1 camera.

Sample Tracking Sequence
(Features tracked automatically)

Applications

We have developed a multimodal interface to view panorama images, where we use the gaze tracker to scroll through the 360 degree panorama images and spoken commands to zoom in and out. With such an interface, the user can fully control the panorama image viewer without using his hands.
Other potential applications include virtual reality and video-teleconferencing.

back to top

---

KEYWORDS Face Tracking, Gaze Tracking, Pose Estimation, Head Orientation