head mounted projector

Today, the pace of surgical innovations has increased
dramatically, as have the societal demands for safe and
effective practices. The mechanisms for training and retraining
suffer from inflexible timing, extended time
commitments, and limited content. Video instruction has
long been available to help surgeons learn new
procedures, but it is generally viewed as marginally
effective at best for a number of reasons, such as the fixed
point of view that is integral to the narration, lack of depth
perception and interactivity, and missing information [1].
In short, the experience of watching a video is not
sufficiently close to being there and seeing the procedure.
A paradigm that uses immersive Virtual Reality could
be a more effective approach to allow surgeons to witness
and explore a past surgical procedure as if they were
there. We are indeed pursuing such an immersive
paradigm together with our medical collaborators at the
UNC-Chapel Hill School of Medicine (Dr. Bruce Cairns
and Dr. Anthony Meyer), and our computer graphics
collaborators at Brown University (Andy van Dam et al).
This paradigm demands methods to record the procedure
and to reconstruct the original time-varying events to
create an immersive 3D virtual environment of the real
scene. A more complete solution should also allow
relevant instructions and information, such as vocal
narration, 3D annotations and illustrations, to be added by
the original surgeon or other instructors.
Besides the recording and the reconstruction, providing
an effective way to display a 3D virtual environment to
the user is also a major challenge. In this paper, we
introduce a hybrid approach to address this challenge.
During a typical use of the training system, the trainee
would usually stand beside the patient paying close
attention to the surgery. She might even stand in the
position of a surgeon and observe the procedure from his
(a) (b)
Figure 1. Different views of a surgical operation.
Figure 2. A user using our prototype system based
on our hybrid display approach that combines a
HMD and a projector-based display.
point of view. At the same time, the trainee is also
required to be aware of the surrounding events that could
affect the surgeons’ actions. Such surrounding events
include the actions of other surgeons and technicians,
changes in monitoring and life-support devices, and
overall awareness of the patient’s dynamic condition.
Figure 1(a) shows a close-up view of a real surgical
operation in progress, and Figure 1(b) shows a snapshot
of the many events happening in the operation room.
The visual needs of the trainee can be divided into two
main parts. The first part requires high-quality stereo view
of the objects and events that the trainee is paying direct
attention to, such as the main surgical procedure. Highquality
and high-resolution views are needed to discern
the great intricacy of the surgery, and stereovision is
needed for better spatial understanding. The second part
of a trainee’s visual needs is the peripheral view of her
surroundings. This is needed by the trainee to maintain
visual awareness of the surrounding events. Our medical
collaborators, and others in the field, feel that visual
awareness of the entire patient and the surroundings is a
critical component of surgical training. In particular, with
trauma surgery there is typically a lot of relevant activity
in the operating room. It has been found that in the human
visual system, resolution in the periphery is less dense
than in the fovea [2], therefore peripheral view need not
be high-resolution and high-quality.
Traditionally, head-mounted displays (also called
head-worn displays) have been used to provide highquality
stereo visualization of 3D virtual environments.
However, most HMDs offer limited fields of view, often
only 40° to 60° horizontally and 30° to 45° vertically.
Wide-FOV HMDs have been manufactured, but they are
rare, expensive and heavy to wear. We are aware of no
HMD that can fully cover the human field of view of
approximately 200° horizontally and 135° vertically [3].
Although HMDs are good at providing high-quality stereo
views, the generally narrow FOV has rendered them less
than ideal for providing peripheral views.
The common alternatives to HMDs for immersive
visualization of 3D virtual environments are immersive
projector-based displays, such as the CAVETM [4]. Most
immersive projector-based displays are capable of
providing very wide-field-of-view visualization, and like
CAVETM, some of them are even capable of fully
covering the human field of view. Because of the
relatively large display surfaces and the fact that the user
may move close to them, the image quality and resolution
of such projector-based systems may be insufficient for
applications that require the display of fine details.