Telepresence using panoramic imaging and directional sound and motion
An imaging system comprising a panoramic visual image display, an associated directional sound playback device, and an associated motion reproduction device is disclosed. The imaging system conveys visual, sound and motion information related to a particular viewing direction to provide a realistic experience for the viewer. The imaging system can also comprise a panoramic visual image recording device capable of recording panoramic images, an associated directional sound capturing device capable of recording sound, and an associated directional motion capturing device capable of recording motion. Recorded panoramic images, sound and motion can be synchronously recorded to a common time code for simultaneous playback.
1. An apparatus comprising:
a panoramic visual image capturing device capable of recording panoramic visual images;
a sound capturing device capable of recording sound associated with viewing directions for the panoramic visual images; and
a motion capturing device capable of recording motion of the panoramic visual image capturing device to produce captured motion data that can be synchronized with the panoramic visual images captured by the panoramic visual image capturing device, and can be associated with a particular image view corresponding to only a portion of the panoramic visual images.
2. The apparatus of claim 1 , further comprising at least one additional sound capturing device positioned remotely from the panoramic visual image capturing device.
3. The apparatus of claim 1 , further comprising at least one additional motion capturing device positioned remotely from the panoramic visual image capturing device.
4. The apparatus of claim 1 , further comprising a processor for processing signals from the sound capturing device and the motion capturing device to determine a location of a source of sound and motion.
5. A method comprising the steps of:
recording panoramic visual images;
recording sound associated with viewing directions for the panoramic visual images;
recording motion of the panoramic visual image capturing device to produce captured motion data; and
synchronizing the captured motion data with the panoramic visual images captured by the panoramic visual image capturing device, wherein the captured motion data can be associated with a particular image view corresponding to only a portion of the panoramic visual images.
6. The method of claim 5 , wherein the sound associated with viewing directions for the visual images is recorded at a position remote from a visual image capturing device.
7. The method of claim 5 , wherein the motion associated with viewing directions for the visual images is recorded at a position remote from a visual image capturing device.
8. The method of claim 5 , wherein the sound associated with viewing directions for the visual images is recorded on two or more channels, and a temporal and spatial correspondence is established between the panoramic visual image and the recorded sound.
9. The method of claim 5 , wherein at least one of the panoramic visual images, the recorded sound and/or the recorded motion is synthetic.
10. The method of claim 5 , wherein a spatial correspondence between the panoramic images and recorded sound is achieved by localizing sources of the sound and embedding localized information in a data stream that contains both panoramic image data and sound data.
11. The method of claim 10 , wherein the sources of sound are localized by measuring both the loudness and phase of the sound, and by computing an estimate of the location of the sources.
12. The method of claim 5 , wherein the step of recording motion associated with viewing directions for the visual images includes the steps of:
sensing motion in different areas of an environment; and
coupling the sensed motion to the panoramic visual images.
13. The method of claim 5 , wherein the sound and motion are recorded over time and synchronized to a common time code.
14. The method of claim 13 , wherein the common time code is generated and stamped on digital samples of the panoramic visual image, the recorded sound, and the recorded motion.
15. An image playback system comprising:
a panoramic visual image display for displaying panoramic visual images;
a sound playback device for reproducing sound associated with viewing directions for the panoramic visual images; and
a motion reproduction device for reproducing motion associated with a particular image view corresponding to only a portion of the panoramic visual images, wherein the panoramic visual images, sound and motion are synchronously reproduced.
16. An image playback method comprising the steps of:
displaying previously recorded panoramic visual images;
reproducing sound associated with viewing directions for the previously recorded panoramic visual images; and
reproducing motion associated with a particular image view corresponding to only a portion of the previously recorded panoramic visual images, wherein the panoramic visual images, sound and motion are synchronously reproduced.
17. The image playback method of claim 16 , further comprising the step of:
changing the viewing directions for the panoramic visual image by utilizing a pre-determined computer program, or by a user manipulating a haptic device.
18. The image playback method of claim 16 , wherein as the viewing direction changes, the strength of the sound and the motion parameters vary as a function of the angle between the viewing direction and a direction to a source of the sound and motion.
19. The image playback method of claim 16 , wherein the sound is modulated according to the alignment of a sound source with respect to a current viewing direction.
20. The image playback method of claim 16 , wherein the phase and loudness of the sound is modulated to emulate a position of a sound source with respect to a current viewing direction.
21. The image playback method of claim 16 , wherein a user chooses a direction of view that defines an offset from a reference frame, and the sound and motion are played back in a modified manner based on the offset from the panoramic visual image reference frame.