Advanced head display unit for fire fighters
In this patent, an advanced head display unit designed primarily for fire fighters is disclosed. The advanced augmented reality/virtual reality (AR/VR) head display unit improves coordination between teammates through eye tracking coupled with augmented reality features. This allows one fire fighter to know where another fire fighter is looking and helps coordinate tasks by dividing the scene into sectors and visibly marking each sector. Further, this system helps determine where to hose with a smart target system. Further, multiple sensors are utilized together to triangulate the location of a victim's voice. Additional advantages are also disclosed above.
1. A method comprising:
performing scene understanding of an image of a physical scene obtained from a sensor system on a head display unit
wherein the physical scene contains physical items,
wherein said image contains structures,
wherein said structures correspond to said physical items,
wherein said scene understanding comprises segmentation of said image into segmented structure(s);
displaying said image on said head display unit;
performing eye tracking of a first user's eye movements while said first user is wearing the head display unit
wherein the head display unit has eye tracking capabilities, and
wherein the first user visualizes at least one of said segmented structures which corresponds to at least one of the physical items; and
analyzing eye tracking data of the first user to determine which segmented structure(s) said first user has viewed.
2. The method of claim 1 further comprising:
generating a digital object to designate segmented structures that are not detected; and
displaying said digital object on said head display unit.
3. The method of claim 1 further comprising:
determining a relative location of the head display unit worn by the first user as compared to a second head display unit worn by a second user;
determining the second head display unit's pointing direction; and
displaying a digital line on the first head display unit wherein the digital line originates in proximity to the second head display unit and extends in the pointing direction of the second head display unit.
4. The method of claim 1 further comprising:
determining a look angle direction of a second user wearing a second head display unit with eye tracking capabilities;
displaying a digital line on the head display unit wherein the digital line originates in proximity to the second head display unit and extends in the look angle of the second user.
5. The method of claim 1 further comprising:
determining a convergence point of a second user in the scene;
displaying a digital line on the head display unit wherein the digital line originates in proximity to the second head display unit and extends to the convergence point of the second user.
6. The method of claim 1 further comprising:
determining a pointing direction of an object held by a second user; and
displaying a digital line on the head display unit wherein the digital line originates in proximity to the second user and extends in the pointing direction of the object.
7. The method of claim 1 further comprising:
performing an aided target recognition (AiTR) of said segmented structure(s) within said image.
8. The method of claim 7 further comprising:
wherein if said AiTR determines that a segmented structure is a target, providing an alert to said first user wherein said alert comprises a visual alert, an audio alert or a tactile alert.
9. The method of claim 1 further comprising:
determining said first user's fixation location of a physical item in said scene comprising:
using said first user's GPS,
using said first user s laser range finder, and
using said first user's digital compass, and
recording said first user's fixation location of said physical item in said scene.
10. The method of claim 9 further comprising:
performing an aided target recognition (AiTR) of said segmented structure at said first user's fixation location,
recording results of said AiTR; and
passing results of said AiTR and said first user's fixation location to a second user.
11. The method of claim 1 further comprising placing a digital object on said head display unit in proximity to a moving physical item within the scene, which enables the first user to perform smooth tracking of the moving physical item.
12. The method of claim 1 further comprising placing an appearing-disappearing digital object on said head display unit in proximity to a moving physical item within the scene, which enables the first user to perform saccades of the moving physical item.
13. The method of claim 1 further comprising wherein the head display unit comprises at least one forward looking infrared camera and wherein an artificial intelligence algorithm uses the data from the at least one forward looking infrared camera to determine the optimum aim point for a fire hose and wherein the optimum aim point is displayed on the head display unit.
14. The method of claim 1 further comprising wherein the head display unit comprises a laser range finder and wherein the laser range finder generates a 3D image of the physical items and wherein a digital 3D image of the physical items is displayed on the head display unit to the first user.
15. The method of claim 1 further comprising wherein the first user's head display unit contains a first acoustic sensor and a first position locator and a second user's head display unit contains a second acoustic sensor and a second position locator and wherein data from the first user's head display unit and data from the second user's head display unit are utilized to triangulate the location of a sound.
16. The method of claim 1 further comprising wherein the first user's head display unit contains a hologram generator wherein a second user can view a hologram generated by said hologram generator with the his or her naked eye.
17. The method of claim 1 further comprising
wherein the first user's left hand is located on the first user's left thigh;
wherein the first user's right hand is located on the first user's right thigh;
wherein the first user's left hand, right hand, left thigh and right thigh are within a camera's field of view;
wherein a movement of a finger of the user's right hand causes a first digital keystroke; and
wherein a movement of a finger of the user's left hand causes a second digital keystroke.
18. The method of claim 17 further comprises wherein a movement comprises a tapping motion, a lifting motion, or a dragging motion.
19. The method of claim 17 further comprises wherein a keystroke is determined by at least one of the group consisting of:
a position of a finger on a thigh;
a speed of a finger movement; and
an artificial intelligence algorithm.
20. A system comprising:
a head display unit (HDD) with a left eye display and a right eye display;
an eye tracking system configured to track a left eye of a user and a right eye of said user,
a sensor system mounted on said HDU comprising at least one of:
a camera system;
a forward looking infrared (FLIR) sensor;
a laser range finder;
a digital compass;
a GPS; and
acoustic sensor(s);
a processor wherein the processor performs processing comprising:
performing scene understanding of an image of a physical scene obtained from said sensor system
wherein the physical scene contains physical items,
wherein said image contains structures,
wherein said structures correspond to said physical items,
wherein said scene understanding comprises segmentation of said image into segmented structure(s);
performing eye tracking of said user s eye movements while said user is wearing the head display unit
wherein the head display unit has eye tracking capabilities, and
wherein said user visualizes at least one of the physical items; and
analyzing eye tracking data of said user to determine which segmented structure(s) said user has viewed.