Context-aware hazard detection using world-facing cameras in virtual, augmented, and mixed reality (xR) applications
Systems and methods for alerting a user to safety hazards in real-world while the user is wearing a Head-Mounted Display (HMD). In some embodiments, an Information Handling System (IHS) may include: a processor; and a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to: receive an image during execution of a xR application displayed to a user wearing a Head-Mounted Display (HMD) coupled to the IHS; detect an object in the image; associate the object with a landmark selected among a plurality of landmarks usable by the xR application to determine a position of the HMD; and provide to the user, via the HMD, a safety instruction related to the object in response to a distance between the HMD and the selected landmark meeting a distance threshold.
1. An Information Handling System (IHS), comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to:
receive a frame from a camera mounted on a Head-Mounted Display (HMD) worn by a user;
associate an object detected in the frame with a Simultaneous Localization and Mapping (SLAM) landmark extracted from an infra-red (IR) image;
request that the SLAM landmark be maintained within a tracked set of SLAM landmarks;
in response to a resource utilization reaching a selected value, reduce the tracked set of SLAM landmarks by eliminating one or more SLAM landmarks with low Kalman Gains;
retain the requested SLAM landmark, wherein the requested SLAM landmark has a lower Kalman Gain than the eliminated one or more SLAM landmarks; and
provide to the user, via the HMD, a safety instruction in response to a distance between the user and the SLAM landmark meeting a threshold.
2. The IHS of claim 1 , wherein the frame is received from a World-Facing (WF) camera mounted on the HMD.
3. The IHS of claim 2 , wherein the requested SLAM landmark is detected using an Infra-Red (IR) or near-IR (NIR) camera decoupled from the HMD.
4. The IHS of claim 2 , wherein the object is detected in the frame using a feature extraction process.
5. The IHS of claim 4 , wherein one or more extracted features match one or more corresponding features of an object model stored in a database.
6. The IHS of claim 5 , wherein the database indicates, for the object model, a plurality of different severity values, each of the severity values associated with a different user identity, wherein providing the safety instruction further comprises rendering an image for display by the HMD corresponding to a severity value associated with the user.
7. The IHS of claim 6 , wherein the program instructions, upon execution by the processor, further cause the IHS to track a height of the object relative to the ground based upon the requested SLAM landmark.
8. The IHS of claim 7 , wherein the severity value associated with the user is based upon a height of the user, and wherein the safety instruction is provided, in part, in response to the object being located above a selected height from the ground.
9. The IHS of claim 6 , wherein the program instructions, upon execution by the processor, further cause the IHS to track a speed of the object based upon the requested SLAM landmark.
10. The IHS of claim 9 , wherein the severity value is proportional to the speed.
11. The IHS of claim 6 , wherein the program instructions, upon execution, further cause the IHS to, in response to an IHS resource utilization reaching a predetermined value, reduce the number of landmarks used to determine the position of the HMD.
12. The IHS of claim 11 , wherein the program instructions, upon execution, further cause the IHS to, maintain the requested SLAM landmark among the used landmarks in response to the severity value being above a safety threshold.
13. The IHS of claim 6 , wherein rendering the image further comprises providing a feed of camera frames from the WF camera to a display within the HMD.
14. A hardware memory device having program instructions stored thereon that, upon execution by a Head-Mounted Display (HMD) worn by a user during execution of a Virtual, Augmented, or Mixed Reality (xR) application, cause the HMD to:
receive a frame from a camera mounted on a Head-Mounted Display (HMD) worn by a user;
associate an object detected in the frame with a Simultaneous Localization and Mapping (SLAM) landmark extracted from an infra-red (IR) image;
request that the SLAM landmark be maintained within a tracked set of SLAM landmarks;
in response to a resource utilization reaching a selected value, reduce the tracked set of SLAM landmarks by eliminating one or more SLAM landmarks with low Kalman Gains;
retain the requested SLAM landmark, wherein the requested SLAM landmark has a lower Kalman Gain than the eliminated one or more SLAM landmarks; and
provide to the user, via the HMD, a safety instruction in response to a distance between the user and the SLAM landmark meeting a threshold.
15. The hardware memory device of claim 14 , wherein the detected object is associated with a severity value, and wherein providing the safety instruction further comprises selecting an image corresponding to a severity value associated with the detected object.
16. The hardware memory device of claim 15 , wherein the program instructions, upon execution, further cause the HMD to track a height or speed of the detected object based upon the requested SLAM landmark.
17. The hardware memory device of claim 15 , wherein the object comprises a chemical, and wherein the severity value is based upon an allergy of the user to the chemical.
18. A method, comprising:
receiving a frame from a camera mounted on a Head-Mounted Display (HMD) worn by a user;
associating an object detected in the frame with a Simultaneous Localization and Mapping (SLAM) landmark extracted from an infra-red (IR) image;
requesting that the SLAM landmark be maintained within a tracked set of SLAM landmarks;
in response to a resource utilization reaching a selected value, reducing the tracked set of SLAM landmarks by eliminating one or more SLAM landmarks with low Kalman Gains;
retaining the requested SLAM landmark, wherein the requested SLAM landmark has a lower Kalman Gain than the eliminated one or more SLAM landmarks; and
providing to the user, via the HMD, a safety instruction in response to a distance between the user and the SLAM landmark meeting a threshold.
19. The method of claim 18 , further comprising:
tracking a height or speed of the detected object using the SLAM landmark;
associating a severity value with the object based upon the height or speed; and
providing a notification to the user corresponding to the severity value.