Smart audio augmented reality system
A head-mounted device (HMD) has a transparent display and an array of microphones. The HMD determines a task-based context of the HMD based on a task being performed by a user of the HMD, and determines an ambient-based context of the HMD based on audio data produced by the array of microphones. The HMD detects a first notification configured to be displayed in the transparent display of the HMD. The HMD intercepts the first notification in response to detecting the first notification and generates a second notification based on a priority level of the first notification, the task-based context, and the ambient-based context. The second notification includes a combination of an audio notification and a visual notification (AR content). The HMD replaces the first notification with the second notification and provides the second notification at the HMD.
1. A head-mounted device (HMD) comprising:
a transparent display;
an array of microphones; and
one or more hardware processors comprising a notification application, the notification application being configured to perform operations comprising:
determining a task-based context of the HMD based on a detection of a task being performed by a user of the HMD;
determining an ambient-based context of the HMD based on audio data produced by the array of microphones;
intercepting a first notification configured to be displayed in the transparent display of the HMD;
generating a second notification based on a priority level of the first notification, the task-based context, and the ambient-based context, the second notification comprising a combination of an audio notification and a visual notification, the visual notification including augmented reality (AR) content;
replacing the first notification with the second notification; and
providing the second notification at the HMD.
2. The HMD of claim 1 , wherein the operations further comprise:
detecting an audio alert using the array of microphones;
determining a location of the audio alert using the array of microphones;
generating the AR content based on the task-based context and the ambient-based context, the AR content identifying the location of the audio alert; and
causing the AR content to be displayed in the transparent display.
3. The HMD of claim 1 , wherein determining the task-based context further comprises:
evaluating a level of difficulty of the task; and
determining a priority level of the task,
wherein the task-based context is determined based on the level of difficulty of the task and the priority level of the task.
4. The HMD of claim 3 , wherein determining the task-based context further comprises:
determining biometric data related to the user of the HMD, the biometric data identifying a physiological state of the user and a state of mind of the user while the user is performing the task,
wherein the task-based context is determined based on the biometric data.
5. The HMD of claim 1 , wherein determining the ambient-based context further comprises:
determining an ambient noise level; and
identifying an audio setting of the HMD and an audio level of speakers in the HMD,
wherein the ambient-based context is determined based on the ambient noise level, the audio setting, and the audio level of the speakers in the HMD.
6. The HMD of claim 5 , wherein the operations further comprise:
determining that the ambient noise level is above a first audio threshold;
determining that the audio level of the speakers is below a second audio threshold;
increasing the audio level of the speakers in the HMD in response to determining that the ambient noise level is above the first audio threshold and the audio level of the speakers is below the second audio threshold; and
playing the audio notification at the increased audio level of the speakers in the HMD.
7. The HMD of claim 1 , wherein the AR content further identifies:
an event corresponding to an audio alert; and
a location of the event relative to an orientation and a position of the HMD.
8. The HMD of claim 7 , wherein the operations further comprise:
accessing a floorplan corresponding to a location of the HMD,
wherein the AR content further comprises an escape route based on the floorplan, the escape route corresponding to the event and the location of the event.
9. The HMD of claim 7 , wherein the operations further comprise:
increasing an audio level of the audio notification based on a distance between the location of the HMD and the location of the event.
10. The HMD of claim 7 , wherein the operations further comprise:
increasing a frequency of the audio notification based on a distance between the location of the HMD and the location of the event.
11. A method comprising:
determining a task-based context of a head-mounted device (HMD) based on a detection of a task being performed by a user of the HMD;
determining an ambient-based context of the HMD based on audio data produced by an array of microphones at the HMD;
intercepting a first notification configured to be displayed in the transparent display of the HMD;
generating a second notification based on a priority level of the first notification, the task-based context, and the ambient-based context, the second notification comprising a combination of an audio notification and a visual notification, the visual notification including augmented reality (AR) content;
replacing the first notification with the second notification; and
providing the second notification at the HMD.
12. The method of claim 11 , further comprising:
detecting an audio alert using the array of microphones;
determining a location of the audio alert using the array of microphones;
generating the AR content based on the task-based context and the ambient-based context, the AR content identifying the location of the audio alert; and
causing the AR content to be displayed in the transparent display.
13. The method of claim 11 , wherein determining the task-based context further comprises:
evaluating a level of difficulty of the task; and
determining a priority level of the task,
wherein the task-based context is determined based on the level of difficulty of the task and the priority level of the task.
14. The method of claim 13 , wherein determining the task-based context further comprises:
determining biometric data related to the user of the HMD, the biometric data identifying a physiological state of the user and a state of mind of the user while the user is performing the task,
wherein the task-based context is determined based on the biometric data.
15. The method of claim 11 , wherein determining the ambient-based context further comprises:
determining an ambient noise level; and
identifying an audio setting of the HMD and an audio level of speakers in the HMD,
wherein the ambient-based context is determined based on the ambient noise level, the audio setting, and the audio level of the speakers in the HMD.
16. The method of claim 15 , further comprising:
determining that the ambient noise level is above a first audio threshold;
determining that the audio level of the speakers is below a second audio threshold;
increasing the audio level of the speakers in the HMD in response to determining that the ambient noise level is above the first audio threshold and the audio level of the speakers is below the second audio threshold; and
playing the audio notification at the increased audio level of the speakers in the HMD.
17. The method of claim 11 , wherein the AR content further identifies:
an event corresponding to an audio alert; and
a location of the event relative to an orientation and a position of the HMD.
18. The method of claim 17 , further comprising:
accessing a floorplan corresponding to a location of the HMD,
wherein the AR content further comprises an escape route based on the floorplan, the escape route corresponding to the event and the location of the event.
19. The method of claim 17 , further comprising:
adjusting the audio notification in the HMD based on a location of the HMD relative to the location of the event.
20. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
determining a task-based context of a head-mounted device (HMD) based on a detection of a task being performed by a user of the HMD;
determining an ambient-based context of the HMD based on audio data produced by an array of microphones at the HMD;
intercepting a first notification configured to be displayed in the transparent display of the HMD;
generating a second notification based on a priority level of the first notification, the task-based context, and the ambient-based context, the second notification comprising a combination of an audio notification and a visual notification, the visual notification including augmented reality (AR) content;
replacing the first notification with the second notification; and
providing the second notification at the HMD.