Methods and systems for providing health interaction information in an augmented reality presentation
An interaction checking method, device, and system determines health interactions, in real time, as images are collected from an environment and provides augmented informational data presentations of the same. The interactions are based on health information, historical information, and stored interaction data. In particular, the images are received from a camera viewing the environment and an identity is determined of the objects in the environment. Based on the identity determined, information about the objects is retrieved from a memory device, and an interaction warning is determined for a specific combination of the objects if the objects were to be used together. A display device augments visual data of the environment with the information about the objects in a presentation to a user that includes the interaction warning and information about the objects.
1. A mobile device, comprising: a camera;
a display device;
a processor coupled to the camera and the display device;
and a memory coupled with and readable by the processor and storing therein instructions that, when executed by the processor, cause the processor to:
receive, from the camera, images of an environment;
determine, from the images of the environment received, an identity of at least two objects in the environment, wherein a first object of the at least two objects corresponds to at least one of a medication and a vitamin and a second object of the at least two objects corresponds to a food;
retrieve, from a memory device based on the identity of the at least two objects in the environment, information about the at least two objects;
determine, based on the information about the at least two objects, that an interaction warning is associated with a combination of the at least two objects used together;
present, by the display device as the images of the environment are received and while the second object is present in the images of the environment, the interaction warning adjacent an image of the second object in the images of the environment; and
present, by the display device as the images of the environment are received, a health hazard warning when both of the at least two objects appear in the images of the environment at a same time,
wherein the health hazard warning comprises a graphical icon linking both of the at least two objects together.
2. The mobile device of claim 1 , wherein the instructions further cause the processor to:
present, by the display device as the images of the environment are received from the camera, the information about at least one object of the at least two objects when the at least one object appears in the images of the environment.
3. The mobile device of claim 2 , wherein the mobile device comprises a head mounted display, and wherein presenting the information and the interaction warning comprises projecting light comprising the information and the interaction warning directly onto a retina of a user of the head mounted display.
4. The mobile device of claim 2 , wherein the display device comprises a display screen, and wherein presenting the information and the interaction warning comprises rendering the images of the environment to the display screen as the images of the environment are received and rendering images of the information and the interaction warning in an augmented reality combined presentation.
5. The mobile device of claim 1 , wherein the instructions further cause the processor to:
authenticate a user of the mobile device; and
retrieve, from a user account database in response to authenticating the user, heath-health information associated with the user, wherein the health information comprises at least one medical treatment, prescribed medication, preexisting condition, and state of health associated with the user.
6. The mobile device of claim 5 , wherein the health information comprises a vision disability as the preexisting condition, and wherein presenting the information about each object when each object appears in the images of the environment further comprises at least one of adjusting a contrast, a color, and a size of the information presented based on the vision disability.
7. The mobile device of claim 5 , wherein the instructions further cause the processor to:
determine, based on the information about the at least two objects and the health information, that an interaction between at least one object of the at least two objects and a portion of the health information associated with the user comprises a personal health hazard to the user.
8. The mobile device of claim 7 , wherein the health hazard warning comprises information about the interaction.
9. The mobile device of claim 8 , wherein the health hazard warning comprises a severity interaction warning icon that identifies a severity of the personal health hazard, and wherein the severity interaction warning icon is arranged adjacent the first object and is arranged adjacent the second object while both of the at least two objects remain visible in the images.
10. The mobile device of claim 1 , wherein the instructions further cause the processor to:
determine, as part of determining the identity of the at least two objects, an identity of a first object of the at least two objects at a first time; and
determine, as part of determining the identity of the at least two objects, an identity of a second object of the at least two objects at a second time subsequent to the first time, and
wherein the interaction warning is presented by the display device only when the second object of the at least two objects appears in the images of the environment.
11. The mobile device of claim 1 , wherein the instructions further cause the processor to output audio, via a speaker of the mobile device, comprising the information about each object when each object appears in the images of the environment.
12. The mobile device of claim 1 , wherein the instructions further cause the processor to output a vibration, via a tactile transducer of the mobile device, upon determining the identity of the at least two objects in the environment.
13. A method, comprising:
receiving, from at least one camera of a mobile device, digital images of a medication and a food object, wherein the food object is a consumable other than the medication;
determining, based on the digital images received, an identity of the medication and of the food object;
retrieving, based on the identity, information about the medication and information about the food object;
determining an authentication of a user of the mobile device with a user health account;
presenting, by a display device of the mobile device when the user is determined to be unauthenticated with the user health account, the information about the food object while the food object remains visible in the digital images as the digital images are received from the at least one camera;
retrieving, from the user health account, health information for the user only when the user is determined to be authenticated with the user health account;
determining whether an interaction between the food object and a portion of the health information for the user comprises a health hazard to the user;
presenting, by the display device when the interaction between the food object and the portion of the health information for the user is determined to comprise the health hazard to the user, an interaction warning describing the interaction while the food object remains visible in the digital images,
wherein the interaction warning is presented in a display location adjacent the food object; and
presenting, by the display device when the portion of the health information comprises a known interaction between the medication and the food object, a health hazard warning when both the medication and the food object appear in the digital images at a same time,
wherein the health hazard warning comprises a graphical icon linking the medication and the food object together.
14. The method of claim 13 , wherein the interaction warning is presented as the digital images are received from the at least one camera.
15. The method of claim 14 , wherein at least one of the information about the food object and the interaction warning are rendered via the display device as an augmented reality presentation, and wherein the at least one of the information about the food object and the interaction warning are overlaid upon the digital images received from the at least one camera, in real time, as the digital images are received from the at least one camera.
16. The method of claim 13 , wherein the health information for the user comprises at least one medical treatment, prescribed medication, preexisting condition, and state of health of the user.
17. The method of claim 16 , wherein the health information comprises a vision impairment as the preexisting condition, and wherein presenting the information about the food object while the food object remains visible in the digital images further comprises at least one of adjusting a contrast, a color, a filter of light, and a size of the information presented based on the vision impairment.
18. A non-transitory computer-readable medium comprising instructions stored therein that, when executed by a processor, cause the processor to:
receive, from a camera, video images of an environment;
determine, from the video images received, an identity of at least two objects in the environment, wherein a first object of the at least two objects corresponds to a medication and a vitamin and a second object of the at least two objects corresponds to a food;
retrieve, from a memory device based on the identity of the at least two objects in the environment, information about the at least two objects, wherein at least a portion of the information about the at least two objects is not visible on the at least two objects;
determine, based on the information about the at least two objects, that an interaction warning is associated with combining the at least two objects together;
present, by a display device as the video images of the environment are received and while the second object is present in the video images of the environment, the interaction warning adjacent the second object; and
present, by the display device as the video images of the environment are received, a health hazard warning when both of the at least two objects appear in the video images of the environment at a same time,
wherein the health hazard warning comprises a graphical icon linking both of the at least two objects together.
19. The non-transitory computer-readable medium of claim 18 , wherein determining the identity of the at least two objects in the environment comprises at least one of visual recognition and optical character recognition of the at least two objects using artificial intelligence enabled by a machine learning engine analyzing the video images.
20. The non-transitory computer-readable medium of claim 18 , wherein retrieving the information about the at least two objects comprises accessing a database comprising identities of objects and corresponding known interactions, ingredients, and names, and wherein the database comprises the identity of the at least two objects.