Augmented reality device and methods of use
Computer-implemented methods of operating an augmented reality device can involve capturing camera images, processing the camera images, and displaying virtual display images. The camera images can be captured automatically using a camera disposed within an augmented reality device worn by a user. The camera images can be processed automatically using a processor located within the augmented reality device. The virtual display images can be displayed automatically to the user within the augmented reality device while the user is looking through the augmented reality device and simultaneously viewing real objects through the augmented reality device. The virtual display images can be based on the processed camera images. Additional steps can include accepting a first user input, storing camera image(s) on a memory located within the augmented reality device based on the first input, accepting a second user input, and displaying stored image(s) to the user based on the second input.
1. A computer-implemented method of operating an augmented reality device, the computer-implemented method comprising:
capturing a first plurality of camera images using a camera disposed within an augmented reality device, wherein the augmented reality device includes a display arrangement configured to provide a virtual display to a user;
processing the first plurality of camera images using a processor located within the augmented reality device;
receiving a first audio signal from a microphone of the augmented reality device;
processing the first audio signal using the processor located within the augmented reality device to create a textual description of the audio signal;
periodically collecting sensor data by the augmented reality device over a period of time;
generating a package including the processed camera images and/or the textual description of the audio signal, wherein the generated package includes text of a question or a query and includes context data comprising the collected sensor data, wherein the augmented reality device includes a pre-set listing of questions for submission to the machine learning model, wherein each of the pre-set listing of questions are associated with a type of sensor data used for a respective question;
transmitting the generated package for input to a machine learning model, wherein the machine learning model comprises a large language model external to the augmented reality device;
receiving an output at least partially generated by the machine learning model; and
displaying a portion of the output via the display arrangement.
2. The method of claim 1 , wherein the augmented reality device further comprises one or more of the sensors: an accelerometer; a gyroscope, position sensor, global positioning system, a temperature sensor.
3. The method of claim 1 , further comprising:
generating a list of available sensor data of the augmented reality device; and
providing the generated list of available sensor data and a request to the machine learning model of what available sensor data of the augmented reality device to use to process the generated package.
4. The method of claim 3 , further comprising:
receiving a response indicating additional sensor data that would be helpful to the processing of the request by the machine learning model; and
providing to the machine learning model data values of the additional sensor data.
5. The method of claim 1 , further comprising the operations of:
receiving a selection of one of the pre-set listing of questions; and
selecting sensor data to be used for the selected question.
6. The method of claim 1 , further comprising:
determining which of the collected sensor data to use in conjunction with the question or the query.
7. The method of claim 1 , wherein the collected sensor data includes location information obtained from a GPS system or a position sensor.
8. The method of claim 7 , wherein the context data comprises a current position or location data and a historic position or location data.
9. An augmented reality device, comprising a camera, a display arrangement, a microphone, and one or more processors configured to perform the operations of:
capturing a first plurality of camera images using the camera, wherein the display arrangement is configured to provide a virtual display to a user;
processing the first plurality of camera images using the one or more processors;
receiving a first audio signal from the microphone;
processing the first audio signal using the one or more processors to create a textual description of the audio signal;
generating a package including the processed camera images and/or the textual description of the audio signal, wherein the augmented reality device includes a pre-set listing of questions for submission to the machine learning model, wherein each of the pre-set listing of questions are associated with a type of sensor data used for a respective question;
periodically collecting sensor data by the augmented reality device;
transmitting the generated package for input to a machine learning model, wherein the generated package includes text of a question or a query and context data comprising the collected sensor data;
receiving an output at least partially generated by the machine learning model, wherein the machine learning model comprises a large language model external to the augmented reality device; and
displaying a portion of the output via the display arrangement.
10. The augmented reality device of claim 9 , wherein the augmented reality device further comprises one or more of the sensors: an accelerometer; a gyroscope, position sensor, global positioning system, a temperature sensor.
11. The augmented reality device of claim 9 , further comprising the operations of:
generating a list of available sensor data of the augmented reality device; and
providing the generated list of available sensor data and a request to the machine learning model of what available sensor data of the augmented reality device to use to process the generated package.
12. The augmented reality device of claim 11 , further comprising the operations of:
receiving a response indicating additional sensor data that would be helpful to the processing of the request by the machine learning model; and
providing to the machine learning model data values of the additional sensor data.
13. The augmented reality device of claim 9 , further comprising the operations of:
receiving a selection of one of the pre-set listing of questions; and
selecting sensor data to be used for the selected question.
14. The augmented reality device of claim 9 , further comprising:
determining which of the collected sensor data to use in conjunction with the question or the query.
15. The augmented reality device of claim 9 , wherein the collected sensor data includes location information obtained from a GPS system or a position sensor.
16. The augmented reality device of claim 15 , wherein the context data comprises a current position or current location data, and a historic position or historic location data.