Gaze assisted search query
A system enables a user to query based on a user's gaze by receiving a query from the user and capturing, via an eye tracking system on a headset, the user's gaze location near an object in a local area. The system captures one or more images of the local area with the object and formats the images based in part on a region of interest in the one or more images that includes the object. The system generates a formatted query based in part on the query. The formatted query is provided to a search engine. Information describing the object determined from the one or more formatted images and information describing the query are used by the search engine to determine an answer to the query about the object. The system presents the answer to the query about the object.
1. A method comprising:
receiving a trigger from a user,
determining a query associated with the trigger, wherein the query is about an object in a local area,
determining, via an eye tracking system on a headset, a gaze location of the user on the object,
capturing, via a camera on the headset, one or more images of the local area that include the object associated with the gaze location,
formatting, based in part on a region of interest (ROI) in the one or more images that includes the object, the one or more images to generate one or more formatted images,
determining, using the one or more formatted images, information describing the object,
generating a formatted query using the query and the information describing the object,
providing the formatted query to a search engine, wherein the information describing the object and information describing the query are used by the search engine to determine an answer to the query about the object, and
presenting the answer to the query about the object.
2. The method of claim 1 , wherein the query is a spoken query that identifies the object in the captured one or more images using a demonstrative.
3. The method of claim 1 , wherein the search engine is a large language model.
4. The method of claim 1 , wherein formatting the one or more images comprises:
identifying, using the gaze location, the ROI in the one or more images; and
adjusting the one or more images to fit the ROI, wherein sizes of the adjusted one or more images are less than sizes of the one or more images.
5. The method of claim 1 , wherein the one or more images are a plurality of images, and each of the plurality of images includes the object and has a same resolution, and at least two of the plurality of images have different fields of view.
6. The method of claim 1 , wherein formatting the one or more images comprises:
identifying, using the gaze location, the ROI in the one or more images; and
generating, using an image of the one or more images, a plurality of images that each includes the object and has a same resolution, and at least two of the plurality of images have different fields of view.
7. The method of claim 1 , wherein formatting the one or more images comprises:
identifying, using the gaze location, the ROI in the one or more images; and
down sampling a region outside of the ROI in each of the one or more images, such that a resolution outside the ROI is lower than a resolution inside the ROI for each of the one or more images.
8. The method of claim 1 , further comprising:
providing the one or more formatted images to an object identification system in order to determine the information describing the object,
wherein the query is a spoken query, and generating the formatted query comprises:
converting the spoken query to text, the text being the information describing the query, and
generating a prompt based in part on the information describing the object and the information describing the query.
9. The method of claim 1 , wherein the search engine is a multi-modal large language model, and providing the formatted query to the search engine further comprises:
providing at least one of the one or more formatted images to the search engine for determining the answer to the query about the object.
10. The method of claim 1 , wherein determining the query associated with the trigger, determining the gaze location of the user on the object, and capturing the one or more images of the local area that include the object associated with the gaze location are performed concurrently.
11. A gaze assisted search system comprising:
an eye tracking system configured to determine a gaze location of a user;
a camera configured to capture one or more images of a local area that include an object associated with the gaze location; and
a controller comprising a processor and a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the processor, cause the controller to:
determine a query associated with a trigger, wherein the query is about the object,
format, based in part on a region of interest (ROI) in the one or more images that includes the object, the one or more images to generate one or more formatted images,
determine, using the one or more formatted images, information describing the object,
generate a formatted query using the query and the information describing the object,
provide the formatted query to a search engine, wherein the information describing the object and information describing the query are used by the search engine to determine an answer to the query about the object, and
instruct a component of the gaze assisted search system to present the answer to the query about the object.
12. The system of claim 11 , wherein the query is a spoken query that identifies the object in the captured one or more images using a demonstrative.
13. The system of claim 11 , wherein the search engine is a large language model.
14. The system of claim 11 , wherein the instructions to format the one or more images further comprise instructions that when executed cause the controller to:
identify, using the gaze location, the ROI in the one or more images; and
adjust the one or more images to fit the ROI, wherein sizes of the adjusted one or more images are less than sizes of the one or more images.
15. The system of claim 11 , wherein the one or more images are a plurality of images, and each of the plurality of images includes the object and has a same resolution, and at least two of the plurality of images have different fields of view.
16. The system of claim 11 , wherein the instructions to format the one or more images further comprise instructions that when executed cause the controller to:
identify, using the gaze location, the ROI in the one or more images; and
generate, using an image of the one or more images, a plurality of images that each includes the object and has a same resolution, and at least two of the plurality of images have different fields of view.
17. The system of claim 11 , wherein the instructions to format the one or more images further comprise instructions that when executed cause the controller to:
identify, using the gaze location, the ROI in the one or more images; and
down sample a region outside of the ROI in each of the one or more images, such that a resolution outside the ROI is lower than a resolution inside the ROI for each of the one or more images.
18. The system of claim 11 , wherein the controller is further configured to:
provide the one or more formatted images to an object identification system in order to determine the information describing the object,
wherein the query is a spoken query, and the instructions to generate the formatted query further comprise instructions that when executed cause the controller to:
convert the spoken query to text, the text being the information describing the query, and
generate a prompt based in part on the information describing the object and the information describing the query.
19. The system of claim 11 , wherein the search engine is a multi-modal large language model, and the instructions to provide the formatted query to the search engine further comprise instructions that when executed cause the controller to:
provide at least one of the one or more formatted images to the search engine for determining the answer to the query about the object.
20. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor of a computer system, cause the processor to:
responsive to receiving a trigger from a user,
determine a query associated with the trigger, wherein the query is about an object in a local area,
determine, via an eye tracking system on a headset, a gaze location of the user on the object,
capture, via a camera on the headset, one or more images of the local area that include the object associated with the gaze location,
format, based in part on a region of interest (ROI) in one or more images that includes the object, the one or more images to generate one or more formatted images,
determine, using the one or more formatted images, information describing the object,
generate a formatted query using the query and the information describing the object,
provide the formatted query to a search engine, wherein the information describing the object and information describing the query are used by the search engine to determine an answer to the query about the object, and
present the answer to the query about the object.