Identifying an object for object recognition based on a user's gaze determined by a headset

1 . A method comprising:

receiving, at a headset worn by a user, an input from the user to register objects in a local area surrounding the headset for subsequent detection;

determining, by the headset, a gaze direction of the user in the local area based on captured information describing eyes of the user;

identifying an object in the local area based on the determined gaze direction of the user;

capturing images of the identified object from one or more imaging devices included in the headset and configured to capture images of the local area;

storing the captured images of the identified object in association with a label corresponding to the identified object;

determining, by the headset without input from the user, a first bounding box comprising captured images of the objects in the local area; and

determining the identified object included in the first bounding box that comprises a location of the gaze direction of the user.

2 . The method of claim 1 , wherein identifying the object in the local area based on the determined gaze direction of the user comprises:

identifying one or more regions of the images of the local area captured by the one or more imaging devices as candidate objects;

identifying regions in a three-dimensional model of the local area corresponding to the candidate objects; and

identifying the object corresponding to a candidate object within a region in the three-dimensional model of the local area to which the gaze direction of the user is directed.

3 . The method of claim 2 , wherein identifying the object corresponding to a candidate object within a region in the three-dimensional model of the local area to which the gaze direction of the user is directed comprises:

determining, by the headset without input from the user, a second bounding box corresponding to the candidate objects in the three-dimensional model of the local area; and

identifying a candidate object included in the second bounding box that comprises a location of the gaze direction of the user.

4 . The method of claim 3 , wherein identifying the candidate object included in the bounding box that comprises the location of the gaze direction of the user comprises:

identifying the candidate object included in second bounding box comprising the location of the gaze direction of the user for at least a threshold amount of time.

5 . The method of claim 2 , wherein identifying one or more regions of images of the local area captured by the one or more imaging devices as candidate objects comprises:

applying a category classifier to the images of the local area, the category classifier identifying regions of the image comprising at least one object.

6 . The method of claim 1 , further comprising:

receiving the label corresponding to the identified object from the user after capturing the images of the identified object.

7 . The method of claim 1 , further comprising:

receiving the label corresponding to the identified object from the user after identifying the object and before capturing the images of the identified object.

8 . The method of claim 1 , wherein capturing the images of the identified object from the imaging devices included in the headset and configured to capture the images of the local area comprises:

capturing a plurality of images of the identified object, the plurality of images corresponding to a different position of the identified object relative to the imaging devices.

9 . The method of claim 1 , further comprising:

training an instance classifier to detect the identified object within the images of the local area based on the captured images of the identified object.

10 . The method of claim 9 , further comprising:

displaying an indication to the user that the instance classifier has been trained via a display element of the headset.

11 . A headset comprising:

a frame;

one or more display elements coupled to the frame, the one or more display elements configured to generate image light for presentation to a user;

one or more imaging devices coupled to the frame, the one or more imaging devices configured to capture images of a local area surrounding the frame;

an eye tracking unit configured to determine a gaze direction of the user based on captured information describing eyes of the user; and

an object registration module comprising a processor and a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by the processor, cause the headset to:

receive an input from the user to register objects in the local area surrounding the headset for subsequent detection;

identify an object in the local area based on the determined gaze direction of the user;

capture images of the identified object from the one or more imaging devices;

store the captured images of the identified object in association with a label corresponding to the identified object;

determine without input from the user, a first bounding box comprising captured images of the objects in the local area; and

determine the identified object included in the first bounding box that comprises a location of the gaze direction of the user.

12 . The headset of claim 11 , wherein identify the object in the local area based on the determined gaze direction of the user comprises:

identify one or more regions of the images of the local area captured by the one or more imaging devices as candidate objects;

identify regions in a three-dimensional model of the local area corresponding to the candidate objects; and

identify the object corresponding to a candidate object within a region in the three-dimensional model of the local area to which the gaze direction of the user is directed.

13 . The headset of claim 12 , wherein identify the object corresponding to a candidate object within a region in the three-dimensional model of the local area to which the gaze direction of the user is directed comprises:

determine, by the headset without input from the user, a second bounding box corresponding to the candidate objects in the three-dimensional model of the local area; and

identify a candidate object included in the second bounding box that comprises a location of the gaze direction of the user.

14 . The headset of claim 13 , wherein identify the candidate object included in the bounding box that comprises the location of the gaze direction of the user comprises:

identify a candidate object included in the second bounding box

comprising the location of the gaze direction of the user for at least a threshold amount of time.

15 . The headset of claim 12 , wherein identify one or more regions of images of the local area captured by the one or more imaging devices as candidate objects comprises:

apply a category classifier to the images of the local area, the category classifier identifying regions of the image comprising at least one object.

16 . The headset of claim 11 , wherein when the one or more processors further execute the instructions, the headset is configured to:

receive the label corresponding to the identified object from the user after capturing the images of the identified object.

17 . The headset of claim 11 , wherein when the one or more processors further execute the instructions, the headset is configured to:

receive the label corresponding to the identified object from the user after identifying the object and before capturing the images of the identified object.

18 . The headset of claim 11 , wherein capture the images of the identified object from the one or more imaging devices included in the headset and configured to capture the images of the local area comprises:

capture a plurality of images of the identified object, each image of the plurality of images corresponding to a different position of the identified object relative to the one or more imaging devices.

19 . A non-transitory computer-readable medium storing instructions that, when executed, cause:

receiving, at a headset worn by a user, an input from the user to register objects in a local area surrounding the headset for subsequent detection;

determining, by the headset, a gaze direction of the user in the local area based on captured information describing eyes of the user;

identifying an object in the local area based on the determined gaze direction of the user;

capturing images of the identified object from one or more imaging devices included in the headset and configured to capture images of the local area;

storing the captured images of the identified object in association with a label corresponding to the identified object;

determining, by the headset without input from the user, a first bounding box comprising captured images of the objects in the local area; and

determining the identified object included in the first bounding box that comprises a location of the gaze direction of the user.

20 . The non-transitory computer-readable medium of claim 19 , wherein identifying the object in the local area based on the determined gaze direction of the user comprises:

identifying one or more regions of the images of the local area captured by the one or more imaging devices as candidate objects;

identifying regions in a three-dimensional model of the local area corresponding to the candidate objects; and

identifying the object corresponding to a candidate object within a region in the three-dimensional model of the local area to which the gaze direction of the user is directed.