Apparatus and methods for augmenting vision with region-of-interest based processing
Systems, apparatus, and methods for augmenting vision with region-of-interest based processing. In one specific example, smart glasses may use an eye-tracking camera to monitor the user's gaze and determine the user's gaze point. When triggered, the camera assembly captures a high-resolution image. The high-resolution image may be cropped to a much smaller region-of-interest (ROI) image based on computer-vision analysis of the user's gaze point. For example, if the smart glasses detect a human face at the gaze point, then the ROI is cropped to the human face. In this manner, the smart glasses may leverage specific capabilities of the smart glasses to augment the user experience; for example, telephoto lenses provide long distance vision, or computer-assisted search may direct the user to interesting activity. Other aspects may include e.g., external database assisted operation and/or ongoing cataloging throughout the day.
1 . A smart glasses apparatus, comprising:
a physical frame;
a first camera assembly comprising a telephoto lens;
a second camera assembly;
a processor; and
a non-transitory computer-readable medium comprising instructions that, when executed by the processor, cause the smart glasses apparatus to:
determine a distance to a target object;
select an appropriate camera assembly from the first camera assembly and the second camera assembly based on the distance;
capture an image of the target object with the appropriate camera assembly at a capture resolution; and
digitally zoom a portion of the image from the capture resolution to a display resolution, where an amount of digital zoom is based on a duration of gaze fixation.
2 . The smart glasses apparatus of claim 1 , further comprising a display, where the capture resolution is greater than 60 pixels per degree.
3 . The smart glasses apparatus of claim 2 , where the display is characterized by the display resolution less than or equal to 60 pixels per degree.
4 . The smart glasses apparatus of claim 1 , where the portion of the image is cropped to the target object.
5 . The smart glasses apparatus of claim 1 , where a first portion of the image cropped to the target object is sent to a companion device for display.
6 . The smart glasses apparatus of claim 5 , where a second portion of the image adjacent to the first portion is sent to the companion device to enable active panning.
7 . The smart glasses apparatus of claim 1 , where an amount of digital zoom is also based on a hand gesture.
8 . The smart glasses apparatus of claim 1 , where the instructions, when executed by the processor, further cause the smart glasses apparatus to:
determining a gaze point of a wearer of the smart glasses apparatus; and
determining the target object based on the gaze point of the wearer.
9 . A smart glasses apparatus, comprising:
a physical frame;
an inward-facing camera assembly;
an outward-facing camera assembly comprising a telephoto lens;
a processor; and
a non-transitory computer-readable medium comprising instructions that, when executed by the processor, cause the smart glasses apparatus to:
track eye movement via the inward-facing camera assembly;
determine gaze vectors based on the eye movement comprising a first gaze vector of a plurality of gaze vectors determined based on a first location of a first pupil of a first eye relative to a first retina of the first eye;
determine a gaze point based on the gaze vectors;
determine a target object based on the gaze point;
determine a distance of the target object; and
capture an image via the telephoto lens of the outward-facing camera assembly based on the distance.
10 . The smart glasses apparatus of claim 9 , where:
a second gaze vector of the gaze vectors is determined from a second location of a second pupil of a second eye relative to a second retina of the second eye, and
determining the distance comprises determining a convergence of the first gaze vector and the second gaze vector.
11 . The smart glasses apparatus of claim 9 , where the instructions, when executed by the processor, further cause the smart glasses apparatus to:
detect a face at the gaze point via the image; and
crop a region-of-interest image from the image around the face.
12 . The smart glasses apparatus of claim 11 , further comprising a heads-up display, where the instructions, when executed by the processor, further cause the smart glasses apparatus to:
transfer the region-of-interest image to a companion device;
receive identification information associated with the face in the region-of-interest image from the companion device; and
display the identification information on the heads-up display.
13 . The smart glasses apparatus of claim 9 , where the outward-facing camera assembly further comprises a periscope prism configured to divert light perpendicular to a capture direction.
14 . The smart glasses apparatus of claim 9 , where the instructions, when executed by the processor, further cause the smart glasses apparatus to:
perform a low-resolution image capture via the outward-facing camera assembly; and
perform facial detection based on the low-resolution image capture.
15 . A method of operating a smart glasses apparatus, comprising:
determining a target object based on a first image captured via a first camera assembly of a plurality of camera assemblies;
determining a distance to the target object;
selecting a second camera assembly from the plurality of camera assemblies based on the distance; and
capturing a second image of the target object with the selected second camera assembly.
16 . The method of claim 15 , further comprising:
identifying the target object based on the second image;
determining historical information about the target object based on identifying the target object; and
displaying the historical information via the smart glasses apparatus.
17 . The method of claim 15 , where selecting the selected second camera assembly is based on the distance being greater than a threshold distance.
18 . The method of claim 15 , where:
the first camera assembly comprises a wide-angle lens,
the second camera assembly comprises a telephoto lens characterized by a smaller field-of-view than the wide-angle lens, and
a first-field of view of the first image is greater than a second field-of-view of the second image.
19 . A smart glasses apparatus, comprising:
a physical frame;
an inward-facing camera assembly;
an outward-facing camera assembly comprising a telephoto lens;
a processor; and
a non-transitory computer-readable medium comprising instructions that, when executed by the processor, cause the smart glasses apparatus to:
track eye movement via the inward-facing camera assembly;
determine gaze vectors based on the eye movement;
determine a gaze point based on the gaze vectors;
determine a target object based on the gaze point;
determine a distance of the target object;
capture an image via the telephoto lens of the outward-facing camera assembly based on the distance;
detect a face at the gaze point via the image;
crop a region-of-interest image from the image around the face;
transfer the region-of-interest image to a companion device; and
receive identification information associated with the face in the region-of-interest image from the companion device.
20 . The smart glasses apparatus of claim 19 , where the outward-facing camera assembly further comprises a periscope prism configured to divert light perpendicular to a capture direction.
21 . The smart glasses apparatus of claim 19 , where each of the gaze vectors are determined based on a relative location of a pupil and a retina.
22 . The smart glasses apparatus of claim 21 , where the gaze point is determined based on a convergence of the gaze vectors in space.
23 . The smart glasses apparatus of claim 19 , where the instructions, when executed by the processor, further cause the smart glasses apparatus to determine a gaze of a user is fixated.
24 . The smart glasses apparatus of claim 23 , where the instructions, when executed by the processor, further cause the smart glasses apparatus to determine the gaze point is responsive to determining the gaze is fixated.
25 . The smart glasses apparatus of claim 19 , further comprising a heads-up display, where the instructions, when executed by the processor, further cause the smart glasses apparatus to display the identification information on the heads-up display.