Granted Patent B2

US 12,322,166 · App. 17/561,591 · Granted Jun 3, 2025

Inventor: Kyle Goodrich (Venice, CA)

Assignee: Snap Inc.

G06V10/82G06V10/25G06V10/764

Abstract

Claims (51)

1. A method, comprising:

receiving first image data captured by a camera of an eyewear device;

detecting, using a machine learning model, a representation of a display screen of an electronic device in the first image data;

selecting at least a portion of the representation of the display screen of the electronic device;

adjusting, while the representation of the display screen of the electronic device is detected in a field of view of a user while using the eyewear device, a visual appearance of the portion of the representation of the display screen, wherein adjusting, while the representation of the display screen of the electronic device is detected in the field of view of the user while using the eyewear device, the visual appearance of the portion of the representation of the display screen comprises at least:

generating a message indicating that a screen has been detected, the message being overlaid on a current object that was detected as the display screen of the electronic device,

generating a representation of a bounding box around the representation of the display screen, the representation of the bounding box including a set of pixels corresponding to at least four sides, and

modifying the set of pixels from the representation of the bounding box by at least changing a first color value of a first pixel to a second color value, wherein the second color value is different than the first color value; and

causing display of the adjusted visual appearance using a display system of the eyewear device.

2. The method of claim 1 , wherein detecting, using the machine learning model, the representation of the display screen comprises:

performing an object detection process on the first image data to determine the representation of the display screen, wherein the machine learning model determines a prediction of the representation of the display screen being included in the first image data.

3. The method of claim 2 , wherein the machine learning model comprises a convolutional neural network (CNN).

4. The method of claim 3 , wherein the CNN determines a region of interest where the representation of the display screen is present, the region of interest comprising a portion of the first image data.

5. The method of claim 4 , wherein the region of interest comprises a candidate bounding box including the representation of the display screen.

6. The method of claim 4 , wherein the CNN generates a feature map based on the region of interest and provides a vector of values corresponding to the feature map as an output, the vector including respective values describing contents of the region of interest.

7. The method of claim 6 , wherein the feature map includes a set of features, and a classifier model generates a classification of the set of features from the feature map, the classification comprising a display screen object.

8. The method of claim 1 , wherein adjusting the visual appearance of the portion of the representation of the display screen further comprises:

modifying a second set of pixels corresponding to the representation of the display screen by at least changing a first luminosity value of a third pixel to a second luminosity value, wherein the second luminosity value is greater than the first luminosity value.

9. The method of claim 1 , further comprising:

receiving second image data captured by the camera, the second image data being received upon movement of the camera of the eyewear device based on a change of a position of a head of a user wearing the eyewear device;

detecting that the representation of the display screen is no longer present in the second image data; and

modifying the second image data to indicate that the representation of the display screen is no longer present, the modifying comprising reducing a portion of the second image data to decrease at least one luminosity value of a pixel from the second image data.

10. A system comprising:

a processor, and

a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:

receiving first image data captured by a camera of an eyewear device;

detecting, using a machine learning model, a representation of a display screen of an electronic device in the first image data;

selecting at least a portion of the representation of the display screen of the electronic device;

adjusting, while the representation of the display screen of the electronic device is detected in a field of view of a user while using the eyewear device, a visual appearance of the portion of the representation of the display screen, wherein adjusting, while the representation of the display screen of the electronic device is detected in the field of view of the user while using the eyewear device, the visual appearance of the portion of the representation of the display screen comprises at least:

generating a message indicating that a screen has been detected, the message being overlaid on a current object that was detected as the display screen of the electronic device,

generating a representation of a bounding box around the representation of the display screen, the representation of the bounding box including a set of pixels corresponding to at least four sides, and

modifying the set of pixels from the representation of the bounding box by at least changing a first color value of a first pixel to a second color value, wherein the second color value is different than the first color value; and

causing display of the adjusted visual appearance using a display system of the eyewear device.

11. The system of claim 10 , wherein detecting, using the machine learning model, the representation of the display screen comprises:

performing an object detection process on the first image data to determine the representation of the display screen, wherein the machine learning model determines a prediction of the representation of the display screen being included in the first image data.

12. The system of claim 11 , wherein the machine learning model comprises a convolutional neural network (CNN).

13. The system of claim 12 , wherein the CNN determines a region of interest where the representation of the display screen is present, the region of interest comprising a portion of the first image data.

14. The system of claim 13 , wherein the region of interest comprises a candidate bounding box including the representation of the display screen.

15. The system of claim 13 , wherein the CNN generates a feature map based on the region of interest and provides a vector of values corresponding to the feature map as an output, the vector including respective values describing contents of the region of interest.

16. The system of claim 15 , wherein the feature map includes a set of features, and a classifier model generates a classification of the set of features from the feature map, the classification comprising a display screen object.

17. The system of claim 10 , wherein adjusting the visual appearance of the portion of the representation of the display screen further comprises:

modifying a second set of pixels corresponding to the representation of the display screen by at least changing a first luminosity value of a third pixel to a second luminosity value, wherein the second luminosity value is greater than the first luminosity value.

18. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

receiving first image data captured by a camera of an eyewear device;

detecting, using a machine learning model, a representation of a display screen of an electronic device in the first image data;

selecting at least a portion of the representation of the display screen of the electronic device;

adjusting, while the representation of the display screen of the electronic device is detected in a field of view of a user while using the eyewear device, a visual appearance of the portion of the representation of the display screen, wherein adjusting, while the representation of the display screen of the electronic device is detected in the field of view of the user while using the eyewear device, the visual appearance of the portion of the representation of the display screen comprises at least:

generating a message indicating that a screen has been detected, the message being overlaid on a current object that was detected as the display screen of the electronic device,

generating a representation of a bounding box around the representation of the display screen, the representation of the bounding box including a set of pixels corresponding to at least four sides, and

modifying the set of pixels from the representation of the bounding box by at least changing a first color value of a first pixel to a second color value, wherein the second color value is different than the first color value; and

causing display of the adjusted visual appearance using a display system of the eyewear device.

Assignments (1)

Continuity (2)

References Cited (11)