Systems, non-transitory computer readable mediums, and methods for spectacle removal and virtual try-on

1 . A system comprising:

a computing device including at least one processor and communicatively coupled to a camera, the computing device is configured to, in real-time and in response to receiving a request:

capture at least one image via the camera of a user;

detect whether a first object is present within the at least one image, wherein the first object is an object that, when worn by the user, is required by the user to properly view a user interface of the computing device, and wherein the first object is selectively removable from the user such that when the first object is removed from the user, the user interface becomes incomprehensible to the user;

if the first object is detected within the at least one image, segment each pixel of a first plurality of pixels within the at least one image, each pixel of the first plurality of pixels corresponding to a pixel within the at least one image having a likelihood the pixel includes a part of the first object;

based on a segmentation of the first plurality of pixels of the first object, inpaint a second plurality of pixels of the at least one image to produce at least one inpainted images, each pixel of the second plurality of pixels corresponding to a paired pixel of the first plurality of pixels where the likelihood of the paired pixel is above a threshold value;

based on producing the at least one inpainted image, generate an augmented image by superimposing a selected second object over the at least one inpainted image;

if the first object is not detected within the at least one image, skip the segmenting and the inpainting to generate the augmented image by superimposing the selected second object over the at least one image; and

display the augmented image to the user on the user interface of the computing device, the augmented image configured to be properly viewed by the user utilizing the first object.

2 . The system of claim 1 , wherein the first plurality of pixels includes more pixels than the second plurality of pixels.

3 . The system of claim 1 , wherein the first plurality of pixels includes fewer pixels than the second plurality of pixels.

4 . The system of claim 1 , wherein the first plurality of pixels and the second plurality of pixels are the same.

5 . The system of claim 1 , wherein the computing device is configured to:

capture a real-time video via the camera, the real-time video including the at least one image and at least one subsequent image, the at least one subsequent image being within a subset of a number of frames of the real-time video;

fill a third plurality of pixels in the at least one subsequent image, each pixel of the third plurality of pixels corresponding to a paired pixel of the second plurality of pixels;

based on the filling of the third plurality of pixels in the at least one subsequent image, generate an augmented subsequent image by superimposing the selected second object over the at least one subsequent image; and

display the augmented subsequent image on the user interface of the computing device.

6 . The system of claim 5 , wherein the subset of the number of frames is based, at least in part, on computational capabilities of the computing device.

7 . The system of claim 1 , wherein the computing device is configured to, prior to generating the augmented image, reduce spurious noise by blending a previous image with a present image, wherein the blending includes performing a warping operation, a binarization operation, and a pixel-wise multiplication to transform a previous inpainting result prior to mixing with a current inpainting result.

8 . The system of claim 1 , wherein the computing device is configured, based on each pixel of the first plurality of pixels having the likelihood below a bottom threshold, to generate the augmented image by superimposing the selected second object over the at least one image.

9 . The system of claim 1 , further comprising a remote database configured to store a set of objects including the selected second object, wherein the computing device is configured to obtain the selected second object from the remote database.

10 . The system of claim 1 , wherein the selected second object is indicated in the request.

11 . The system of claim 1 , wherein the computing device is a mobile computing device.

12 . The system of claim 1 , wherein the first object is a set of spectacles.

13 . The system of claim 1 , wherein the part of the first object is a frame of a set of spectacles.

14 . The system of claim 1 , wherein the computing device is configured to:

detect, within the at least one image, the first object by implementing a first neural network;

segment the first plurality of pixels of the first object by determining, for each respective pixel of the first plurality of pixels, the likelihood the respective pixel includes the part of the first object using a second neural network; and

inpaint the second plurality of pixels of the at least one image corresponding to the paired pixel of the first plurality of pixels where the likelihood of the paired pixel is above the threshold value by implementing a third neural network.

15 . The system of claim 1 , wherein the first object is configured to modify vision of the user in viewing the user interface.

16 . The system of claim 15 , wherein the first object is a pair of spectacles.

17 . A method comprising:

in response to receiving a request, capturing, via a computing device including at least one processor and communicatively coupled to a camera, at least one image via the camera;

detecting whether a first object is present within the at least one image, wherein the first object is an object that, when worn by a user, is required by the user to properly view a user interface of the computing device, and wherein the first object is selectively removable from the user such that when the first object is removed from the user, the user interface becomes incomprehensible to the user;

if the first object is detected within the at least one image, segmenting a first plurality of pixels within the at least one image, each pixel of the first plurality of pixels corresponding to a pixel within the at least one image having a likelihood that the pixel includes a part of the first object;

based on a segmentation of the first plurality of pixels of the first object, inpainting a second plurality of pixels of the at least one image to produce at least one inpainted image, each location of the second plurality of pixels corresponding to a paired location of the first plurality of pixels where the likelihood of the paired pixel is above a threshold value;

based on a production of the at least one inpainted image, generating an augmented image by superimposing a selected second object over the at least one inpainted image;

if the first object is not detected within the at least one image, skipping the segmenting and the inpainting to generate the augmented image by superimposing the selected second object over the at least one image; and

displaying the augmented image on the user interface of the computing device, the augmented image configured to be properly viewed by the user utilizing the first object.

18 . The method of claim 17 , wherein the detecting is performed by a spectacle detection module, the segmenting is performed by a segmenting module, and the inpainting is performed by an inpainting module.

19 . The method of claim 18 , wherein the spectacle detection module, the segmenting module, and the inpainting module comprise separate neural networks.

20 . The method of claim 18 , wherein the spectacle detection module, the segmenting module, and the inpainting module are separate modules, and wherein breaking a processing pipeline into the spectacle detection module, the segmenting module, and the inpainting module processes the at least one image in real using less memory compared to using a single neural network.

21 . The method of claim 18 , wherein the inpainting module inpaints fewer than all frames of a real-time video based on computational capabilities of the computing device, and wherein on frames on which the inpainting is not performed, segmented pixels are filled with information from a most recently inpainted image.

22 . The method of claim 21 , wherein a number of frames that are inpainted is based on a processing capability of the computing device.

23 . The method of claim 18 , further comprising:

determining computational capabilities of the computing device;

capturing a real-time video including a first plurality of frames including the at least one image via the camera; and

filling a third plurality of pixels in a second plurality of frames, each pixel of the third plurality of pixels corresponding to a paired pixel of the second plurality of pixels, wherein the second plurality of frames is based on the computational capabilities of the computing device.

24 . The method of claim 23 , wherein the second plurality of frames is different from the first plurality of frames.

25 . The method of claim 18 , further comprising, prior to generating the augmented image, blending a previous image with the at least one inpainted image to reduce spurious noise.

26 . The method of claim 18 , further comprising, in response to detecting an absence of the plurality of pixels of the first object, skipping the segmenting and the inpainting to generate the augmented image.

27 . The method of claim 17 , further comprising:

segmenting the first plurality of pixels of the first object by determining, for each respective pixel of the first plurality of pixels, the likelihood the respective pixel includes the part of the first object using a second neural network; and

training the second neural network using a dataset, the dataset including images which were manually segmented.

28 . A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause a device to in real-time, perform operations comprising:

in response to receiving a request, capturing, via a computing device including the at least one processor and communicatively coupled to a camera, at least one image via the camera;

detecting whether a first object is present within the at least one image, wherein the first object is an object that, when worn by a user, is required by the user to properly view a user interface of the computing device, and wherein the first object is selectively removable from the user such that when the first object is removed from the user, the user interface becomes incomprehensible to the user;

if the first object is detected within the at least one image, segmenting a first plurality of pixels within the at least one image, each pixel of the first plurality of pixels corresponding to a pixel within the at least one image having a likelihood that the pixel includes a part of the first object;

based on a segmentation of the first plurality of pixels of the first object, inpainting a second plurality of pixels of the at least one image to produce at least one inpainted image, each pixel of the second plurality of pixels corresponding to a paired pixel of the first plurality of pixels where the likelihood of the paired pixel is above a threshold value;

based on a production of the at least one inpainted image, generating an augmented image by superimposing a selected second object over the at least one inpainted image;

if the first object is not detected within the at least one image, skipping the segmenting and the inpainting to generate the augmented image by superimposing the selected second object over the at least one image; and

displaying the augmented image on the user interface of the computing device, the augmented image configured to be properly viewed by the user utilizing the first object.

29 . The non-transitory computer readable medium of claim 28 , wherein the detecting is performed by a spectacle detection module, the segmenting is performed by a segmenting module, and the inpainting is performed by an inpainting module.

30 . The non-transitory computer readable medium of claim 29 , wherein the operations include:

determining computational capabilities of the device;

capturing a real-time video including a first plurality of frames including the at least one image via the camera; and

filling a third plurality of pixels in a second plurality of frames, each pixel of the third plurality of pixels corresponding to a paired pixel of the second plurality of pixels, wherein the second plurality of frames is based on the computational capabilities of the device.

31 . The non-transitory computer readable medium of claim 29 , wherein the operations include blending a previous image with the at least one inpainted image to reduce spurious noise prior to generating the augmented image.