Bystander and attached shadow removal

1 . A computer-implemented method comprising:

deriving a bystander mask from an image by analyzing the image with a bystander segmentation model, wherein the image depicts a bystander and the bystander mask identifies a plurality of first pixels in the image that are associated with the bystander;

providing the image and the bystander mask as input to a shadow segmentation model,

wherein the shadow segmentation model is trained with a training dataset of a plurality of training images, a segmentation mask associated with a person in each training image of the plurality of training images, and a groundtruth shadow mask for each training image of the plurality of training images, and

wherein the plurality of training images are generated by including a candidate image in the plurality of training images that is selected from a group of the candidate image including the person that is associated with a body bounding box that is less than a threshold overlap value with a vehicle bounding box, the candidate image including the person that is associated with a body bounding box that has an aspect ratio that is less than a threshold aspect ratio, the candidate image including the segmentation mask that is outside of the vehicle bounding box, the candidate image including the person that is associated with a body bounding box intersecting with a mobile object bounding box less than a threshold overlap value, the candidate image including the groundtruth shadow mask that is between a threshold first size and a threshold second size, and combinations thereof;

outputting, with the shadow segmentation model, a shadow mask for the bystander, wherein the shadow mask identifies a plurality of second pixels in the image that are associated with a shadow of the bystander; and

modifying the image to update pixel values of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are erased from the image.

2 . The computer-implemented method of claim 1 , further comprising determining a likelihood of existence of the shadow using a shadow classifier model based on the image and the bystander mask, wherein providing the image and the bystander mask as input to the shadow segmentation model, outputting the shadow mask for the bystander, and modifying the image are performed if the likelihood of existence of the shadow meets a threshold.

3 . The computer-implemented method of claim 1 , wherein modifying the image comprises applying an inpainting technique to update the pixel values of the plurality of first pixels and the plurality of second pixels.

4 . The computer-implemented method of claim 3 , wherein the inpainting technique is performed by an inpainter model.

5 . The computer-implemented method of claim 1 , further comprising, prior to the modifying, merging the bystander mask and the shadow mask.

6 . The computer-implemented method of claim 1 , wherein the shadow segmentation model includes a convolutional neural network (CNN) and training the shadow segmentation model comprises, for each training image of the plurality of training images:

obtaining a predicted shadow mask based on the training image and the segmentation mask associated with the person in the training image;

calculating a loss value based on a comparison of the predicted shadow mask and the groundtruth shadow mask for the training image; and

updating a weight of one or more nodes of the CNN based on the loss value.

7 . The computer-implemented method of claim 1 , wherein one or more training images in the plurality of training images are associated with an empty groundtruth shadow mask.

8 . The computer-implemented method of claim 1 , wherein one or more training images in the plurality of training images are generated from a series of candidate images of a scene that includes the person captured at different times by:

generating a clean-plate image that contains static elements of the scene;

comparing each candidate image in the series of candidate images to the clean-plate image to identify dynamic parts of the scene;

generating a plurality of segmentation masks for the person in the series of candidate images; and

determining from the plurality of segmentation masks that pixels for the groundtruth shadow mask are adjacent to pixels corresponding to the segmentation mask of the plurality of segmentation masks.

9 . The computer-implemented method of claim 1 , wherein the shadow segmentation model includes a convolutional neural network (CNN) and training the shadow segmentation model comprises, for each training image of the plurality of training images:

obtaining a predicted shadow mask based on the training image, wherein one or more pixels of the training image that are associated with the person have values associated with a particular color that does not occur in natural images;

calculating a loss value based on a comparison of the predicted shadow mask and the groundtruth shadow mask for the training image; and

updating a weight of one or more nodes of the CNN based on the loss value.

10 . The computer-implemented method of claim 1 , wherein the image depicts two or more bystanders, and wherein the providing, outputting, and modifying are performed for each of the two or more bystanders.

11 . The computer-implemented method of claim 1 , wherein the image is a single frame of a video.

12 . The computer-implemented method of claim 1 , wherein at least one pixel in the plurality of first pixels of the bystander mask and at least one pixel in the plurality of second pixels of the shadow mask are adjacent.

13 . A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:

deriving a bystander mask from an image by analyzing the image with a bystander segmentation model, wherein the image depicts a bystander and the bystander mask identifies a plurality of first pixels in the image that are associated with the bystander;

providing the image and the bystander mask as input to a shadow segmentation model,

wherein the shadow segmentation model is trained with a training dataset of a plurality of training images, a segmentation mask associated with a person in each training image of the plurality of training images, and a groundtruth shadow mask for each training image of the plurality of training images,

wherein the plurality of training images are generated by including a candidate image in the plurality of training images that is selected from a group of the candidate image including the person that is associated with a body bounding box that is less than a threshold overlap value with a vehicle bounding box, the candidate image including the person that is associated with a body bounding box that has an aspect ratio that is less than a threshold aspect ratio, the candidate image including the segmentation mask that is outside of the vehicle bounding box, the candidate image including the person that is associated with a body bounding box intersecting with a mobile object bounding box less than a threshold overlap value, the candidate image including the groundtruth shadow mask that is between a threshold first size and a threshold second size, and combinations thereof;

outputting, with the shadow segmentation model, a shadow mask for the bystander, wherein the shadow mask identifies a plurality of second pixels in the image that are associated with a shadow of the bystander; and

modifying the image to update pixel values of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are erased from the image.

14 . The non-transitory computer-readable medium of claim 13 , wherein the operations further include determining a likelihood of existence of the shadow using a shadow classifier model based on the image and the bystander mask, wherein providing the image and the bystander mask as input to the shadow segmentation model, outputting the shadow mask for the bystander, and modifying the image are performed if the likelihood of existence of the shadow meets a threshold.

15 . The non-transitory computer-readable medium of claim 13 , wherein modifying the image comprises applying an inpainting technique to update the pixel values of the plurality of first pixels and the plurality of second pixels.

16 . The non-transitory computer-readable medium of claim 13 , wherein the operations further include, prior to the modifying, merging the bystander mask and the shadow mask.

17 . A computing device comprising:

a processor; and

a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:

deriving a bystander mask from an image by analyzing the image with a bystander segmentation model, wherein the image depicts a bystander and the bystander mask identifies a plurality of first pixels in the image that are associated with the bystander;

providing the image and the bystander mask as input to a shadow segmentation model,

wherein the shadow segmentation model is trained with a training dataset of a plurality of training images, a segmentation mask associated with a person in each training image of the plurality of training images, and a groundtruth shadow mask for each training image of the plurality of training images, and

wherein the plurality of training images are generated by including a candidate image in the plurality of training images that is selected from a group of the candidate image including the person that is associated with a body bounding box that is less than a threshold overlap value with a vehicle bounding box, the candidate image including the person that is associated with a body bounding box that has an aspect ratio that is less than a threshold aspect ratio, the candidate image including the segmentation mask that is outside of the vehicle bounding box, the candidate image including the person that is associated with a body bounding box intersecting with a mobile object bounding box less than a threshold overlap value, the candidate image including the groundtruth shadow mask that is between a threshold first size and a threshold second size, and combinations thereof;

outputting, with the shadow segmentation model, a shadow mask for the bystander, wherein the shadow mask identifies a plurality of second pixels in the image that are associated with a shadow of the bystander; and

modifying the image to update pixel values of the plurality of first pixels and the plurality of second pixels such that the bystander and the shadow are erased from the image.

18 . The computing device of claim 17 , wherein the operations further include determining a likelihood of existence of the shadow using a shadow classifier model based on the image and the bystander mask, wherein providing the image and the bystander mask as input to the shadow segmentation model, outputting the shadow mask for the bystander, and modifying the image are performed if the likelihood of existence of the shadow meets a threshold.

19 . The computing device of claim 17 , wherein modifying the image comprises applying an inpainting technique to update the pixel values of the plurality of first pixels and the plurality of second pixels.

20 . The computing device of claim 17 , wherein the operations further include, prior to the modifying, merging the bystander mask and the shadow mask.