Systems and methods for motion correction in synthetic images

1. A system for generating synthetic video comprising:

one or more memory units storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising:

generating a static background image;

determining a location of a reference edge in the static background image;

determining a perspective of an observation point;

generating, based on the static background image, synthetic difference images comprising a respective synthetic object movement edge;

determining, for individual synthetic difference images, a location of the respective synthetic object movement edge;

generating adjusted difference images corresponding to the individual synthetic difference images, the adjusted difference images being based on:

the corresponding individual synthetic difference images,

the determined locations of the respective synthetic object movement edges of the corresponding individual synthetic difference images,

the perspective of the observation point, and

the location of the reference edge; and

generating texturized images based on the adjusted difference images.

2. The system of claim 1 , wherein generating adjusted difference images comprises using a method of gradient domain processing and using a method of homography.

3. The system of claim 1 , wherein the one or more processors are configured to execute the instructions to perform operations further comprising:

receiving video data comprising a sequence of images;

generating background difference images based on the sequence of images; and

generating merged difference images based on the adjusted difference images and the background difference images,

wherein:

generating the static background image is based on the sequence of images, and

generating the texturized images is based on the merged difference images.

4. The system of claim 3 , wherein generating merged difference images comprises:

determining, for individual adjusted difference images, a location of a respective adjusted object movement edge;

determining, for individual background difference images, a location of a respective background movement object edge; and

merging the individual adjusted difference images with the individual background difference images based on:

the locations of the respective adjusted object movement edges of the individual adjusted difference images, and

the locations of the respective background movement object edges of the individual background difference images.

5. The system of claim 3 , wherein the one or more processors are configured to execute the instructions to perform operations further comprising generating a sequence of normalized images based on the sequence of images and the static background image, wherein generating the background difference images comprises applying a method of image subtraction to the normalized sequence of images.

6. The system of claim 1 , wherein generating individual adjusted difference images comprises scaling the respective synthetic object movement edge of the corresponding synthetic difference image.

7. The system of claim 1 , wherein generating individual adjusted difference images comprises layering a synthetic object movement edge and a background object edge.

8. The system of claim 1 , wherein the one or more processors are configured to execute the instructions to perform operations further comprising:

determining luminance values of the static background image; and

estimating a location and brightness of a light source based on the luminance values of the static background image,

wherein generating adjusted difference images comprises adjusting luminance values of the corresponding individual synthetic difference images based on the location and brightness of the light source.

9. The system of claim 8 , wherein the one or more processors are configured to execute the instructions to perform operations further comprising:

receiving video data comprising a sequence of images, and

generating a converted sequence of images based on the received sequence of images, the converted sequence of images having luminance channel data,

wherein:

generating the static background image is based on the converted sequence of images, and

determining luminance values of the static background image is based on the luminance channel data of the converted sequence of images.

10. The system of claim 1 , wherein determining the location of a reference edge in the static background image comprises:

identifying a plurality of edges in the static background image;

identifying a reference object based on the plurality of edges;

identifying a reference edge of the reference object;

determining a plurality of background edge distances between with the reference edge and the plurality of edges; and

determining a location of the reference edge in the static background image based on the plurality of background edge distances.

11. The system of claim 10 , wherein the observation point is a first observation point, the adjusted difference images are first adjusted difference images corresponding to a first subset of the individual synthetic difference images, and the texturized images are first texturized images, the operations further comprising:

determining a perspective of a second observation point;

generating second adjusted difference images corresponding to a second subset of the individual synthetic difference images, the second adjusted difference images being based on:

a corresponding synthetic difference image,

the location of the respective synthetic object movement edge of the corresponding synthetic difference image,

the perspective of the second observation point, and

the location of the reference edge; and

generating second texturized images based on the second adjusted difference images.

12. The system of claim 10 , the operations further comprising estimating a size of the reference object, wherein determining the plurality of background edge distances is based on the estimated size of the reference object.

13. The system of claim 10 , wherein the one or more processors are configured to execute the instructions to perform operations further comprising generating a three-dimensional model of the static background image based on the plurality of background edge distances, wherein determining the location of the reference edge is based on the three-dimensional model.

14. The system of claim 1 , wherein the one or more processors are configured to execute the instructions to perform operations further comprising generating a three-dimensional spatial model of the individual synthetic difference images based on the perspective of the observation point and the location of the reference edge, wherein the adjusted difference images are based on the three-dimensional spatial model.

15. The system of claim 1 , wherein the synthetic object movement edge is associated with a shadow of a synthetic object, and wherein the generating adjusted difference images comprises determining transparency values associated with the shadow.

16. The system of claim 1 , wherein generating synthetic difference images is performed using a machine learning model comprising at least one of an autoencoder model, a recurrent neural network model, or a convolutional neural network model.

17. The system of claim 1 , wherein generating adjusted difference images is performed using a machine learning model comprising at least one of an autoencoder model, a recurrent neural network model, or a convolutional neural network model.

18. The system of claim 1 , wherein generating merged difference images is performed using a machine learning model comprising at least one of an autoencoder model, a recurrent neural network model, or a convolutional neural network model.

19. A method of generating synthetic video comprising:

generating a static background image;

determining a location of a reference edge in the static background image;

determining the perspective of an observation point;

generating, based on the static background image, synthetic difference images comprising a respective synthetic object movement edge;

determining, for individual synthetic difference images, a location of the respective synthetic object movement edge;

generating adjusted difference images corresponding to the individual synthetic difference images, the adjusted difference images being based on:

the corresponding individual synthetic difference images,

the locations of the respective synthetic object movement edges of the corresponding individual synthetic difference images,

the perspective of the observation point, and

the location of the reference edge; and

generating texturized images based on the adjusted difference images.

20. A system for generating synthetic video comprising:

one or more memory units storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising:

generating a static background image;

identifying a plurality of edges of the static background image;

generating a three-dimensional model of the static background image using homography, the three-dimensional model being based on the plurality of edges;

determining a perspective of an observation point;

determining luminance values of the static background image;

estimating a location and brightness of a light source based on the luminance values;

generating, based on the static background image, synthetic difference images comprising a respective plurality of synthetic object movement edges;

determining, for individual synthetic difference images, a plurality of locations of the plurality of respective synthetic object movement edges;

generating, using gradient domain processing and homography, adjusted difference images corresponding to the individual synthetic difference images, the adjusted difference images being based on:

a corresponding synthetic difference image,

the locations of the respective plurality of synthetic object movement edges of the corresponding synthetic difference image,

the perspective of the observation point,

the three-dimensional model of the static background image, and

the location and brightness of the light source; and

generating texturized images based on the adjusted difference images.