System and method to digitally replace objects in images or video

1. A method comprising:

receiving video data;

identifying a second object depicted in a video frame of the video data based on a comparison of the second object as depicted in the video frame to a set of three-dimensional object models;

determining whether to replace the second object in the video frame with a first object based on an object matching rule; and

in response to determining that the second object is to be replaced with the first object:

identifying a second visual property of the second object failing to meet a requirement;

manipulating a three-dimensional model of the first object to generate a representation of the first object that matches a visual property of the second object, wherein a third visual property of the representation of the first object satisfies the requirement; and

replacing the second object with the representation of the first object in the video frame.

2. The method of claim 1 , wherein the object matching rule comprises a threshold number of matching features of the first object and the second object, a user defined set of matching rules, a context rule, or any combination thereof.

3. The method of claim 2 , wherein the context rule specifies audio criteria, visual criteria, lighting criteria, texture criteria, geometric criteria, or any combination thereof.

4. The method of claim 3 , wherein the context rule specifies that the second object is not to be replaced with the representation of the first object when a particular third object is present in the video frame.

5. The method of claim 2 , wherein the matching features of the second object include color of the second object, size of the second object, shape of the second object, or any combination thereof.

6. The method of claim 1 , further comprising automatically performing pixel hallucination for missing pixels resulting from the replacement of the second object with the representation of the first object.

7. The method of claim 1 , further comprising computing the three-dimensional model of a portion of the first object based on a media content item that depicts the first object and storing the three-dimensional model in a database.

8. The method of claim 7 , wherein the media content item comprises a graphics file, a video file, an automatically crawled network accessible media content item, a user provided media content item, or any combination thereof.

9. The method of claim 7 , further comprising computing the three-dimensional model of the first object based on partial models of the first object, two-dimensional models of the first object, scene data depicting the first object, multi-view images of the first object, motion of the first object in a scene, or any combination thereof.

10. The method of claim 7 , further comprising extracting scene data associated with the first object, wherein the scene data includes data regarding visual content surrounding the first object in the media content item, audio content in the media content item, or any combination thereof.

11. The method of claim 10 , further comprising:

extracting the audio content via speech recognition; and

associating the first object with a particular scene based on the visual content and the audio content.

12. The method of claim 1 , wherein the visual property comprises shape, texture, shade, topology, or any combination thereof.

13. A system comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to perform operations including:

identifying a second object depicted in video frames of video data based on a comparison of the second object as depicted in the video frames to a set of three-dimensional object models;

determining whether to replace the second object in the video frames with a first object based on an object matching rule; and

in response to determining that the second object is to be replaced with the first object:

identifying a second visual property of the second object failing to meet a requirement;

manipulating a three-dimensional model of the first object to generate a representation of the first object that matches a visual property of the second object, wherein a third visual property of the representation of the first object satisfies the requirement; and

replacing the second object with the representation of the first object in the video frames.

14. The system of claim 13 , wherein the operations further include:

computing the three-dimensional model of a portion of the first object based on a media content item that depicts the first object; and

storing the three-dimensional model in the memory.

15. The system of claim 14 , wherein the media content item comprises a graphics file, a video file, an automatically crawled network accessible media content item, a user provided media content item, or any combination thereof.

16. The system of claim 13 , wherein the object matching rule comprises a threshold number of matching features of the first object and the second object.

17. The system of claim 13 , wherein the visual property comprises a bi-directional reflectance distribution function of the second object.

18. The system of claim 13 , wherein the requirement indicates a particular feature of a particular object is visible, and wherein the three-dimensional model is manipulated so that the particular feature of the first object is visible in the video frames.

19. A computer-readable storage device comprising instructions that, when executed by a processor, cause the processor to perform operations including:

identifying a second object depicted in a video frame of video data based on a comparison of the second object as depicted in the video frame to a set of three-dimensional object models;

determining whether to replace the second object in the video frame with a first object based on an object matching rule; and

in response to determining that the second object is to be replaced with the first object:

identifying a second visual property of the second object failing to meet a requirement;

manipulating a three-dimensional model of the first object to generate a representation of the first object that matches a visual property of the second object, wherein a third visual property of the representation of the first object satisfies the requirement; and

replacing the second object with the representation of the first object in the video frame.

20. The computer-readable storage device of claim 19 , wherein the operations further include:

computing the three-dimensional model of a portion of the first object based on a media content item that depicts the first object; and

storing the three-dimensional model in a memory.