Methods and systems for pre-processing two-dimensional image files to be converted to three-dimensional image files
Disclosed herein are methods and systems of efficiently, effectively, and accurately preparing images for a 2D to 3D conversion process by pre-treating occlusions and transparencies in original 2D images. A single 2D image, or a sequence of images, is ingested, segmented into discrete elements, and the discrete elements are individually reconstructed. The reconstructed elements are then re-composited and ingested into a 2D to 3D conversion process.
1. A method of preparing a two-dimensional image prior to subjecting the two-dimensional image through a two-dimensional to three-dimensional image conversion process, the method comprising:
receiving, by a computing device, a two-dimensional digital image;
identifying, using the computing device, one or more objects of the two-dimensional image;
for each of the identified objects, segmenting a discrete element within said single two-dimensional digital image, each discrete element including a corresponding object and one or more occluded regions; and
reconstructing said one or more occluded regions of each discrete element.
2. The method of claim 1 , further comprising:
ingesting said one or more reconstructed discrete elements into a two-dimensional to three dimensional conversion process;
initiating said two-dimensional to three dimensional conversion process.
3. The method of claim 2 , wherein said one or more reconstructed discrete elements include a plurality of revealed image data in the place of previously one or more occluded regions.
4. The method of claim 3 , further comprising:
re-composting said one or more reconstructed discrete elements;
ingesting said one or more reconstructed and re-composted discrete elements into said two-dimensional to three dimensional conversion process.
5. The method of claim 4 , further comprising:
wherein said one or more reconstructed discrete elements are re-composited in a monoscopic state that approximates said single two-dimensional digital image prior to segmentation.
6. The method of claim 4 , wherein said re-compositing of said one or more reconstructed discrete elements is accomplished using one or more of: Over; Add; Multiply; or Divide.
7. The method of claim 6 , wherein said re-compositing is not made permanent until said two-dimensional to three-dimensional conversion process is completed.
8. The method of claim 4 , wherein said re-compositing of said one or more reconstructed discrete elements is performed in one or more of:
a two-dimensional digital composite system;
a 2.5D digital composite system;
a three-dimensional digital composite system;
a two-dimensional digital cg model creation system;
a 2.5D digital cg model creation system;
a three-dimensional digital cg model creation system;
a two-dimensional digital animation system;
a 2.5D digital animation system;
a three-dimensional digital animation system;
a two-dimensional digital editing system;
a 2.5D digital editing system;
a three-dimensional digital editing system;
a two-dimensional digital lighting system;
a 2.5D digital lighting system; or
a three-dimensional digital lighting system.
9. The method of claim 3 , further comprising:
estimating one or more occluded regions by pre-visualizing said two-dimensional to three dimensional conversion process of said single two-dimensional digital image and offsetting said one or more discrete elements accordingly;
wherein said one or more occluded regions of said one or more discrete elements is reconstructed based on pre-visualization estimation;
pre-composing said one or more discrete elements as encoded RGB, RGBA, or RGB and alpha image files prior to ingestion of said one or more discrete elements into said two-dimensional to three dimensional conversion process.
10. The method of claim 3 , further comprising:
pre-composing said one or more discrete elements as encoded RGB, RGBA, or RGB and alpha image files prior to ingestion of said one or more discrete elements into said two-dimensional to three dimensional conversion process.
11. The method of claim 3 , wherein one or more alternate perspective images during said two-dimensional to three dimensional conversion process are a result of one or more of:
two-dimensional image modifications driven by three-dimensional geometry manipulation systems;
two-dimensional image modifications driven by two-dimensional depth map systems;
two-dimensional image modifications driven by color or luminance systems;
two-dimensional image modifications driven by two-dimensional warping and filtering systems;
two-dimensional image manipulations driven by motion vector estimation systems;
two-dimensional image manipulations driven by temporal estimation systems;
three-dimensional modeling and texturing based off of camera tracking or solving systems;
three-dimensional modeling and texturing systems;
three-dimensional modeling and displacement systems; or three-dimensional modeling and projection systems.
12. The method of claim 3 , wherein one or more alternate perspective images include one or more revealed regions that were occluded in said single two-dimensional digital image.
13. The method of claim 3 , wherein said two-dimensional to three dimensional conversion process includes determining one or more depths of said single two-dimensional digital image;
wherein said single two-dimensional digital image is recorded by an original monoscopic camera that includes a depth system selected from one or more of:
an additional camera that is mounted to said original monoscopic camera, which is oriented toward a same subject as said original monoscopic camera;
one or more beam splitter camera rigs; said additional camera being placed in proximity to said original monoscopic camera to capture one or more additional perspectives;
a lidar scanning system; a motion capture system;
a computerized motion control camera system; or
a multicamera timed camera system.
14. The method of claim 3 , wherein said two-dimensional to three dimensional conversion process includes determining one or more depths of said single two-dimensional digital image;
wherein said one or more depths are determined through an analysis of one or more depth cues;
wherein said one or more depth cues are selected from one or more of: an atmospheric haze; an aerial dispersion; a volumetric light; a relative size of one or more objects in relation to a standard size of said one more objects; said relative size of said one or more objects in relation to said one or more objects proximity to a horizon; one or more foreground objects occluding one or more background objects; a relative speed of one or more moving objects over an equal distance; and a color, a shadow, or a lighting of said one or more objects.
15. The method of claim 1 , further comprising:
estimating one or more occluded regions based on a pre-visualization of a two-dimensional to three-dimensional conversion of said single two-dimensional digital image.
16. The method of claim 15 , wherein said one or more occluded regions of said one or more discrete elements is fully reconstructed.
17. The method of claim 16 , further comprising:
pre-composing said one or more reconstructed discrete elements as encoded RGB, RGBA, or RGB and alpha image files prior to ingestion of said one or more reconstructed discrete elements into said two-dimensional to three-dimensional conversion process.
18. The method of claim 1 , wherein said segmentation of one or more discrete elements is accomplished by a method selected from one or more of:
rotoscoping said single two-dimensional digital image manually through an operator; rotoscoping said single two-dimensional digital image partially manually and partially automatically; procedural color keying luminance keying; procedural color keying color difference keying; selective image isolation through motion vector analysis; selective image isolation through planar tracking; or selective image isolation through camera solving.
19. The method of claim 1 , further comprising:
inserting a plurality of temporary image data into said one or more reconstructed occluded regions, wherein said plurality of temporary image data is in context of a scene within said original sequence of two-dimensional digital images;
wherein said plurality of temporary image data is selected from one or more of:
a solid color; a repeated pattern of image data; a repeated pattern of a plurality of colors; a replication of an image data formerly occluding said one or more occluded regions; an inverted version of said image data formerly occluding said one or more occluded regions; or blank data.
20. The method of claim 1 , wherein said reconstructing of said one or more occluded regions of said one or more discrete elements is performed by one or more of:
a manual procedural painting;
a manual non-procedural painting;
a matte painting;
a computer generated replacement via three-dimensional model;
a computer generated replacement via two-dimensional model;
a computer generated replacement via particle system;
a computer generated replacement via procedural scripting;
a camera tracking, nodal tracking, planar tracking, motion vector tracking, and matchmoving;
a model;
a partially automated inpainting;
a partially automated image filtering system;
a fully automated inpainting;
a fully automated image filtering system; or
a grain removal, matching, and creation system.
21. A method of preparing a two-dimensional image sequence prior to subjecting the two-dimensional image through a two-dimensional to three-dimensional image conversion process, the method comprising:
receiving, by a computing device, an original sequence of two-dimensional digital images, wherein said original sequence of two-dimensional digital images includes a plurality of two-dimensional digital images;
segmenting, using the computing device, one or more discrete elements within said original sequence of two-dimensional digital images, wherein said one or more discrete elements have one or more occluded regions;
reconstructing said one or more occluded regions of said one or more discrete elements.
22. The method of claim 21 , further comprising:
ingesting said one or more reconstructed discrete elements into a two-dimensional to three dimensional conversion process;
initiating said two-dimensional to three dimensional conversion process.
23. The method of claim 22 , wherein said one or more reconstructed discrete elements include a plurality of revealed image data;
wherein said plurality of revealed image data is used during said two-dimensional to three dimensional conversion process to create one or more alternate perspective image sequences;
wherein said plurality of revealed image data allows said two-dimensional to three dimensional conversion process to be accomplished more efficiently.
24. The method of claim 23 , further comprising:
re-composting said one or more reconstructed discrete elements;
ingesting said one or more reconstructed and re-composted discrete elements into said two-dimensional to three dimensional conversion process.
25. The method of claim 24 , further comprising:
wherein said one or more reconstructed discrete elements are re-composited in a monoscopic state that approximates said original sequence of two-dimensional digital images prior to segmentation.
26. The method of claim 25 , further comprising:
estimating one or more occluded regions by pre-visualizing a two-dimensional to three dimensional conversion of said original sequence of two-dimensional digital images.
27. The method of claim 25 , wherein said one or more occluded regions of said one or more discrete elements is fully reconstructed.
28. The method of claim 27 , further comprising:
pre-composing said one or more reconstructed discrete elements as encoded RGB, RGBA, or RGB and Alpha image files before ingestion of said one or more reconstructed discrete elements into said two-dimensional to three dimensional conversion process.
29. The method of claim 24 , wherein said re-compositing is accomplished using one or more of: Over; Add; Multiply; or Divide.
30. The method of claim 29 , wherein said re-compositing is not made permanent until said two-dimensional to three-dimensional conversion process is completed.
31. The method of claim 23 , further comprising:
estimating one or more occluded regions by pre-visualizing said two-dimensional to three dimensional conversion process of said original sequence of two-dimensional digital images and offsetting said one or more discrete elements accordingly;
wherein said one or more occluded regions of said one or more discrete elements is reconstructed based on pre-visualization estimation;
pre-composing said one or more discrete elements as encoded RGB, RGBA, or RGB and Alpha image files before ingestion of said one or more discrete elements into said two-dimensional to three dimensional conversion process.
32. The method of claim 23 , further comprising:
wherein said one or more occluded regions of said one or more discrete elements is fully reconstructed;
pre-composing said one or more discrete elements as encoded RGB, RGBA, or RGB and Alpha image files before ingestion of said one or more discrete elements into said two-dimensional to three dimensional conversion process.
33. The method of claim 23 , wherein said one or more alternate perspective image sequences created during said two-dimensional to three dimensional conversion process are a result of one or more of:
two-dimensional image modifications driven by three-dimensional geometry manipulation systems;
two-dimensional image modifications driven by two-dimensional depth map systems;
two-dimensional image modifications driven by color or luminance systems;
two-dimensional image modifications driven by two-dimensional warping and filtering systems;
two-dimensional image manipulations driven by motion vector estimation systems;
two-dimensional image manipulations driven by temporal estimation systems;
three-dimensional modeling and texturing based off of camera tracking or solving systems;
three-dimensional modeling and texturing systems;
three-dimensional modeling and displacement systems; or
three-dimensional modeling and projection systems.
34. The method of claim 23 , wherein said one or more alternate perspective image sequences include one or more revealed regions that were occluded in said original sequence of two-dimensional digital images.
35. The method of claim 23 , wherein said two-dimensional to three dimensional conversion process includes determining one or more depths of said original sequence of two-dimensional digital images;
wherein said original sequence of two-dimensional digital images is recorded by an original monoscopic camera that includes a depth system selected from one or more of: an additional camera that is mounted to said original monoscopic camera, which is oriented toward a same subject as said original monoscopic camera; one or more beam splitter camera rigs; said additional camera being placed in proximity to said original monoscopic camera to capture one or more additional perspectives; a lidar scanning system; a motion capture system; a computerized motion control camera system; or a multicamera timed camera system.
36. The method of claim 23 , wherein said two-dimensional to three dimensional conversion process includes determining one or more depths of said original sequence of two-dimensional digital images;
wherein said one or more depths are determined through an analysis of one or more depth cues;
wherein said one or more depth cues are selected from one or more of: an atmospheric haze; an aerial dispersion; a volumetric light; a relative size of one or more objects in relation to a standard size of said one more objects; said relative size of said one or more objects in relation to said one or more objects proximity to a horizon; one or more foreground objects occluding one or more background objects; a relative speed of one or more moving objects over an equal distance; and a color, a shadow, or a lighting of said one or more objects.
37. The method of claim 21 , wherein said segmentation of one or more discrete elements is accomplished by a method selected from one or more of:
rotoscoping said single two-dimensional digital image manually through an operator; rotoscoping said single two-dimensional digital image partially manually and partially automatically; procedural color keying luminance keying; procedural color keying color difference keying; selective image isolation through motion vector analysis; selective image isolation through planar tracking; or selective image isolation through camera solving.
38. The method of claim 21 , further comprising:
inserting a plurality of temporary image data into said one or more reconstructed occluded regions;
wherein said plurality of temporary image data is in context of a scene within said original sequence of two-dimensional digital images;
wherein said plurality of temporary image data is selected from one or more of:
a solid color; a repeated pattern of image data; a repeated pattern of a plurality of colors; a replication of an image data formerly occluding said one or more occluded regions; an inverted version of said image data formerly occluding said one or more occluded regions; or blank data.
39. The method of claim 21 , wherein said reconstructing of said one or more occluded regions of said one or more discrete elements is performed using one or more of:
a manual procedural painting;
a manual non-procedural painting;
a matte painting;
a computer generated replacement via three-dimensional model;
a computer generated replacement via two-dimensional model;
a computer generated replacement via particle system;
a computer generated replacement via procedural scripting;
a camera tracking, nodal tracking, planar tracking, motion vector tracking, and matchmoving;
a model;
a partially automated inpainting;
a partially automated image filtering system;
a fully automated inpainting;
a fully automated image filtering system; or
a grain removal, matching, and creation system.
40. A system comprising:
a processor;
a memory device coupled to the processor and configured to store code corresponding to a series of operations for preparing a two-dimensional image prior to subjecting the two-dimensional image through a two-dimensional to three-dimensional image conversion process, the series of operations including receiving, by a computing device, a two-dimensional digital image;
identifying, using the computing device, one or more objects of the two-dimensional image;
for each of the identified objects, segmenting a discrete element within said single two-dimensional digital image, each discrete element including a corresponding object and one or more occluded regions; and
reconstructing said one or more occluded regions of each discrete element.
41. The system of claim 40 , wherein the set of operations further comprise:
ingesting said one or more reconstructed discrete elements into a two-dimensional to three dimensional conversion process;
initiating said two-dimensional to three dimensional conversion process.
42. The system of claim 41 , wherein said one or more reconstructed discrete elements include a plurality of revealed image data in the place of previously one or more occluded regions.
43. The system of claim 42 , wherein the set of operations further comprises:
re-composting said one or more reconstructed discrete elements;
ingesting said one or more reconstructed and re-composted discrete elements into said two-dimensional to three dimensional conversion process.
44. The system of claim 43 , wherein the set of operations further comprises:
wherein said one or more reconstructed discrete elements are re-composited in a monoscopic state that approximates said single two-dimensional digital image prior to segmentation.
45. The system of claim 40 , wherein the set of operations further comprises:
estimating one or more occluded regions based on a pre-visualization of a two-dimensional to three-dimensional conversion of said single two-dimensional digital image.
46. The system of claim 45 , wherein said one or more occluded regions of said one or more discrete elements is fully reconstructed.
47. The system of claim 46 , wherein the set of operations further comprises:
pre-composing said one or more reconstructed discrete elements as encoded RGB, RGBA, or RGB and alpha image files prior to ingestion of said one or more reconstructed discrete elements into said two-dimensional to three-dimensional conversion process.
48. The system of claim 40 , wherein said segmentation of one or more discrete elements is accomplished by a method selected from one or more of:
rotoscoping said single two-dimensional digital image manually through an operator; rotoscoping said single two-dimensional digital image partially manually and partially automatically; procedural color keying luminance keying; procedural color keying color difference keying; selective image isolation through motion vector analysis; selective image isolation through planar tracking; or selective image isolation through camera solving.
49. The system of claim 40 , wherein the set of operations further comprises:
inserting a plurality of temporary image data into said one or more reconstructed occluded regions, wherein said plurality of temporary image data is in context of a scene within said original sequence of two-dimensional digital images;
wherein said plurality of temporary image data is selected from one or more of:
a solid color; a repeated pattern of image data; a repeated pattern of a plurality of colors; a replication of an image data formerly occluding said one or more occluded regions; an inverted version of said image data formerly occluding said one or more occluded regions; or blank data.
50. The system of claim 40 , wherein said reconstructing of said one or more occluded regions of said one or more discrete elements is performed by one or more of:
a manual procedural painting;
a manual non-procedural painting;
a matte painting;
a computer generated replacement via three-dimensional model;
a computer generated replacement via two-dimensional model;
a computer generated replacement via particle system;
a computer generated replacement via procedural scripting;
a camera tracking, nodal tracking, planar tracking, motion vector tracking, and matchmoving;
a model;
a partially automated inpainting;
a partially automated image filtering system;
a fully automated inpainting;
a fully automated image filtering system; or
a grain removal, matching, and creation system