Method and system for real-time 3D capture and live feedback with monocular cameras
Forming a 3D representation of an object using a portable electronic device with a camera. A sequence of image frames, captured with the camera may be processed to generate 3D information about the object. This 3D information may be used to present a visual representation of the object as real-time feedback to a user, indicating confidence of 3D information for regions of the object. Feedback may be a composite image based on surfaces in a 3D model derived from the 3D information given visual characteristics derived from the image frames. The 3D information may be used in a number of ways, including as an input to a 3D printer or an input, representing, in whole or in part, an avatar for a game or other purposes. To enable processing to be performed on a portable device, a portion of the image frames may be selected for processing with higher resolution.
1. A non-transitory computer readable medium encoding processor-executable instructions that, when executed by at least one hardware processor, perform a method of forming a three-dimensional (3D) representation of an object from a plurality of image frames depicting the object from a plurality of orientations, the method comprising, iteratively:
computing 3D information about the object based on processing the plurality of image frames, wherein the plurality of image frames comprises higher resolution image frames and lower resolution image frames interleaved with the higher resolution image frames, wherein:
processing the higher resolution image frames comprises computing depth information for the object, the depth information comprising a plurality of depth estimates for pixels of the higher resolution image frames, and associated confidence values; and
processing the lower resolution image frames comprises refining the computed depth information for the object by adjusting the plurality of depth estimates an amount determined from pixels of the lower resolution image frames and weighted by the confidence values associated with the plurality of depth estimates; and
displaying a representation of the object based on the 3D information, wherein the representation of the object changes based on the computed depth information being refined.
2. The non-transitory computer readable medium of claim 1 , wherein:
computing 3D information comprises determining confidences with which the computed 3D information represents regions of the object; and
displaying the representation of the object comprises visually differentiating regions of the object for which the 3D information has a higher confidence from regions of the object for which the 3D information has a lower confidence.
3. The non-transitory computer readable medium of claim 1 , wherein the computer-executable instructions are formatted for execution by a processor of a smartphone.
4. The non-transitory computer readable medium of claim 1 , wherein:
the processor executable instructions are formatted for execution by a processor of a portable electronic device comprising a camera and a display; and
displaying the representation of the object comprises displaying the representation after processing a first portion of the plurality of image frames while capturing a second portion of the plurality of image frames.
5. The non-transitory computer readable medium of claim 1 , wherein:
the plurality of image frames comprise a stream of image frames captured with a camera; and
the method further comprises selectively capturing image frames of the higher resolution image frames at a predetermined frequency and/or based on a rate of motion of the camera.
6. The non-transitory computer readable medium of claim 1 , wherein:
the 3D information comprises a mesh, representing 3D locations of features of the object and processing the plurality of image frames comprises computing the mesh from the computed depth information.
7. The non-transitory computer readable medium of claim 1 , wherein:
displaying the representation of the object comprises depicting surfaces based on the 3D information with a surface characteristic based on the at least one image frame of the plurality of image frames.
8. The non-transitory computer readable medium of claim 7 , wherein:
computing 3D information comprises determining confidences with which the computed 3D information represents regions of the object; and
displaying the representation of the object comprises presenting surfaces with a surface characteristic representing regions of the object for which the 3D information has a higher confidence and surfaces without the surface characteristic representing regions of the object for which the 3D information has a lower confidence.
9. A portable electronic device comprising at least one hardware processor, a camera and a display, wherein the portable electronic device is configured to perform a method of forming a three-dimensional (3D) representation of an object from a stream of image frames captured with the camera, the stream comprising a plurality of image frames depicting the object from a plurality of orientations, the method comprising, iteratively:
computing 3D information about the object based on computing depth information by:
generating a plurality of depth estimates by processing a first subset of the plurality of image frames at a first, higher resolution; and
refining the generated plurality of depth estimates by processing a second subset of the plurality of image frames at a second, lower resolution,
wherein the image frames processed at a lower resolution are interleaved with the image frames processed at a higher resolution;
displaying a first representation of the object based on the 3D information computed after image frames of a first portion of the stream of image frames have been processed, the first portion of the stream of image frames comprising one or more of the first subset of the plurality of image frames at the first, higher resolution and one or more of the second subset of the plurality of image frames at the second, lower resolution; and
displaying a second representation of the object after the first portion and a second portion of the stream of image frames have been processed.
10. The portable electronic device of claim 9 , wherein:
computing 3D information comprises determining confidences with which the computed information represents regions of the object; and
displaying the first representation of the object comprises visually differentiating regions of the object for which the 3D information has a higher confidence from regions of the object for which the 3D information has a lower confidence.
11. The portable electronic device of claim 9 , wherein computing the 3D information comprises selecting the first subset of the plurality of image frames from the stream at a predetermined frequency.
12. The portable electronic device of claim 9 , wherein:
the portable electronic device comprises an inertial measurement unit; and
computing the 3D information comprises selecting the first subset of the plurality of image frames from the stream based on a rate of motion of the portable electronic device as indicated by the inertial measurement unit.
13. The portable electronic device of claim 9 , wherein displaying the first representation of the object comprises superimposing a mesh, representing 3D locations of features of the object, on an image of the object.
14. The portable electronic device of claim 9 , wherein displaying the first representation of the object comprises depicting surfaces based on the 3D information with a surface characteristic based on at least one image of the plurality of image frames.
15. A portable electronic device comprising at least one hardware processor, a camera and a display, wherein the portable electronic device is configured to perform a method of forming a three-dimensional (3D) representation of an object from a stream of image frames captured with the camera, the stream comprising a plurality of image frames depicting the object from a plurality of orientations, the method comprising, iteratively:
computing 3D information about the object based on computing depth information by:
generating a plurality of depth estimates and associated confidence values based on a first subset of the plurality of image frames at a first, higher resolution; and
adjusting the generated plurality of depth estimates based on a second subset of the plurality of image frames at a second, lower resolution and based on the confidence values associated with the plurality of depth estimates,
wherein the image frames processed at a lower resolution are interleaved with the image frames processed at a higher resolution; and
displaying a first representation of the object based on the 3D information computed after image frames of a first portion of the stream of image frames have been processed and displaying a second representation of the object after the first portion and a second portion of the stream of image frames have been processed.
16. The portable electronic device of claim 15 , wherein displaying the first representation of the object comprises visually differentiating regions of the object for which the plurality of depth estimates have a higher confidence value from regions of the object for which the plurality of depth estimates have a lower confidence value.
17. The portable electronic device of claim 15 , wherein computing the 3D information comprises selecting the first subset of the plurality of image frames from the stream at a predetermined frequency.
18. The portable electronic device of claim 15 , wherein:
the portable electronic device comprises an inertial measurement unit; and
computing the 3D information comprises selecting the first subset of the plurality of image frames from the stream based on a rate of motion of the portable electronic device as indicated by the inertial measurement unit.
19. The portable electronic device of claim 15 , wherein displaying the first representation of the object comprises superimposing a mesh, representing 3D locations of features of the object, on an image of the object.
20. The portable electronic device of claim 15 , wherein displaying the first representation of the object comprises depicting surfaces based on the 3D information with a surface characteristic based on at least one image of the plurality of image frames.