3D space carving using hands for object capture
A method for carving a 3D space using hands tracking is described. In one aspect, a method includes accessing a first frame from a camera of a display device, tracking, using a hand tracking algorithm operating at the display device, hand pixels corresponding to one or more user hands depicted in the first frame, detecting, using a sensor of the display device, depths of the hand pixels, identifying a 3D region based on the depths of the hand pixels, and applying a 3D reconstruction engine to the 3D region.
1. A method comprising:
accessing a first frame from a camera of a display device;
tracking, using a hand tracking algorithm operating at the display device, hand pixels corresponding to one or more user hands depicted in the first frame;
detecting, using a sensor of the display device, depths of the hand pixels;
tracking a motion of a left hand and a right hand of a user of the display device;
carving a 3D region that includes a 3D hull of a physical object based on the motion of the left hand and the right hand relative to the physical object;
identifying the 3D region based on the depths of the hand pixels; and
applying a 3D reconstruction engine to the 3D region.
2. The method of claim 1 , wherein the 3D region includes an unoccupied 3D space between the camera and the one or more user hands.
3. The method of claim 1 , wherein the sensor includes a depth sensor or stereo cameras.
4. The method of claim 1 , wherein detecting the depths is based on contour matching of the one or more user hands in two images.
5. The method of claim 1 , wherein carving the 3D region comprises:
detecting the 3D region between the left hand and right hand; and
identifying a 3D envelope comprising the physical object based on the motion of the left hand and the right hand, the motion of the left hand indicating a left side boundary of the 3D region, the right hand indicating a right side boundary of the 3D region.
6. The method of claim 5 , wherein applying the 3D reconstruction engine to the 3D region comprises:
generating a 3D model of the physical object included in the 3D envelope based on point cloud data from the 3D envelope.
7. The method of claim 6 , further comprising:
identifying the physical object based on the 3D model of the physical object.
8. The method of claim 7 , further comprising:
identifying virtual content corresponding to the physical object or the 3D model of the physical object; and
displaying, in a display of the display device, the virtual content as an overlay to the physical object.
9. The method of claim 1 , wherein identifying the 3D region is based on a motion of the one or more user hands comprises:
filtering a first portion of the first frame to identify a first area of interest based on a location of the one or more user hands in the first frame;
filtering a second portion of a second frame to identify a second area of interest based on a location of the one or more user hands in the second frame;
identifying first hand pixel depths of the one or more user hands in the first frame;
identifying second hand pixel depths of the one or more user hands in the second frame; and
identifying the 3D region based on the first area of interest, the second area of interest, the first hand pixel depths, and the second hand pixel depths.
10. The method of claim 1 , wherein carving the 3D region comprises:
detecting the 3D region based on the motion of the left hand and the right hand, the left hand and right hand moving in front of the physical object, behind the physical object, and adjacent to the physical object.
11. A computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the apparatus to:
access a first frame from a camera of a display device;
track, using a hand tracking algorithm operating at the display device, hand pixels corresponding to one or more user hands depicted in the first frame;
detect, using a sensor of the display device, depths of the hand pixels;
track a motion of a left hand and a right hand of a user of the display device;
carve a 3D region that includes a 3D hull of a physical object based on the motion of the left hand and the right hand relative to the physical object;
identify the 3D region based on the depths of the hand pixels; and
apply a 3D reconstruction engine to the 3D region.
12. The computing apparatus of claim 11 , wherein the 3D region includes an unoccupied 3D space between the camera and the one or more user hands.
13. The computing apparatus of claim 11 , wherein the sensor includes a depth sensor or stereo cameras.
14. The computing apparatus of claim 11 , wherein detecting the depths is based on contour matching of the one or more user hands in two images.
15. The computing apparatus of claim 11 , wherein carving the 3D region comprises:
detecting the 3D region between the left hand and right hand tracking; and
identifying a 3D envelope comprising the physical object based on the motion of the left hand and the right hand, the motion of the left hand indicating a left side boundary of the 3D region, the right hand indicating a right side boundary of the 3D region.
16. The computing apparatus of claim 15 , wherein applying the 3D reconstruction engine to the 3D region comprises:
generate a 3D model of the physical object included in the 3D envelope based on point cloud data from the 3D envelope.
17. The computing apparatus of claim 16 , wherein the instructions further configure the apparatus to:
identify the physical object based on the 3D model of the physical object.
18. The computing apparatus of claim 17 , wherein the instructions further configure the apparatus to:
identify virtual content corresponding to the physical object or the 3D model of the physical object; and
display, in a display of the display device, the virtual content as an overlay to the physical object.
19. The computing apparatus of claim 11 , wherein identifying the 3D region is based on a motion of the one or more user hands comprises:
filter a first portion of the first frame to identify a first area of interest based on a location of the one or more user hands in the first frame;
filter a second portion of a second frame to identify a second area of interest based on a location of the one or more user hands in the second frame;
identify first hand pixel depths of the one or more user hands in the first frame;
identify second hand pixel depths of the one or more user hands in the second frame; and
identify the 3D region based on the first area of interest, the second area of interest, the first hand pixel depths, and the second hand pixel depths.
20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
access a first frame from a camera of a display device;
track, using a hand tracking algorithm operating at the display device, hand pixels corresponding to one or more user hands depicted in the first frame;
detect, using a sensor of the display device, depths of the hand pixels;
track a motion of a left hand and a right hand of a user of the display device;
carve a 3D region that includes a 3D hull of a physical object based on the motion of the left hand and the right hand relative to the physical object;
identify the 3D region based on the depths of the hand pixels; and
apply a 3D reconstruction engine to the 3D region.