Depth map movement tracking via optical flow and velocity prediction
Techniques for efficiently tracking points on a depth map using an optical flow are disclosed. In order to optimize the use of optical flow, isolated regions of the depth map may be tracked. The sampling regions may comprise a 3-dimensional box (width, height and depth). Each region may be “colored” as a function of depth information to generate a “zebra” pattern as a function of depth data for each sample. The disclosed techniques may provide for handling optical flow tracking when occlusion occurs by utilizing a weighting process for application of optical flow vs. velocity prediction to stabilize tracking.
1. A method for performing optical flow on a depth map comprising a selected point and a second point, each point indicating a depth value, the method comprising:
determining region comprising the selected and second points;
processing the region by assigning a grayscale value to each point as a function of each point's associated depth value;
processing the region using an optical flow algorithm to determine optical flow vectors for the region;
performing a velocity prediction of the selected point;
analyzing the region to determine an amount of occlusion of the region;
determining a new location for the selected point based on a weighting of the optical flow vectors for the region and the velocity prediction, the weighting being based on the determined amount of occlusion in the region such that a relative weight given to the velocity prediction and a relative weight given to the optical flow vectors depends on the determined amount of occlusion, and the weight given to the velocity prediction is increased relative to the weight given to the optical flow vectors as the determined amount of occlusion increases; and
setting the location of the selected point to the new location.
2. The method of claim 1 , wherein the region comprises a three-dimensional volume oriented around the selected point.
3. The method of claim 2 , wherein the three-dimensional volume filters out points beyond a selected depth.
4. The method of claim 1 , wherein the optical flow algorithm is the Lucas-Kanade algorithm.
5. A computer-readable storage device storing optical flow bearing computer-executable instructions used for performing a process for determining user input to a computer that, when executed on the computer, cause the computer to perform operations comprising:
receiving depth map data representing positions of a computer user with respect to a depth camera;
determining an associated region comprising a selected point and a second point in the depth map;
determining a grayscale value for the selected and second points based on a respective depth value of that point;
performing optical flow on the depth map data to determine optical flow vectors for the region;
performing a velocity prediction of the selected point;
analyzing the region to determine an amount of occlusion of the region;
determining a new location for the selected point based on a weighting of the optical flow vectors for the associated region and the velocity prediction, the weighting being based on the determined amount of occlusion in the associated region, and the weight given to the velocity prediction is increased relative to the weight given to the optical flow vectors as the determined amount of occlusion increases;
setting the location of the selected point to the new location; and
determining a user input to the computer based on the new location of the selected point.
6. The computer-readable storage device of claim 5 , wherein the computer is a game console.
7. The computer-readable storage device of claim 5 , wherein the user is a video game player.
8. The computer-readable storage device of claim 5 , wherein the region comprises a three-dimensional volume oriented around the respective selected point.
9. A system for performing optical flow on depth map data comprising a selected point, comprising:
a processor; and
a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to:
determine an associated region comprising the selected and second points;
process the region by assigning a grayscale value to each point as a function of each point's associated depth value;
process the region using an optical flow algorithm to determine optical flow vectors for the region;
perform a velocity prediction of the selected point;
analyze the region to determine an amount of occlusion of the region;
determine a new location for the selected point based on a weighting of the optical flow vectors for the associated region and the velocity prediction, the weighting being based on the determined amount of occlusion in the associated region, and the weight given to the velocity prediction is increased relative to the weight given to the optical flow vectors as the determined amount of occlusion increases; and
set the location of the selected point to the new location.
10. The system of claim 9 , wherein the optical flow algorithm is the Lucas-Kanade algorithm.