Machine learning feature vector generator using depth image foreground attributes

1. A method of determining foreground object characteristics of image data, comprising:

receiving a depth image of a first size from a depth sensor;

receiving a segmentation mask of the first size, wherein the segmentation mask corresponds to the depth image and segments the depth image into a set of foreground pixels and a set of background pixels;

determining a set of seed pixels in the depth image, wherein the set of seed pixels either fully or partially encloses a foreground object in the depth image;

for each respective seed pixel of the set of seed pixels:

determining a sampling line in the depth image that starts at the respective seed pixel and passes through a portion of the depth image;

for each respective sampling line pixel in the sampling line having a value in the segmentation mask indicating the foreground object in the depth image:

determining one or more data attribute values based on a depth value for the respective sampling line pixel in the depth image; and

adding the one or more data attribute values to a feature vector.

2. The method of claim 1 , further comprising:

providing the feature vector to a machine learning model; and

receiving an output from the machine learning model.

3. The method of claim 2 , wherein the machine learning model is a fully-connected neural network.

4. The method of claim 2 , further comprising: for each respective sampling line associated with each respective seed pixel of the set of seed pixels:

determining a first boundary pixel in the respective sampling line whose location in the segmentation mask is identified as the foreground object; and

determining a closest depth pixel in the respective sampling line whose depth value is closer to the depth sensor's center of projection than a depth value of any other pixel in the respective sampling line.

5. The method of claim 2 , further comprising: for each respective sampling line associated with each respective seed pixel of the set of seed pixels:

determining a first boundary pixel in the respective sampling line whose location in the segmentation mask is identified as the foreground object; and

determining one or more subsequent depth pixels according to a predetermined stride.

6. The method of claim 2 , wherein the depth sensor comprises a depth camera configured to generate depth images with pixel resolution of 640×480 at 30 frames per second.

7. The method of claim 2 , wherein the depth sensor comprises a dense stereo reconstruction camera.

8. The method of claim 2 , wherein the depth sensor comprises a synthetic depth image renderer.

9. The method of claim 3 , further comprising: inferring one or more skeletal joint positions of a human subject based on the output from the machine learning model.

10. The method of claim 4 , wherein the one or more data attribute values for each respective pixel comprises one or more of:

a distance in pixels from a starting pixel of the respective sampling line to the first boundary pixel of the respective sampling line;

a depth value from the depth image at coordinates of the first boundary pixel;

a distance in pixels from the starting pixel of the respective sampling line to the closest depth pixel;

a depth value from the depth image at coordinates of the closest depth pixel;

a mean depth along the respective sampling line normalized by a quantity of respective sampling line pixels whose Boolean values in the segmentation mask identify as the foreground object in the depth image; or

a mean depth along the respective sampling line normalized by respective sampling line length.

11. The method of claim 5 , wherein the one or more data attribute values for each respective pixel comprises at least:

a depth value from the depth image at the coordinates of the first boundary pixel; and

a depth value from the depth image at the coordinates of each of the one or more subsequent depth pixels.

12. A system for determining foreground object characteristics of image data, comprising:

a memory comprising computer-executable instructions;

one or more processors configured to execute the computer-executable instructions and cause the system to:

receive a depth image of a first size from a depth sensor;

receive a segmentation mask of the first size, wherein the segmentation mask corresponds to the depth image and segments the depth image into a set of foreground pixels and a set of background pixels;

determine a set of seed pixels in the depth image, wherein the set of seed pixels either fully or partially encloses a foreground object in the depth image;

for each respective seed pixel of the set of seed pixels:

determine a sampling line in the depth image that starts at the respective seed pixel and passes through a portion of the depth image;

for each respective sampling line pixel in the sampling line having a value in the segmentation mask indicating the foreground object in the depth image:

determine one or more data attribute values based on a depth value for the respective sampling line pixel in the depth image; and

add the one or more data attribute values to a feature vector.

13. The system of claim 12 , wherein the one or more processors are further configured to:

provide the feature vector to a machine learning model; and

receive an output from the machine learning model.

14. The system of claim 13 , wherein the machine learning model is a fully-connected neural network.

15. The system of claim 13 , wherein the one or more processors are further configured to: for each respective sampling line associated with each respective seed pixel of the set of seed pixels:

determine a first boundary pixel in the respective sampling line whose location in the segmentation mask is identified as the foreground object; and

determine a closest depth pixel in the respective sampling line whose depth value is closer to the depth sensor's center of projection than a depth value of any other pixel in the respective sampling line.

16. The system of claim 13 , wherein the one or more processors are further configured to: for each respective sampling line associated with each respective seed pixel of the set of seed pixels:

determine a first boundary pixel in the respective sampling line whose location in the segmentation mask is identified as the foreground object; and

determine one or more subsequent depth pixels according to a predetermined stride.

17. The system of claim 13 , wherein the depth sensor comprises one of:

a depth camera configured to generate depth images with pixel resolution of 640×480 at 30 frames per second,

a dense stereo reconstruction camera, or

a synthetic depth image renderer.

18. The system of claim 14 , wherein the one or more processors are further configured to: infer one or more skeletal joint positions of a human subject based on the output from the machine learning model.

19. The system of claim 15 , wherein the one or more data attribute values for each respective pixel comprises one or more of:

a distance in pixels from a starting pixel of the respective sampling line to the first boundary pixel of the respective sampling line;

a depth value from the depth image at coordinates of the first boundary pixel;

a distance in pixels from the starting pixel of the respective sampling line to the closest depth pixel;

a depth value from the depth image at coordinates of the closest depth pixel;

a mean depth along the respective sampling line normalized by a quantity of respective sampling line pixels whose Boolean values in the segmentation mask identify as the foreground object in the depth image; or

a mean depth along the respective sampling line normalized by respective sampling line length.

20. The system of claim 16 , wherein the one or more data attribute values for each respective pixel comprises at least:

a depth value from the depth image at the coordinates of the first boundary pixel; and

a depth value from the depth image at the coordinates of each of the one or more subsequent depth pixels.