Method to generate neural network training image annotations
A method of generating neural network training image annotations includes training a first neural network to identify and segment hands in images using a first set of 2D images with hand portions segmented in each image; substantially simultaneously capturing both a second set of 2D images, and a third set of images including depth images, depicting hands holding a particular type of object; correlating each of the second set of images with corresponding images of the third set to identify and segment foregrounds from backgrounds in the second set of images; applying the trained first neural network to the identified foregrounds to identify hand portions of the foregrounds and segment object portions from identified hand portions; and training a second neural network, using the segmented object portions of the second set of images as training data, to identify the particular type of object in new images.
1. A computer-implemented method, comprising:
accessing a first set of images, wherein each image of the first set of images is a two-dimensional image depicting a hand holding a target object, and wherein for each two-dimensional image of the first set of images, a hand portion of the two-dimensional image is segmented from the remaining portion of the two-dimensional image;
training a first convolutional neural network (CNN) to identify and segment depictions of hands in digital images using the segmented hand portion of each of the first set of images;
capturing, by a two-dimensional camera, a second set of images, wherein each image of the second set of images is a two-dimensional image depicting a hand holding a particular type of target object;
capturing, by a depth-sensing camera, a third set of images, wherein each image of the third set of images is a depth image depicting a hand holding the particular type of target object, and wherein each depth image of the third set of images is captured substantially simultaneously as a corresponding two-dimensional image of the second set of images depicting the same hand and the same target object;
correlating each two-dimensional image of the second set of images with the corresponding image of the third set of images to identify a foreground and a background of each two-dimensional image of the second set of images;
segmenting, in each two-dimensional image of the second set of images, the foreground of the two-dimensional image from the background of the two-dimensional image;
applying the trained first CNN to the foreground of each two-dimensional image of the second set of images to identify a hand portion of the foreground of each two-dimensional image and segment the remaining target object portion from the hand portion of the foreground of each two-dimensional image; and
training a second CNN to identify the particular type of target object using the segmented remaining target object portion of the foreground of each two-dimensional image of the second set of images.
2. The computer-implemented method of claim 1 , further comprising:
capturing a subsequent two-dimensional image by a two-dimensional camera; and
applying the second CNN to the subsequent two-dimensional image to determine whether the subsequent two-dimensional image depicts the particular type of target object.
3. The computer-implemented method of claim 1 , wherein the hand portion of each two-dimensional image of the first set of images is segmented from the remaining portion of the two-dimensional image manually.
4. The computer-implemented method of claim 1 , wherein the particular type of target object is a first type of target object, the method further comprising:
capturing, by a two-dimensional camera, a fourth set of images, wherein each image of the fourth set of images is a two-dimensional image depicting a hand holding a second type of target object;
capturing, by a depth-sensing camera, a fifth set of images, wherein each image of the fifth set of images is a depth image depicting a hand holding the second type of target object, and wherein each depth image of the fifth set of images is captured substantially simultaneously as a corresponding two-dimensional image of the fourth set of images depicting the same hand and the same target object;
correlating each two-dimensional image of the fourth set of images with the corresponding image of the fifth set of images to identify a foreground and a background of each two-dimensional image of the fourth set of images;
segmenting, in each two-dimensional image of the fourth set of images, the foreground of the two-dimensional image from the background of the two-dimensional image;
applying the trained first CNN to the foreground of each two-dimensional image of the fourth set of images to identify a hand portion of the foreground of each two-dimensional image and segment the remaining target object portion from the hand portion of the foreground of each two-dimensional image; and
training a third CNN to identify the second type of target object using the segmented remaining target object portion of the foreground of each two-dimensional image of the fourth set of images.
5. A system, comprising:
one or more two-dimensional cameras configured to capture a first set of images, wherein each image of the first set of images is a two-dimensional image depicting a hand holding a particular type of target object;
one or more depth-sensing cameras configured to capture a second set images, wherein each image of the second set of images is a depth image depicting the hand holding the particular type of target object, wherein each depth image of the second set of images is captured substantially simultaneously as a corresponding two-dimensional image of the first set images depicting the same hand and the same target object;
one or more processors configured to interface with the two-dimensional camera and the depth-sensing camera; and
a non-transitory program memory communicatively coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the processors to:
correlate each two-dimensional image of the first set of images with the corresponding image of the second set of images to identify a foreground and a background of each two-dimensional image of the second set of images;
segment, in each two-dimensional image of the first set of images, the foreground of the two-dimensional image from the background of the two-dimensional image;
access a third set of images, wherein each image of the first set of images is a two-dimensional image depicting a hand holding a target object, and wherein for each two-dimensional image of the third set of images, a hand portion of the two-dimensional image is segmented from the remaining portion of the two-dimensional image;
train a first convolutional neural network (CNN) to identify and segment depictions of hands in digital images using the segmented hand portion of each of the third set of images;
apply the trained first CNN to the foreground of each two-dimensional image of the first set of images to identify a hand portion of the foreground of each two-dimensional image and segment the remaining target object portion from the hand portion of the foreground of each two-dimensional image; and
train a second CNN to identify the particular type of target object using the segmented remaining target object portion of the foreground of each two-dimensional image of the first set of images.
6. The system of claim 5 , wherein the one or more two-dimensional cameras are configured to capture a subsequent two-dimensional image, and wherein the executable instructions, when executed by the one or more processors, cause the processors to:
apply the second CNN to the subsequent two-dimensional image to determine whether the two-dimensional image depicts the particular type of target object.
7. The system of claim 5 , wherein the hand portion of each two-dimensional image of the third set of images is segmented from the remaining portion of the two-dimensional image manually.
8. The system of claim 5 , wherein the particular type of target object is a first type of target object, wherein the one or more two-dimensional cameras are further configured to capture a fourth set of images, wherein each image of the fourth set of images is a two-dimensional image depicting a hand holding a second type of target object; wherein the one or more depth-sensing cameras are further configured to capture a fifth set of images, wherein each image of the fifth set of images is a depth image depicting a hand holding the second type of target object, and wherein each depth image of the fifth set of images is captured substantially simultaneously as a corresponding two-dimensional image of the fourth set of images depicting the same hand and the same target object; and wherein the executable instructions, when executed by the one or more processors, cause the processors to:
correlate each two-dimensional image of the fourth set of images with the corresponding image of the fifth set of images to identify a foreground and a background of each two-dimensional image of the fourth set of images;
segment, in each two-dimensional image of the fourth set of images, the foreground of the two-dimensional image from the background of the two-dimensional image;
apply the trained first CNN to the foreground of each two-dimensional image of the fourth set of images to identify a hand portion of the foreground of each two-dimensional image and segment the remaining target object portion from the hand portion of the foreground of each two-dimensional image; and
train a third CNN to identify the particular type of target object using the segmented remaining target object portion of the foreground of each two-dimensional image of the fourth set of images.