Systems and methods for keypoint detection with convolutional neural networks

1. A system comprising:

a camera system comprising at least one camera; and

a processor and memory, the processor and memory being configured to:

receive an image captured by the camera system;

compute a plurality of keypoints in the image, using a convolutional neural network comprising:

a first layer implementing a first convolutional kernel,

a second layer implementing a second convolutional kernel,

an output layer, and

a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in the memory, wherein the weights are computed by a training system comprising a processor and memory, the memory of the training system having instructions stored thereon that, when executed by the processor of the training system, cause the training system to:

generate training data comprising a plurality of training images and training detected keypoints of the training images,

transform the training detected keypoints to a response space,

uniformly sample patches from the response space,

identify patches of the training images corresponding to the sampled patches to generate a first training set for training the convolutional neural network, and

compute a plurality of first stage learned neural network parameters from the first training set using backpropagation, and

further train the convolutional neural network using additional patches from the response space and patches of the training images corresponding to the sampled response space patches; and

output the plurality of keypoints of the image computed by the convolutional neural network.

2. The system of claim 1 , wherein the training detected keypoints comprise manually generated training detected keypoints.

3. The system of claim 1 , wherein the training detected keypoints comprise keypoints generated by supplying the training images to one or more keypoint detection algorithms.

4. The system of claim 3 , wherein the one or more keypoint detection algorithms comprise KAZE.

5. The system of claim 3 , wherein the one or more keypoint detection algorithms comprise SIFT.

6. The system of claim 3 , wherein the training data is computed using at least two different keypoint detection algorithms.

7. The system of claim 3 , wherein the training detected keypoints further comprise manually generated training detected keypoints.

8. The system of claim 1 , wherein the memory of the training system further has instructions stored thereon that, when executed by the processor of the training system, cause the training system to train the convolutional neural network further by:

configuring the convolutional neural network using the first stage learned neural network parameters;

computing inferred responses of the convolutional neural network, configured with the first stage learned neural network parameters, to a plurality of training images;

adding, to the first training set, patches from the training images where a difference between the inferred response and the response space exceeds a threshold to generate a second training set;

adding, to the second training set, additional patches randomly sampled from the training images; and

computing a plurality of second stage learned neural network parameters from the second training set using backpropagation.

9. The system of claim 8 , wherein the memory of the training system further has instructions stored thereon that, when executed by the processor of the training system, cause the training system to train the convolutional neural network further by repeatedly updating the second stage learned neural network parameters until a convergence state of the second stage learned neural network parameters, the updating the second stage learned neural network parameters comprising:

configuring the convolutional neural network using the second stage learned neural network parameters;

computing updated inferred responses of the convolutional neural network, configured with the second stage learned neural network parameters, to the plurality of training images;

adding, to the second training set, patches from the training images where a difference between the updated inferred response and the response space exceeds the threshold;

adding, to the second training set, additional patches randomly sampled from the training images; and

re-computing the plurality of second stage learned neural network parameters from the second training set using backpropagation.

10. The system of claim 1 , wherein the first convolutional kernel is separable.

11. The system of claim 1 , wherein the camera system comprises:

a first infrared camera having a first optical axis and a field of view;

a second infrared camera having a second optical axis substantially parallel to the first optical axis and a field of view substantially overlapping the field of view of the first infrared camera; and

an infrared illuminator configured to project light in a direction within the field of view of the first infrared camera and the field of view of the second infrared camera,

wherein the image is a point cloud computed from infrared images captured by the first infrared camera and the second infrared camera.

12. The system of claim 1 , wherein the processor comprises a field programmable gate array, and wherein the field programmable gate array is configured by the weights to implement the convolutional neural network.

13. The system of claim 1 , wherein the weights have values quantized as fixed points.

14. A system comprising:

a camera system comprising at least one camera; and

a processor and memory, the processor and memory being configured to:

receive an image captured by the camera system;

compute a plurality of keypoints in the image using a convolutional neural network comprising:

a first layer implementing a first convolutional kernel;

a second layer implementing a second convolutional kernel;

an output layer; and

a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in the memory; and

output the plurality of keypoints of the image computed by the convolutional neural network,

wherein the camera system comprises:

a first infrared camera having a first optical axis and a field of view;

a second infrared camera having a second optical axis substantially parallel to the first optical axis and a field of view substantially overlapping the field of view of the first infrared camera; and

an infrared illuminator configured to project light in a direction within the field of view of the first infrared camera and the field of view of the second infrared camera,

wherein the image is a point cloud computed from infrared images captured by the first infrared camera and the second infrared camera,

wherein the camera system is arranged to place a conveyor belt in the field of view of the first infrared camera and the field of view of the second infrared camera, and

wherein the processor is configured to:

control the camera system to capture images of objects on the conveyor belt;

compute a plurality of point clouds from the images of the objects;

identify keypoints of the point clouds using the convolutional neural network; and

merge two or more of the point clouds by identifying corresponding ones of the keypoints and rigidly transforming at least one of the point clouds to align the corresponding ones of the keypoints.

15. The system of claim 14 , wherein the processor is further configured to compute a volume of an object in accordance with one or more captured point clouds of the object.

16. A method for computing keypoints of an image, comprising:

controlling, by a processor, a camera system comprising at least one camera to capture an image, wherein the image is a point cloud computed infrared images captured by a camera system comprising:

a first infrared camera having a first optical axis and a field of view,

a second infrared camera having a second optical axis and field of view, the first optical axis being substantially parallel to the second optical axis, and the field of view of the first infrared camera substantially overlapping the field of view of the second infrared camera, and

an infrared illuminator configured to project light in a direction within the field of view of the first camera and the field of view of second infrared camera;

controlling the camera system to capture images of objects on a conveyor belt;

computing a plurality of point clouds from the images of the objects;

identifying keypoints of the point clouds using a convolutional neural network;

merging two or more of the point clouds by identifying corresponding ones of the keypoints;

supplying, by the processor, the image to the convolutional neural network to compute a plurality of keypoints,

the convolutional neural network comprising:

a first layer implementing a first convolutional kernel;

a second layer implementing a second convolutional kernel;

an output layer; and

a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in a memory; and

outputting the plurality of keypoints of the image computed by the convolutional neural network.

17. The method of claim 16 , wherein the weights are computed by: generating training data comprising a plurality of training images and training detected keypoints of the training images; and training the convolutional neural network using the training images and the training detected keypoints of the training data.

18. The method of claim 17 , wherein the training detected keypoints comprise manually generated training detected keypoints.

19. The method of claim 17 , wherein the training detected keypoints comprise keypoints generated by supplying the training images to one or more keypoint detection algorithms.

20. The method of claim 19 , wherein the one or more keypoint detection algorithms comprise KAZE.

21. The method of claim 19 , wherein the one or more keypoint detection algorithms comprise SIFT.

22. The method of claim 19 , wherein the training data is computed using at least two different keypoint detection algorithms.

23. The method of claim 19 , wherein the training detected keypoints further comprise manually generated training detected keypoints.

24. The method of claim 17 , wherein the training the convolutional neural network comprises:

transforming the training detected keypoints to a response space;

uniformly sampling patches from the response space;

identifying patches of the training images corresponding to the sampled patches to generate a first training set for training the convolutional neural network; and

computing a plurality of first stage learned neural network parameters from the first training set using backpropagation.

25. The method of claim 24 , wherein the training the convolutional neural network further comprises:

configuring the convolutional neural network using the first stage learned neural network parameters;

computing inferred responses of the convolutional neural network, configured with the first stage learned neural network parameters, to a plurality of training images;

adding, to the first training set, patches from the training images where a difference between the inferred response and the response space exceeds a threshold to generate a second training set;

adding, to the second training set, additional patches randomly sampled from the training images; and

computing a plurality of second stage learned neural network parameters from the second training set using backpropagation.

26. The method of claim 25 , wherein the training the convolutional neural network further comprises repeatedly updating the second stage learned neural network parameters until a convergence state of the second stage learned neural network parameters, the updating the second stage learned neural network parameters comprising:

configuring the convolutional neural network using the second stage learned neural network parameters;

computing updated inferred responses of the convolutional neural network, configured with the second stage learned neural network parameters, to the plurality of training images;

adding, to the second training set, patches from the training images where a difference between the updated inferred response and the response space exceeds the threshold;

adding, to the second training set, additional patches randomly sampled from the training images; and

re-computing the plurality of second stage learned neural network parameters from the second training set using backpropagation.

27. The method of claim 16 , wherein the first convolutional kernel is separable.

28. The method of claim 16 , further comprising computing a volume of an object in accordance with one or more captured point clouds of the object.

29. The method of claim 16 , A wherein the processor comprises a field programmable gate array, and wherein the field programmable gate array is configured by the weights to implement the convolutional neural network.

30. The method of claim 16 , wherein the weights have values quantized as fixed points.