Sequence-of-sequences model for 3D object recognition
Systems and methods are disclosed for capturing multiple sequences of views of a three-dimensional object using a plurality of virtual cameras. The systems and methods generate aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object. Using a convolutional network, the systems and methods classify the three-dimensional object based on the aligned sequences and identify the three-dimensional object using the classification.
1. A method comprising:
capturing, using a plurality of virtual cameras, multiple sequences of views of a three-dimensional object, each sequence of the multiple sequences representing a unique order of views;
generating, using one or more processors, aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object;
using a convolutional neural network, classifying the three-dimensional object based on the aligned sequences; and
identifying the three-dimensional object using the classification.
2. The method of claim 1 , wherein the arrangement of the plurality of virtual cameras corresponds to an upright orientation and a known rotation axis.
3. The method of claim 1 , wherein each sequence of the multiple sequences has a different starting view.
4. The method of claim 1 , wherein generating aligned sequences further comprises:
aligning each sequence of the multiple sequences using an alignment function to align each sequence of the multiple sequences to a canonical viewpoint.
5. The method of claim 4 , wherein before aligning each sequence, the method further comprises:
generating the alignment function based on a plurality of different starting views.
6. The method of claim 1 , further comprising:
concatenating the aligned sequences; and
classifying the three-dimensional object based on the concatenated aligned sequences.
7. The method of claim 1 , further comprising:
training the convolutional neural network by updating the convolutional neural network with the aligned sequences.
8. The method of claim 1 , further comprising:
classifying the three-dimensional object based on the aligned sequences using a gated recurrent unit.
9. A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to perform operations comprising:
capturing, using a plurality of virtual cameras, multiple sequences of views of a three-dimensional object, each sequence of the multiple sequences representing a unique order of views;
generating, using one or more processors, aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object;
using a convolutional neural network, classifying the three-dimensional object based on the aligned sequences; and
identifying the three-dimensional object using the classification.
10. The system of claim 9 , wherein the arrangement of the plurality of virtual cameras corresponds to an upright orientation and a known rotation axis.
11. The system of claim 9 , wherein each sequence of the multiple sequences has a different starting view.
12. The system of claim 9 , wherein generating aligned sequences further comprises:
aligning each sequence of the multiple sequences using an alignment function to align each sequence to a canonical viewpoint.
13. The system of claim 12 , wherein before aligning each sequence, the operations further comprise:
generating the alignment function based on a plurality of different starting views.
14. The system of claim 9 , further comprising:
concatenating the aligned sequences; and
classifying the three-dimensional object based on the concatenated aligned sequences.
15. The system of claim 9 , further comprising:
training the convolutional neural network by updating the convolutional neural network with the aligned sequences.
16. The system of claim 9 , further comprising:
classifying the three-dimensional object based on the aligned sequences using a gated recurrent unit.
17. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
capturing, using a plurality of virtual cameras, multiple sequences of views of a three-dimensional object, each sequence of the multiple sequences representing a unique order of views;
generating, using one or more processors, aligned sequences from the multiple sequences based on an arrangement of the plurality of virtual cameras in relation to the three-dimensional object;
using a convolutional neural network, classifying the three-dimensional object based on the aligned sequences; and
identifying the three-dimensional object using the classification.
18. The computer-readable storage medium of claim 17 , wherein the arrangement of the plurality of virtual cameras corresponds to an upright orientation and a known rotation axis.
19. The computer-readable storage medium of claim 17 , wherein each sequence of the multiple sequences has a different starting view.
20. The computer-readable storage medium of claim 17 , wherein generating aligned sequences further comprises:
aligning each sequence of the multiple sequences using an alignment function to align each sequence to a canonical viewpoint.