Object pose estimation in visual data
The pose of an object may be estimated based on fiducial points identified in a visual representation of the object. Each fiducial point may correspond with a component of the object, and may be associated with a first location in an image of the object and a second location in a 3D space. A 3D skeleton of the object may be determined by connecting the locations in the 3D space, and the object's pose may be determined based on the 3D skeleton.
1. A method comprising:
identifying via a processor a plurality of fiducial points by applying a first neural network to a plurality of two-dimensional (2D) images of a vehicle captured at a plurality of respective viewpoints by a handheld visible light camera as the handheld visible light camera moves along a path through space, each of the one or more 2D images being captured from a respective camera viewpoint, each fiducial point having a respective first location in one or more of the images, each fiducial point corresponding with a respective component of the vehicle;
for each fiducial point, determining via a processor a respective second location in a three-dimensional (3D) space based on the respective location associated with the respective fiducial point;
determining a 3D skeleton of the vehicle by connecting the respective second locations in the 3D space via a processor;
determining via the processor a pose of the vehicle based on the 3D skeleton with respect to the camera viewpoint, wherein the pose includes a roll value, a pitch value, or a yaw value for the vehicle with respect to the respective viewpoint; and
storing the pose on a storage device.
2. The method recited in claim 1 , wherein the pose includes three translation values identifying a location of the vehicle in the three-dimensional space.
3. The method recited in claim 1 , wherein the 3D skeleton includes a door and a windshield.
4. The method recited in claim 3 , wherein the fiducial points include a headlight portion, a rear-view mirror portion, and a wheel portion.
5. The method recited in claim 1 , wherein the pose of the vehicle includes a respective object viewpoint pose for each of the plurality of images.
6. The method recited in claim 5 , wherein the plurality of 2D images are included in a multi-view capture constructed based on inertial measurement unit (IMU) data and navigable in three dimensions.
7. The method recited in claim 1 , wherein the first neural network is trained to segment objects into components and to classify object components.
8. The method recited in claim 1 , the method further comprising:
determining an object type based on the visual representation.
9. The method recited in claim 8 , the method further comprising:
identifying a pre-determined 3D mesh corresponding with the object type.
10. The method recited in claim 8 , wherein the respective second locations are determined at least in part based on positioning each fiducial point within the pre-determined 3D mesh.
11. A system configured to perform a method, the method comprising:
identifying via a processor a plurality of fiducial points by applying a first neural network to a plurality of two-dimensional (2D) images of a vehicle captured at a plurality of respective viewpoints by a handheld visible light camera as the handheld visible light camera moves along a path through space, each of the one or more 2D images being captured from a respective camera viewpoint, each fiducial point having a respective first location in one or more of the images, each fiducial point corresponding with a respective component of the vehicle;
for each fiducial point, determining via a processor a respective second location in a three-dimensional (3D) space based on the respective location associated with the respective fiducial point;
determining a 3D skeleton of the vehicle by connecting the respective second locations in the 3D space via a processor;
determining via the processor a pose of the vehicle based on the 3D skeleton with respect to the camera viewpoint, wherein the pose includes a roll value, a pitch value, or a yaw value for the vehicle with respect to the respective viewpoint; and
storing the pose on a storage device.
12. The system recited in claim 11 , wherein the 3D skeleton includes a door and a windshield, and wherein the fiducial points include a headlight portion, a rear-view mirror portion, and a wheel portion.
13. The system recited in claim 11 , wherein the pose of the vehicle includes a respective viewpoint pose for each of the plurality of images, wherein the 2D images are included in a multi-view capture constructed based on inertial measurement unit (IMU) data and navigable in three dimensions.
14. The system recited in claim 11 , wherein the 3D skeleton is determined at least in part by applying a second neural network trained to predict 3D coordinates of visible portions of objects represented in two-dimensional (2D) images.
15. The system recited in claim 11 , wherein the first neural network is trained to segment objects into components and to classify object components.
16. The system recited in claim 11 , the method further comprising:
determining an object type based on the visual representation; and
identifying a pre-determined 3D mesh corresponding with the object type.
17. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
identifying via a processor a plurality of fiducial points by applying a first neural network to a plurality of two-dimensional (2D) images of a vehicle captured at a plurality of respective viewpoints by a handheld visible light camera as the handheld visible light camera moves along a path through space, each of the one or more 2D images being captured from a respective camera viewpoint, each fiducial point having a respective first location in one or more of the images, each fiducial point corresponding with a respective component of the vehicle;
for each fiducial point, determining via a processor a respective second location in a three-dimensional (3D) space based on the respective location associated with the respective fiducial point;
determining a 3D skeleton of the vehicle by connecting the respective second locations in the 3D space via a processor;
determining via the processor a pose of the vehicle based on the 3D skeleton with respect to the camera viewpoint, wherein the pose includes a roll value, a pitch value, or a yaw value for the vehicle with respect to the respective viewpoint; and
storing the pose on a storage device.