Neural network object pose determination
A camera is positioned to obtain an image of an object. The image is input to a neural network that outputs a three-dimensional (3D) bounding box for the object relative to a pixel coordinate system and object parameters. Then a center of a bottom face of the 3D bounding box is determined in pixel coordinates. The bottom face of the 3D bounding box is located in a ground plane in the image. Based on calibration parameters for the camera that transform pixel coordinates into real-world coordinates, a) a distance from the center of the bottom face of the 3D bounding box to the camera relative to a real-world coordinate system and b) an angle between a line extending from the camera to the center of the bottom face of the 3D bounding box and an optical axis of the camera are determined. The calibration parameters include a camera height relative to the ground plane, a camera focal distance, and a camera tilt relative to the ground plane. A six degree-of-freedom (6DoF) pose for the object is determined based on the object parameters, the distance, and the angle.
1. A system, comprising:
a camera positioned to obtain an image of an object; and
a computer including a processor and a memory, the memory storing instructions executable by the processor to:
input the image to a neural network that outputs a three-dimensional (3D) bounding box for the object relative to a pixel coordinate system and object parameters;
then determine a center of a bottom face of the 3D bounding box in pixel coordinates, wherein the bottom face of the 3D bounding box is located in a ground plane in the image;
upon determining an intersection between a first line extending through a vanishing point for the camera and the center of the bottom face and a second line extending along a bottom boundary of the image, determine a first distance, relative to the real-world coordinate system, from the center of the bottom face to the intersection;
determine a second distance, relative to the real-world coordinate system, from the intersection to the optical axis of the camera;
based on calibration parameters for the camera that transform pixel coordinates into real-world coordinates and the first and second distances, determine a) a distance from the center of the bottom face of the 3D bounding box to the camera relative to a real-world coordinate system and b) an angle between a line extending from the camera to the center of the bottom face of the 3D bounding box and an optical axis of the camera, wherein the calibration parameters include a camera height relative to the ground plane, a camera focal distance, and a camera tilt relative to the ground plane; and
determine a six degree-of-freedom (6DoF) pose for the object based on the object parameters, the distance, and the angle.
2. The system of claim 1 , wherein the instructions further include instructions to determine the first distance based on a distance, in pixels, from the vanishing point to the center of the bottom face, a distance, in pixels, from the vanishing point to the intersection, and the calibration parameters.
3. The system of claim 2 , wherein the instructions further include instructions to determine the first distance based additionally on a projection angle between the optical axis of the camera and the first line.
4. The system of claim 1 , wherein the instructions further include instructions to determine the second distance based on pixel coordinates of the intersection and measuring fiducials.
5. The system of claim 1 , wherein the instructions further include instructions to determine the calibration parameters based on a calibration image including a calibration pattern located parallel to and coincident with the ground plane.
6. The system of claim 1 , wherein the 6DoF pose is determined in coordinates of the real-world coordinate system based on orthogonal x, y, and z axes and roll, pitch, and yaw rotations about the x, y, and z axes, respectively.
7. The system of claim 1 , wherein the object parameters include at least one of dimensions, a two-dimensional bounding box, and an orientation relative to the camera.
8. The system of claim 1 , wherein the computer and the camera are remote from a vehicle, wherein the instructions further include instructions to provide the 6DoF pose for the object to a second computer in the vehicle.
9. The system of claim 8 , further comprising the second computer including a second processor and a second memory, the second memory storing instructions executable by the second processor to operate the vehicle based on the 6DoF pose for the object.
10. The system of claim 8 , wherein the object is the vehicle.
11. The system of claim 1 , wherein the instructions further include instructions to determine the 6DoF pose for the object based additionally on a 6DoF pose of the camera.
12. A method, comprising:
obtaining, via a camera, an image including an object;
inputting, via a computer, an image to a neural network that outputs a three-dimensional (3D) bounding box for the object relative to a pixel coordinate method and object parameters;
then determining a center of a bottom face of the 3D bounding box in pixel coordinates, wherein the bottom face of the 3D bounding box is located in a ground plane in the image;
upon determining an intersection between a first line extending through a vanishing point for the camera and the center of the bottom face and a second line extending along a bottom boundary of the image, determining a first distance, relative to the real-world coordinate system, from the center of the bottom face to the intersection;
determining a second distance, relative to the real-world coordinate system, from the intersection to the optical axis of the camera;
based on calibration parameters for the camera that transform pixel coordinates into real-world coordinates and the first and second distances, determining a) a distance from the center of the bottom face of the 3D bounding box to the camera relative to a real-world coordinate method and b) an angle between a line extending from the camera to the center of the bottom face of the 3D bounding box and an optical axis of the camera, wherein the calibration parameters include a camera height relative to the ground plane, a camera focal distance, and a camera tilt relative to the ground plane; and
determining a six degree-of-freedom (6DoF) pose for the object based on the object parameters, the distance, and the angle.
13. The method of claim 12 , further comprising determining the first distance based on a distance, in pixels, from the vanishing point to the center of the bottom face, a distance, in pixels, from the vanishing point to the intersection, and the calibration parameters.
14. The method of claim 13 , further comprising determining the first distance based additionally on a projection angle between the optical axis of the camera and the first line.
15. The method of claim 12 , further comprising determining the second distance based on pixel coordinates of the intersection and measuring fiducials.
16. The method of claim 12 , further comprising determining the calibration parameters based on a calibration image including a calibration pattern located parallel to and coincident with the ground plane.
17. The method of claim 12 , wherein the computer and the camera are remote from a vehicle, further comprising providing the 6DoF pose for the object to a second computer in the vehicle.
18. The method of claim 17 , further comprising operating, at the second computer, the vehicle based on the 6DoF pose for the object.
19. The method of claim 17 , wherein the object is the vehicle.
20. The method of claim 12 , further comprising determining the 6DoF pose for the object based additionally on a 6DoF pose of the camera.