Data processing systems for real-time camera parameter estimation
Data processing systems are disclosed for determining semantic and person keypoints for an environment and an image and matching the keypoints for the image to the keypoints for the environment. A homography is generated based on the keypoint matching and decomposed into a matrix. Camera parameters are then determined from the matrix. A plurality of random camera poses can be generated and used to project keypoints for an environment using image keypoints. The projected keypoints can be compared to the actual keypoints for the environment to determine an error and weighting for each of the random camera poses.
1. A system comprising:
one or more computer processors configured as a neural network; and
memory storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
training the neural network to locate and identify semantic keypoints in an image sequence corresponding to keypoints in an environment, wherein each keypoint of the keypoints in the environment has a known location;
training the neural network to locate and identify person keypoints in the image sequence corresponding to the keypoints in the environment;
matching each semantic keypoint of the semantic keypoints in the image sequence to a respective semantic keypoint of the keypoints in the environment;
matching each person keypoint of the person keypoints in the image sequence to a respective person keypoint of the keypoints in the environment;
generating a homography for each image in the image based on the matching of each of the semantic keypoints in the image sequence to the respective semantic keypoint in the environment and the matching of each of the person keypoints in the image sequence to the respective person keypoint in the environment; and
determining a camera parameter based on the homography.
2. The system of claim 1 , wherein determining the camera parameter comprises determining one or more camera parameters selected from a group consisting of:
(a) a camera focal length;
(b) a camera pose;
(c) a camera orientation; and
(d) a camera position.
3. The system of claim 2 , wherein determining the camera parameter comprises determining the camera focal length, wherein determining the camera focal length comprises:
determining a first focal length based on a first algebraic function;
determining a second focal length based on a second algebraic function; and
determining the camera focal length based on a comparison of the first focal length and the second focal length.
4. The system of claim 2 , wherein determining the camera parameter comprises determining the camera orientation, wherein determining the camera orientation comprises:
decomposing the homography into one or more matrices; and
determining the camera orientation based on the one or more matrices.
5. The system of claim 4 , wherein decomposing the homography into the one or more matrices comprises decomposing the homography into one or more extrinsic parameter matrices; and
wherein determining the camera orientation based on the one or more matrices comprises determining the camera orientation based on the one or more extrinsic parameter matrices.
6. The system of claim 1 , wherein each semantic keypoint of the keypoints in the environment is associated with one or more semantic keypoints selected from a group consisting of:
(a) an intersection of two or more lines configured in the environment;
(b) an endpoint of a line configured in the environment; and
(c) a corner formed by two or more lines configured in the environment.
7. The system of claim 2 , wherein determining the camera parameter comprises determining the camera focal length, wherein determining the camera focal length comprises:
decomposing the homography into one or more intrinsic parameter matrices;
determining a set of intrinsic parameters based on the one or more intrinsic parameter matrices;
removing outliers from the set of intrinsic parameters to generate a subset of intrinsic parameters;
determining a median for each intrinsic parameter of the subset of intrinsic parameters to generate the intrinsic parameter matrix; and
determining the camera focal length based on the intrinsic parameter matrix.
8. A non-transitory computer-readable medium storing computer-executable instructions for:
determining person keypoints for an environment;
determining person keypoints for an image;
generating a plurality of random camera poses;
for each random camera pose of the plurality of random camera poses:
generating projected person keypoints for the image based on the person keypoints for the environment and the random camera pose;
determining an error value for the random camera pose based on a comparison of the projected person keypoints for the image and corresponding person keypoints for the image; and
assigning a weight to the random camera pose based on the error value.
9. The non-transitory computer-readable medium of claim 8 , further comprising computer executable instructions for:
for each random camera pose of the plurality of random camera poses:
comparing the weight assigned to the random camera pose to a threshold; and
determining whether to retain the random camera pose based on the comparison of the weight assigned to the random camera pose to the threshold.
10. The non-transitory computer-readable medium of claim 8 , wherein each random camera pose of the plurality of random camera poses is associated with a respective image of a sequences of images.
11. The non-transitory computer-readable medium of claim 8 , wherein the computer-executable instructions for determining the person keypoints for the environment and determining the person keypoints for the image comprise computer-executable instructions for training a neural network to determine the person keypoints for the environment and the person keypoints for the image.
12. The non-transitory computer-readable medium of claim 11 , wherein the computer-executable instructions for training the neural network to determine the person keypoints for the image comprise computer-executable instructions for training the neural network to identify and assign one or more pixels of the image to a class associated with a person keypoint.
13. The non-transitory computer-readable medium of claim 8 , wherein the random camera pose comprises at least one of a random extrinsic camera parameter and a random intrinsic camera parameter.
14. The non-transitory computer-readable medium of claim 8 , wherein determining the error value comprises determining a distance between a projected person keypoint for the environment and a corresponding person keypoint for the environment.
15. A computer-implemented data-processing method for camera pose estimation, the method comprising:
locating and identifying, by one or more computer processors, keypoints in an image sequence corresponding to keypoints in an environment, wherein each keypoint of the keypoints in the image is a person keypoint or a semantic keypoint;
matching, by one or more computer processors, each keypoint of the keypoints in the image sequence to a respective keypoint of the keypoints in the environment;
generating, by one or more computer processors, a homography for each image in the image sequence based on the matching of the keypoints in the image sequence to the keypoints in the environment; and
determining, by one or more computer processors, a camera parameter based on the homography.
16. The computer-implemented data-processing method of claim 15 , further comprising:
decomposing the homography into an extrinsic parameter matrix, wherein determining the camera parameter based on the homography comprises determining a camera pose based on the extrinsic parameter matrix.
17. The computer-implemented data-processing method of claim 15 , further comprising:
refining each image in the image sequence by applying outlier rejection and particle filtering to each image in the image sequence to generate refined parameter estimates, wherein determining the camera parameter based on the homography comprises determining the camera pose based on the refined parameter estimates.
18. The computer-implemented data-processing method of claim 15 , further comprising:
decomposing the homography into one or more extrinsic parameter matrices, wherein determining the camera parameter based on the homography comprises determining a camera pose based on the one or more extrinsic parameter matrices.
19. The computer-implemented data-processing method of claim 15 , wherein the one or more extrinsic parameter matrices comprises one or more rotation matrices and one or more translational matrices.
20. The computer-implemented data-processing method of claim 15 , wherein each keypoint of the keypoints in the environment has a known location.