Data processing systems for real-time camera parameter estimation
Data processing systems are disclosed for determining semantic and person keypoints for an environment and an image and matching the keypoints for the image to the keypoints for the environment. A homography is generated based on the keypoint matching and decomposed into a matrix. Camera parameters are then determined from the matrix. A plurality of random camera poses can be generated and used to project keypoints for an environment using image keypoints. The projected keypoints can be compared to the actual keypoints for the environment to determine an error and weighting for each of the random camera poses.
1. A system for estimating one or more camera parameters, the system comprising:
one or more computer processors; and
memory storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
(a) determining one or more semantic keypoints for an image in an image sequence;
(b) determining one or more person keypoints for the image;
(c) determining, for each of the one or more semantic keypoints for the image, a semantic keypoint location in a world coordinate system of a particular environment depicted in the image sequence;
(d) generating a first estimated homography using the one or more semantic keypoints for the image and the semantic keypoint locations in the world coordinate system;
(e) determining, for each of the one or more person keypoints for the image, a person keypoint location in the world coordinate system of the particular environment;
(f) generating a second estimated homography using the one or more semantic keypoints for the image, the semantic keypoint location for each of the one or more semantic keypoints for the image, the one or more person keypoints for the image, and the person keypoint location for each of the one or more person keypoints for the image;
(g) determining a final estimated homography based at least in part on the second estimated homography;
(h) determining one or more intrinsic parameter estimates based at least in part on the final estimated homography; and
(i) determining one or more extrinsic parameter estimates based at least in part on the final estimated homography.
2. The system of claim 1 , wherein the final estimated homography is the second estimated homography.
3. The system of claim 1 , wherein determining the final estimated homography comprises iteratively generating one or more subsequent homographies by performing one or more iterations of operations comprising:
determining, for each of the one or more person keypoints for the image, a subsequent person keypoint location in the world coordinate system of the particular environment; and
generating a subsequent estimated homography using the one or more semantic keypoints for the image, the semantic keypoint location for each of the one or more semantic keypoints for the image, the one or more person keypoints for the image, and the subsequent person keypoint location for each of the one or more person keypoints for the image.
4. The system of claim 3 , wherein the subsequent estimated homography is generated using random sample consensus.
5. The system of claim 1 , wherein the particular environment comprises a sports field or sports arena on which a sporting event is occurring, and
wherein each of said one or more semantic keypoints corresponds to a predetermined portion of the sports field or sports arena and each person keypoint corresponds to a participant in the sporting event.
6. The system of claim 1 , wherein determining the one or more semantic keypoints in the image comprises determining a classification for each of a plurality of pixels of the image using an artificial neural network.
7. The system of claim 6 , wherein determining the one or more person keypoints in the image comprises said determining of a classification for each of the plurality of pixels of the image using said artificial neural network.
8. The system of claim 1 , wherein the operations comprise:
carrying out operations (a)-(i) for each of a plurality of images in said image sequence; and
for each of said plurality of images, refining the one or more intrinsic parameter estimates using the one or more intrinsic parameter estimates for one or more previous images of the plurality of images.
9. A non-transitory computer-readable medium storing computer-executable instructions for:
(a) identifying semantic keypoints in an image;
(b) identifying person keypoints in the image;
(c) determining, for each of the semantic keypoints, a semantic keypoint location in a coordinate system of a particular environment depicted in the image;
(d) generating a first estimated homography using the semantic keypoints;
(e) determining, for each of the person keypoints for the image a person keypoint location in the coordinate system;
(f) iteratively generating one or more subsequent estimated homographies based at least in part on the semantic keypoints for the image, the semantic keypoint location for each of the semantic keypoints for the image, the person keypoints for the image, and the proximate person keypoint location for each of the person keypoints for the image;
(g) selecting a particular homography of the one or more subsequent estimated homographies as a final estimated homography;
(h) determining one or more intrinsic parameter estimates based at least in part on the final estimated homography; and
(i) determining one or more extrinsic parameter estimates based at least in part on the final estimated homography.
10. The non-transitory computer-readable medium of claim 9 , wherein identifying the semantic keypoints in the image comprises determining a classification for each of a plurality of pixels of the image using an artificial neural network.
11. The non-transitory computer-readable medium of claim 10 , wherein identifying the person keypoints in the image comprises said determining of a classification for each of the plurality of pixels of the image using said artificial neural network.
12. The non-transitory computer-readable medium of claim 9 , wherein the particular environment comprises a sports field or sports arena on which a sporting event is occurring, and
wherein each of said one or more semantic keypoints corresponds to a predetermined portion of the sports field or sports arena and each person keypoint corresponds to a participant in the sporting event.
13. The non-transitory computer-readable medium of claim 9 , wherein the image is part of an image sequence, and wherein the computer-executable instructions are for:
carrying out operations (a)-(i) for each of a plurality of images in said image sequence; and
refining the one or more intrinsic parameter estimates for each of said plurality of images using the one or more intrinsic parameter estimates for one or more previous images of the plurality of images.
14. A computer-implemented data-processing method for camera parameter estimation, the method comprising:
(a) determining one or more semantic keypoints in a particular image of a video sequence;
(b) determining one or more person keypoints in the particular image;
(c) generating a first estimated homography using the one or more semantic keypoints;
(d) determining, for each of the one or more semantic keypoints, a semantic keypoint location in a coordinate system associated with the particular environment that is depicted in the video sequence;
(e) determining, for each of the one or more person keypoints, a person keypoint location in the coordinate system;
(f) generating one or more subsequent estimated homographies based at least in part on the one or more semantic keypoints, the semantic keypoint location for each of the one or more semantic keypoints, the one or more person keypoints, and the person keypoint location for each of the one or more person keypoints;
(g) determining a final estimated homography based at least in part on the one or more subsequent estimated homographies;
(h) determining one or more intrinsic parameter estimates based at least in part on the final estimated homography; and
(i) determining one or more extrinsic parameter estimates based at least in part on the final estimated homography.
15. The computer-implemented data-processing method of claim 14 , wherein said generating of one or more subsequent estimated homographies is performed iteratively, and wherein said determining of a final estimated homography uses random sample consensus.
16. The computer-implemented data-processing method of claim 14 , wherein the particular environment comprises a sports field or sports arena on which a sporting event is occurring, and
wherein each of said one or more semantic keypoints corresponds to a predetermined portion of the sports field or sports arena and each person keypoint corresponds to a participant in the sporting event.
17. The computer-implemented data-processing method of claim 14 , wherein determining the one or more semantic keypoints in the image comprises determining a classification for each of a plurality of pixels of the image using an artificial neural network.
18. The computer-implemented data-processing method of claim 17 , wherein determining the one or more person keypoints in the image comprises said determining of a classification for each of the plurality of pixels of the image using said artificial neural network.
19. The computer-implemented data-processing method of claim 14 , comprising:
carrying out operations (a)-(i) for each of a plurality of images in said video sequence; and
refining the one or more intrinsic parameter estimates for each of said plurality of images using the one or more intrinsic parameter estimates for one or more previous images of the plurality of images.