System and method for determining 3D positional coordinates of a ball
A system includes a camera capturing images in a first field of view of a sports ball bouncing and rolling, a storage arrangement, and a processing arrangement. The storage arrangement includes a three-dimensional 3D model of a part of a sports play area and the processing arrangement is configured to: detect a pixel location of a ball in an image; determine, based on intrinsic and extrinsic calibration parameters, a camera-ball line comprising a straight line passing through the camera in the direction of the sports ball and to determine, based on the 3D model, an intersection point of the camera-ball line with the 3D model. The processor outputs the intersection point as a 3D position of the ball in the image.
1. A system, comprising:
a tracking camera having a first field of view configured to capture images in a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith;
a storage arrangement including values for the intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least part of a sports play area overlapping with the first field of view; and
a processing arrangement coupled to the tracking camera configured to:
perform a ball detection to detect a pixel location of the sports ball in an image;
determine, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising 3D coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system;
determine, based on the 3D model, an intersection point of the camera-ball line with the 3D model; and
output the intersection point as a 3D position of the sports ball in the image.
2. The system of claim 1 , wherein the processing arrangement is further configured to:
determine the sports ball is at rest by comparing the pixel location of the sports ball in successive images; and
outputting the intersection point as a 3D rest position of the sports ball in the image.
3. The system of claim 1 , wherein the captured images are associated with metadata including a time at which the image was captured, an exposure time and a crop region.
4. The system of claim 1 , further comprising:
a tracking radar having a second field of view at least partially overlapping the first field of view configured to capture radar data of the sports ball,
wherein the processing arrangement is further configured to detect the sports ball in dependence on the radar data to narrow an area searched in the images and a number of images searched in time.
5. The system of claim 4 , wherein detections of the sports ball using the radar data in combination with the images are used by the processing arrangement to improve an accuracy of 3D position determinations of the sports ball.
6. The system of claim 1 , wherein the processing arrangement is further configured to detect the sports ball using a neural network trained to detect sports balls in images.
7. The system of claim 6 , wherein the neural network uses information for search regions from previous images to improve an accuracy of detections of the sports ball in current images.
8. The system of claim 3 , wherein the processing arrangement is further configured to determine the camera-ball line based on the crop region for the image.
9. The system of claim 1 , wherein the sports play area is a golf course and wherein the 3D model of the part of the golf course contains a surface model representing a height of terrain for a given position on the golf course.
10. The system of claim 9 , wherein the surface model further represents a terrain type for the given position on the golf course.
11. The system of claim 10 , wherein the processing arrangement is further configured to determine a type of terrain within which the sports ball is detected based on the 3D model.
12. The system of claim 9 , wherein the surface model comprises a mesh of triangles or spline surfaces.
13. The system of claim 9 , wherein the intersection point is determined using an iterative process or using a numerical solver to solve an optimization problem.
14. The system of claim 1 , wherein the 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.
15. The system of claim 1 , wherein the 3D model is provided in the 3D coordinate system for the camera-ball line.
16. A method, comprising:
performing a ball detection to detect a pixel location of a sports ball in an image captured by a tracking camera, the tracking camera having a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith, values for the intrinsic and extrinsic calibration parameters stored to a storage arrangement;
determining, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system;
determining, based on a 3D model stored to the storage arrangement, an intersection point of the camera-ball line with the 3D model, wherein the 3D model comprises at least part of a sports play area overlapping with the first field of view; and
outputting the intersection point as a 3D position of the sports ball in the image.
17. A processor coupled to a tracking camera and a storage arrangement configured to perform operations comprising:
performing a ball detection to detect a pixel location of a sports ball in an image captured by the tracking camera, the tracking camera having a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith, values for the intrinsic and extrinsic calibration parameters stored to the storage arrangement;
determining, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system;
determining, based on a 3D model stored to the storage arrangement, an intersection point of the camera-ball line with the 3D model, wherein the 3D model comprises at least part of a sports play area overlapping with the first field of view; and
outputting the intersection point as a 3D position of the sports ball in the image.
18. A system, comprising:
a tracking camera having a first field of view configured to capture images in an image plane of a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith;
a storage arrangement including values for the intrinsic and extrinsic calibration parameters; and
a processing arrangement coupled to the tracking camera and the storage arrangement configured to:
perform a ball detection to detect a pixel location in the image plane of the sports ball in each of a sequence of images;
generate a first time-series of the ball detections in the image plane;
determine, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the sports ball detections in the first time-series;
generate a second time-series of an elevation angle of each camera-ball line; and
identify, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
19. The system of claim 18 , wherein the storage arrangement further stores a three-dimensional (3D) model of at least part of a sports play area overlapping with the first field of view, wherein the processing arrangement is further configured to:
determine, based on the 3D model, for each identified bounce, an intersection point of the camera-ball line with the 3D model; and
output the intersection points as 3D positions of bounces of the sports ball.
20. The system of claim 18 , further comprising:
a tracking radar having a second field of view at least partially overlapping the first field of view configured to capture radar data of the sports ball, wherein the processing arrangement is further configured to:
identify bounces in dependence on the radar data by determining velocity discontinuities in the radar data.
21. The system of claim 18 , wherein the processing arrangement is further configured to:
distinguish bounces from rolls in a second time series based on a physical model of a bounce; and
classify each minima in the second time series that has not been determined to be a bounce as being a roll.
22. The system of claim 18 , wherein a 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.
23. A method, comprising:
performing a ball detection to detect a pixel location in an image plane of a sports ball in each of a sequence of images captured by a tracking camera, the tracking camera having a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith, values for the intrinsic and extrinsic calibration parameters stored to a storage arrangement;
generating a first time-series of the ball detections in the image plane;
determining, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series;
generating a second time-series of an elevation angle of each camera-ball line; and
identifying, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
24. A processor coupled to a tracking camera and a storage arrangement configured to perform operations comprising:
performing a ball detection to detect a pixel location in an image plane of a sports ball in each of a sequence of images captured by the tracking camera, the tracking camera having a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith, values for the intrinsic and extrinsic calibration parameters stored to the storage arrangement;
generating a first time-series of the ball detections in the image plane;
determining, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series;
generating a second time-series of an elevation angle of each camera-ball line; and
identifying, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.