Motion based pre-processing of two-dimensional image data prior to three-dimensional object tracking with virtual time synchronization
Methods, systems, and apparatus, including medium-encoded computer program products, for pre-processing image data before 3D object tracking include, in at least one aspect, a method including: performing object detection in uncompressed, two-dimensional image data from a camera to produce two-dimensional location data for objects of interest; processing the two-dimensional location data for the objects of interest using a motion criterion to generate possible paths data for the objects of interest; and constructing a flight track of an object in three-dimensional space, from the possible paths data and position information obtained from a sensor, by filtering out false positives in the possible paths data.
1. A method comprising:
performing object detection in uncompressed, two-dimensional image data from a camera to produce two-dimensional location data for objects of interest;
processing the two-dimensional location data for the objects of interest using a motion criterion to generate possible paths data for the objects of interest; and
constructing a flight track of an object in three-dimensional space, from the possible paths data and position information obtained from a sensor, by filtering out false positives in the possible paths data.
2. The method of claim 1 , wherein performing the object detection and processing the two-dimensional location data effectively compresses ball location data, to reduce bandwidth requirements for sending the possible paths data, in a manner that enables virtual time synchronization of measured object positions.
3. The method of claim 1 , wherein performing the object detection comprises storing in a list both a location and a size for each set of one or more pixels in the uncompressed, two-dimensional image data from the camera that satisfy an object detection criteria.
4. The method of claim 3 , wherein performing the object detection comprises adaptively adjusting noise thresholds, and processing the two-dimensional location data comprises initiating a possible new path only when a detected object satisfies a predefined size criterion.
5. The method of claim 1 , wherein performing the object detection comprises using a low threshold to favor zero false negatives, while allowing plenty of false positives in the object detection.
6. The method of claim 5 , wherein processing the two-dimensional location data comprises finding sequences of detected objects over three or more image frames that indicate object movement consistent with that of an object in Newtonian motion.
7. The method of claim 6 , wherein the motion criterion is defined to include displacement, velocity, acceleration, or a combination thereof, in each of x and y image dimensions, being inside a predefined range of values.
8. The method of claim 5 , wherein processing the two-dimensional location data comprises loose filtering that allows objects other than objects in motion to be included in the possible paths data, thereby increasing false positives and minimizing false negatives in the possible paths data.
9. The method of claim 1 , wherein processing the two-dimensional location data comprises predicting a next location to add to a possible path for the objects of interest.
10. The method of claim 1 , comprising only sending a possible path for flight track construction when the possible path includes a sequence of locations that exceeds a specified minimum number of locations.
11. The method of claim 10 , comprising actively determining the specified minimum number of locations based on one or more factors comprising a total number of possible paths currently being generated.
12. The method of claim 10 , comprising delaying output of possible paths data for a given image frame until no further detected objects in the given image frame can be included in a sequence of locations that exceeds the specified minimum number of locations.
13. A system comprising:
a camera;
a sensor; and
two or more computers communicatively coupled with the camera and the sensor;
wherein the two or more computers are configured to
perform object detection in uncompressed, two-dimensional image data from the camera to produce two-dimensional location data for objects of interest,
process the two-dimensional location data for the objects of interest using a motion criterion to generate possible paths data for the objects of interest, and
construct a flight track of an object in three-dimensional space, from the possible paths data and position information obtained from the sensor, by filtering out false positives in the possible paths data.
14. The system of claim 13 , wherein the camera, the sensor, and at least one of the two or more computers is located in a sports stadium, and the two-dimensional location data for the objects of interest, the possible paths data for the objects of interest, or both, are sent from a first of the two or more computers to a second of the two or more computers using one or more lower bandwidth, longer distance data communication technologies.
15. The system of claim 14 , wherein the one or more lower bandwidth, longer distance data communication technologies comprise wireless data connections.
16. The system of claim 13 , wherein the camera, the sensor, and at least one of the two or more computers is located on a golf course, and the two-dimensional location data for the objects of interest, the possible paths data for the objects of interest, or both, are sent from a first of the two or more computers to a second of the two or more computers using one or more lower bandwidth, longer distance data communication technologies.
17. The system of claim 16 , wherein the camera is located behind a green on the golf course for a golf competition, the second of the two or more computers is positioned at a television broadcast facility that is a mile or more from the green on the golf course, and the television broadcast facility is configured to overlay a trace of the flight track of the object onto video data obtained by a television camera.
18. The system of claim 13 , wherein the camera and the sensor are included in a set of three or more sensors located at a golf range, a grass field, or another open area into which golf balls can be launched.
19. The system of claim 13 , wherein the sensor comprises a radar device.
20. The system of claim 13 , wherein the sensor comprises another camera.
21. The system of claim 20 , wherein at least one of the two or more computers is configured to filter out the false positives in the possible paths data at least in part by being configured to perform epipolar line filtering before triangulation.
22. The system of claim 13 , wherein a first computer of the two or more computers comprises embedded firmware configured to process the two-dimensional location data, and a second computer of the two or more computers comprises one or more data processing apparatus encoding software configured to construct the flight track of the object in three-dimensional space.
23. The system of claim 13 , wherein a first computer of the two or more computers is configured to perform the object detection, a second computer of the two or more computers is configured to process the two-dimensional location data, and a third computer of the two or more computers is configured to construct the flight track of the object in three-dimensional space.
24. The system of claim 23 , wherein the first computer is configured to perform pre-tracking in two dimensions using loose constraints.
25. The system of claim 13 , wherein the camera is a rolling shutter camera, and the two or more computers are configured to take into account particular time offsets of detected object locations within a frame of the rolling shutter camera.