METHODS AND SYSTEMS FOR GENERATING SPORTS ANALYTICS WITH A MOBILE DEVICE
Methods and systems for real-time generation of ball shot analytics are disclosed. The methods and systems perform steps for ball and posture detection, ball and posture flow generation, shot event identification and classification, and shot analytics generation based on identified shot events and shooter posture flow. Embodiments of the present invention use computer vision techniques to enable a resource-limited mobile device such as a smartphone to conduct the aforementioned steps. Therefore, the present invention may be implemented using a processor on a single mobile computing device. Also disclosed are benefits of the new methods, and alternative embodiments of implementation.
1 . A method for generating ball shot analytics using a hardware processor on a single mobile computing device, comprising:
receiving, using the single mobile computing device, an input video of a ball game comprising a shooter;
detecting, using the single mobile computing device, a shot attempt ball flow and a shooter posture flow of the shooter from the input video along a time line;
determining, using the single mobile computing device, a ball-from-shooter time by backtracking the shot attempt ball flow from a shot attempt at a goal;
determining, using the single mobile computing device, a shot event occurring before the ball-from-shooter time; and
generating, using the single mobile computing device, one or more shot analytics based on the shot event and the shooter posture flow.
2 . The method of claim 1 , further comprising:
detecting, using the single mobile computing device, a goal location and scene information of a ball game area from the input video;
determining, using the single mobile computing device, a camera projection based on the detected goal location and the scene information;
determining, using the single mobile computing device, shooter foot locations from the shooter posture flow; and
transforming, using the camera projection on the single mobile computing device, the shooter foot locations in image space to a bird's-eye view of the ball game area,
wherein the generating one or more shot analytics based on the shot event and the shooter posture flow comprises generating one or more movement analytics based on the shooter foot locations in the bird's-eye view.
3 . The method of claim 1 , wherein the input video is received from a camera on the single mobile computing device.
4 . The method of claim 1 , wherein the detecting of the shot attempt ball flow and the shooter posture flow comprises using one or more Convolutional Neural Network (CNN) modules.
5 . The method of claim 4 , wherein each CNN module has been trained using one or more prior input videos.
6 . The method of claim 1 , wherein the ball game is selected from the group consisting of basketball, soccer, baseball, football, and hockey.
7 . The method of claim 1 , wherein the input video is streamed.
8 . The method of claim 1 , wherein the detecting of the shot attempt ball flow and the shooter posture flow is applied on a skip frame basis.
9 . The method of claim 1 , wherein the detecting of the shot attempt ball flow and the shooter posture flow comprises applying a bipartite matching to detected balls and player postures, respectively, to existing ball flows and posture flows.
10 . The method of claim 9 , wherein the bipartite matching of a detected ball and an existing ball flow comprises computing a matching score between the detected ball and the existing ball flow, and wherein the computing of the matching score comprises:
generating a predicted ball comprising a next ball location and a next ball size based on the existing ball flow; and
computing the matching score based on a location difference and a size difference between the predicted ball and the detected ball.
11 . The method of claim 1 , further comprising:
receiving a location of the shooter in a shooter identification frame of the input video, wherein the shooter posture flow is a generated player posture flow having a player position closest to the shooter location in the shooter identification frame.
12 . The method of claim 1 , wherein the detecting of the shot attempt ball flow comprises:
identifying the shot attempt ball flow by applying non-max-suppression to a plurality of generated ball flows against the shooter posture flow,
wherein the shot attempt ball flow has a score against the shooter posture flow,
wherein the score is computed based on movements of the shooter, a distance to the shooter, and a confidence value, and
wherein the score is above a pre-defined threshold.
13 . The method of claim 1 , further comprising:
declaring the shot attempt by determining that the shot attempt ball flow traverses a trajectory that comes into close proximity of the goal.
14 . The method of claim 1 , further comprising:
generating a time-series of the one or more shot analytics before and during the shot event.
15 . A single mobile computing device for generating ball shot analytics, comprising:
a processor on the single mobile computing device; and
a non-transitory physical medium for storing program code and accessible by the processor, the program code when executed by the processor causes the processor to:
receive, using the single mobile computing device, an input video of a ball game comprising a shooter;
detect, using the single mobile computing device, a shot attempt ball flow and a shooter posture flow of the shooter from the input video along a time line;
determine, using the single mobile computing device, a ball-from-shooter time by backtracking the shot attempt ball flow from a shot attempt at a goal;
determine, using the single mobile computing device, a shot event occurring before the ball-from-shooter time; and
generate, using the single mobile computing device, one or more shot analytics based on the shot event and the shooter posture flow.
16 . The single mobile computing device of claim 15 , wherein the program code, when executed by the processor, further causes the processor to:
detect, using the single mobile computing device, a goal location and scene information of a ball game area from the input video;
determine, using the single mobile computing device, a camera projection based on the detected goal location and the scene information;
determine, using the single mobile computing device, shooter foot locations from the shooter posture flow; and
transform, using the camera projection on the single mobile computing device, the shooter foot locations in image space to a bird's-eye view of the ball game area,
wherein the program code to generate one or more shot analytics based on the shot event and the shooter posture flow comprises program code to generate one or more movement analytics based on the shooter foot locations in the bird's-eye view.
17 . The single mobile computing device of claim 15 , wherein the input video is received from a camera on the single mobile computing device.
18 . A non-transitory physical medium for generating ball shot analytics, the non-transitory physical medium comprising program code stored thereon, the program code when executed by a processor on a single mobile computing device causes the processor to:
receive, using the single mobile computing device, an input video of a ball game comprising a shooter;
detect, using the single mobile computing device, a shot attempt ball flow and a shooter posture flow of the shooter from the input video along a time line;
determine, using the single mobile computing device, a ball-from-shooter time by backtracking the shot attempt ball flow from a shot attempt at a goal;
determine, using the single mobile computing device, a shot event occurring before the ball-from-shooter time; and
generate, using the single mobile computing device, one or more shot analytics based on the shot event and the shooter posture flow.
19 . The non-transitory physical medium of claim 18 , wherein the program code, when executed by the processor, further causes the processor to:
detect, using the single mobile computing device, a goal location and scene information of a ball game area from the input video;
determine, using the single mobile computing device, a camera projection based on the detected goal location and the scene information;
determine, using the single mobile computing device, shooter foot locations from the shooter posture flow; and
transform, using the camera projection on the single mobile computing device, the shooter foot locations in image space to a bird's-eye view of the ball game area,
wherein the program code to generate one or more shot analytics based on the shot event and the shooter posture flow comprises program code to generate one or more movement analytics based on the shooter foot locations in the bird's-eye view.
20 . The non-transitory physical medium of claim 18 , wherein the input video is received from a camera on the single mobile computing device.