Real-time game tracking with a mobile device using artificial intelligence

1. A method for determining player locations in a multiplayer ball game, comprising:

receiving one or more setup frames and one or more input frames of the multiplayer ball game captured using a camera on a mobile computing device in a gaming area associated with n players, wherein n is greater than or equal to 2;

detecting a plurality of players by performing a computer vision algorithm on the setup frames;

extracting a setup player visual feature from each of the plurality of players to generate a plurality of player visual features;

classifying the plurality of player visual features into one of n player visual feature groups representing the n players respectively;

detecting a plurality of player postures, by performing the computer vision algorithm on the one or more input frames of the multiplayer ball game associated with the n players;

extracting an associated player visual feature from each of the plurality of player postures;

assigning each of the plurality of player postures, based on the player visual feature associated with the player posture, to one of the n player visual feature groups, wherein each player is represented by one of the n player visual feature groups; and

determining a player location for each player, based on the assigned player visual feature group of each player.

2. The method of claim 1 , further comprising:

detecting a ball shot attempt from the input frames, wherein the ball shot attempt is associated with a shot attempt location from which the shot attempt has been initiated; and

determining a shooter for the shot attempt by assigning the ball shot attempt to one of the n players, based on the shot attempt location, and the n player locations.

3. The method of claim 2 , further comprising:

generating a player analytic for the shooter based on a result of the shot attempt.

4. The method of claim 1 , wherein each of the plurality of player visual features is a color feature quantifying color characteristics of one or more regions of a player image.

5. The method of claim 1 , wherein the input frames are non-contiguous.

6. The method of claim 1 , wherein the input frames are from a live video stream.

7. The method of claim 1 , further comprising an initialization step of:

detecting a presence of the n players within a set of boundary lines in the gaming area.

8. The method of claim 7 , wherein the detecting the presence of the n players within the set of boundary lines is by applying a camera projection to initial foot locations of the n players in an image plane of the input frames to determine locations of the players in the gaming area, wherein the initial foot locations are extracted from a second plurality of input frames.

9. The method of claim 1 , wherein each player visual feature is extracted from the setup frames utilizing pose information of each player.

10. The method of claim 1 , wherein the classifying of the plurality of player visual features comprises applying a k-means clustering algorithm to the plurality of player visual features.

11. The method of claim 1 , further comprising computing n player visual feature centroids for the n player visual feature groups, and wherein the assigning each of the plurality of player postures to one of the n player visual feature groups is by bipartite matching to assign each of the plurality of player postures to a player having a player visual feature centroid closer to the player visual feature associated with the player posture.

12. The method of claim 1 , further comprising:

for each player, determining a player foot location in an image plane of the input frames, based on the corresponding player visual feature group, wherein each player location is a location in the gaming area, and wherein the determining of each player location is by applying a camera projection to the corresponding player foot location.

13. The method of claim 12 , further comprising:

determining whether a first player is occluded by a second player; and

in response to determining that the first player is occluded, adjusting the first player location along a camera depth direction associated with the mobile computing device.

14. The method of claim 1 , further comprising:

receiving a user input indicating that n players are participating in the multiplayer ball game.

15. The method of claim 1 , wherein the computer vision algorithm comprises a convolutional neural network (CNN) module.

16. The method of claim 1 , wherein the ball game is basketball.

17. A system for determining player locations in a multiplayer ball game, comprising:

at least one processor; and

a non-transitory computer-readable storage medium for storing program code accessible by the at least one processor, the program code when executed by the processor causes the processor to:

receive one or more setup frames and one or more input frames of the multiplayer ball game captured using a camera on a mobile computing device in a gaming area associated with n players, wherein it is greater than or equal to 2;

detect a plurality of players by performing a computer vision algorithm on the setup frames;

extract a setup player visual feature from each of the plurality of players to generate a plurality of player visual features;

classify the plurality of player visual features into one of n player visual feature groups representing the n players respectively;

detect a plurality of player postures, by performing the computer vision algorithm on the one or more input frames of the multiplayer ball game associated with the n players;

extract an associated player visual feature from each of the plurality of player postures;

assign each of the plurality of player postures, based on the player visual feature associated with the player posture, to one of the n player visual feature groups, wherein each player is represented by one of the n player visual feature groups; and

determine a player location for each player, based on the assigned player visual feature group of each player.

18. The system of claim 17 , wherein the program code when executed by the processor further causes the processor to:

detect a ball shot attempt from the input frames, wherein the ball shot attempt is associated with a shot attempt location from which the shot attempt has been initiated; and

determine a shooter for the shot attempt by assigning the ball shot attempt to one of the n players, based on the shot attempt location, and the n player locations.

19. A non-transitory computer-readable storage medium for determining player locations in a multiplayer ball game, the non-transitory computer-readable storage medium comprising program code stored thereon, and the program code when executed by a processor causes the processor to:

receive one or more setup frames and one or more input frames of the multiplayer ball game captured using a camera on a mobile computing device in a gaming area associated with n players, wherein n is greater than or equal to 2;

detect a plurality of players by performing a computer vision algorithm on the setup frames;

extract a setup player visual feature from each of the plurality of players to generate a plurality of player visual features;

classify the plurality of player visual features into one of n player visual feature groups representing the n players respectively;

detect a plurality of player postures, by performing the computer vision algorithm on the one or more input frames of the multiplayer ball game associated with the n players;

extract an associated player visual feature from each of the plurality of player postures;

assign each of the plurality of player postures, based on the player visual feature associated with the player posture, to one of the n player visual feature groups, wherein each player is represented by one of the n player visual feature groups; and

determine a player location for each player, based on the assigned player visual feature group a each player.

20. The non-transitory computer-readable storage physical medium of claim 19 , wherein the program code when executed by the processor further causes the processor to:

detect a ball shot attempt from the input frames, wherein the ball shot attempt is associated with a shot attempt location from which the shot attempt has been initiated; and

determine a shooter for the shot attempt by assigning the ball shot attempt to one of the n players, based on the shot attempt location, and the n player locations.