System and method for player reidentification in broadcast video
A system and method of re-identifying players in a broadcast video feed are provided herein. A computing system retrieves a broadcast video feed for a sporting event. The broadcast video feed includes a plurality of video frames. The computing system generates a plurality of tracks based on the plurality of video frames. Each track includes a plurality of image patches associated with at least one player. Each image patch of the plurality of image patches is a subset of the corresponding frame of the plurality of video frames. For each track, the computing system generates a gallery of image patches. A jersey number of each player is visible in each image patch of the gallery. The computing system matches, via a convolutional autoencoder, tracks across galleries. The computing system measures, via a neural network, a similarity score for each matched track and associates two tracks based on the measured similarity.
1. A method comprising:
identifying, by a computing system, broadcast video feed of a game, the broadcast video feed captured by a broadcast camera;
identifying, by the computing system from the broadcast video feed, a first track corresponding to a first player, the first track comprising trajectory information corresponding to the first player up to a first time, wherein after the first time, the first player is not a field of view of the broadcast camera;
identifying, by the computing system from the broadcast video feed, a second track corresponding to a second player, the second track comprising second trajectory information corresponding to the second player up to a second time, the second time after the first time;
learning, by the computing system, a first set of visual attributes corresponding to the first player in the first track;
learning, by the computing system, a second set of visual attributes corresponding to the second player in the second track;
determining, by the computing system, that the first player and the second player are the same player by measuring a similarity between the first set of visual attributes and the second set of visual attributes; and
inferring, by the computing system, movement of the first player between the first time and the second time, wherein the first player is not in the field of view of the broadcast camera between the first time and the second time.
2. The method of claim 1 , wherein learning, by the computing system, the first set of visual attributes corresponding to the first player in the first track comprises:
generating a plurality of player patches corresponding to the first player based on body-pose information of the first player and an appearance of the first player.
3. The method of claim 2 , wherein learning, by the computing system, the second set of visual attributes corresponding to the second player in the second track comprises:
generating a second plurality of player patches corresponding to the second player based on second body-pose information of the second player and a second appearance of the second player.
4. The method of claim 3 , wherein determining, by the computing system, that the first player and the second player are the same player by measuring the similarity between the first set of visual attributes and the second set of visual attributes comprises:
comparing the plurality of player patches corresponding to the first player to the second plurality of player patches corresponding to the second player.
5. The method of claim 3 , wherein determining, by the computing system, that the first player and the second player are the same player by measuring the similarity between the first set of visual attributes and the second set of visual attributes comprises:
measuring a further similarity between the plurality of player patches and the second plurality of player patches based on their feature representations using a Siamese neural network.
6. The method of claim 1 , further comprising:
based on the inferring, generating, by the computing system, a tracklet of player movement based on the first track and the second track.
7. The method of claim 1 , wherein the first track and the second track do not include any overlapping time.
8. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by a processor, causes a computing system to perform operations comprising:
identifying, by the computing system, broadcast video feed of a game, the broadcast video feed captured by a broadcast camera;
identifying, by the computing system from the broadcast video feed, a first track corresponding to a first player, the first track comprising trajectory information corresponding to the first player up to a first time, wherein after the first time, the first player is not a field of view of the broadcast camera;
identifying, by the computing system from the broadcast video feed, a second track corresponding to a second player, the second track comprising second trajectory information corresponding to the second player up to a second time, the second time after the first time;
learning, by the computing system, a first set of visual attributes corresponding to the first player in the first track;
learning, by the computing system, a second set of visual attributes corresponding to the second player in the second track;
determining, by the computing system, that the first player and the second player are the same player by measuring a similarity between the first set of visual attributes and the second set of visual attributes; and
inferring, by the computing system, movement of the first player between the first time and the second time, wherein the first player is not in the field of view of the broadcast camera between the first time and the second time.
9. The non-transitory computer readable medium of claim 8 , wherein learning, by the computing system, the first set of visual attributes corresponding to the first player in the first track comprises:
generating a plurality of player patches corresponding to the first player based on body-pose information of the first player and an appearance of the first player.
10. The non-transitory computer readable medium of claim 9 , wherein learning, by the computing system, the second set of visual attributes corresponding to the second player in the second track comprises:
generating a second plurality of player patches corresponding to the second player based on second body-pose information of the second player and a second appearance of the second player.
11. The non-transitory computer readable medium of claim 10 , wherein determining, by the computing system, that the first player and the second player are the same player by measuring the similarity between the first set of visual attributes and the second set of visual attributes comprises:
comparing the plurality of player patches corresponding to the first player to the second plurality of player patches corresponding to the second player.
12. The non-transitory computer readable medium of claim 10 , wherein determining, by the computing system, that the first player and the second player are the same player by measuring the similarity between the first set of visual attributes and the second set of visual attributes comprises:
measuring a further similarity between the plurality of player patches and the second plurality of player patches based on their feature representations using a Siamese neural network.
13. The non-transitory computer readable medium of claim 8 , further comprising:
based on the inferring, generating, by the computing system, a tracklet of player movement based on the first track and the second track.
14. The non-transitory computer readable medium of claim 8 , wherein the first track and the second track do not include any overlapping time.
15. A system, comprising:
a processor; and
a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform operations comprising:
identifying broadcast video feed of a game, the broadcast video feed captured by a broadcast camera;
identifying, from the broadcast video feed, a first track corresponding to a first player, the first track comprising trajectory information corresponding to the first player up to a first time, wherein after the first time, the first player is not a field of view of the broadcast camera;
identifying, from the broadcast video feed, a second track corresponding to a second player, the second track comprising second trajectory information corresponding to the second player up to a second time, the second time after the first time;
learning a first set of visual attributes corresponding to the first player in the first track;
learning a second set of visual attributes corresponding to the second player in the second track;
determining that the first player and the second player are the same player by measuring a similarity between the first set of visual attributes and the second set of visual attributes; and
inferring movement of the first player between the first time and the second time, wherein the first player is not in the field of view of the broadcast camera between the first time and the second time.
16. The system of claim 15 , wherein learning the first set of visual attributes corresponding to the first player in the first track comprises:
generating a plurality of player patches corresponding to the first player based on body-pose information of the first player and an appearance of the first player.
17. The system of claim 16 , wherein learning the second set of visual attributes corresponding to the second player in the second track comprises:
generating a second plurality of player patches corresponding to the second player based on second body-pose information of the second player and a second appearance of the second player.
18. The system of claim 17 , wherein determining that the first player and the second player are the same player by measuring the similarity between the first set of visual attributes and the second set of visual attributes comprises:
comparing the plurality of player patches corresponding to the first player to the second plurality of player patches corresponding to the second player.
19. The system of claim 17 , wherein determining that the first player and the second player are the same player by measuring the similarity between the first set of visual attributes and the second set of visual attributes comprises:
measuring a further similarity between the plurality of player patches and the second plurality of player patches based on their feature representations using a Siamese neural network.
20. The system of claim 15 , wherein the operations further comprise:
based on the inferring, generating a tracklet of player movement based on the first track and the second track.