Action-Actor Detection with Graph Neural Networks from Spatiotemporal Tracking Data
A computing system retrieves tracking data from a data store. The tracking data includes a plurality of frames of data for a plurality of events across a plurality of seasons. The computing system converts the tracking data into a plurality of graph-based representations. A graph neural network learns to generate an action prediction for each player in each frame of the tracking data. The computing system generates a trained graph neural network based on the learning. The computing system receives target tracking data for a target event. The target tracking data includes a plurality of target frames. The computing system converts the target tracking data to a plurality of target graph-based representations. Each graph-based representation corresponds to a target frame of the plurality of target frames. The computing system generates, via the trained graph neural network, an action prediction for each player in each target frame.
1 . A method, comprising:
retrieving, by a computing system, tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;
converting, by the computing system, the tracking data into a plurality of graph-based representations;
learning, by a graph neural network, to generate an action prediction for each player in each frame of the tracking data;
generating, by the computing system, a trained graph neural network based on the learning;
receiving, by the computing system, target tracking data for a target event, the target tracking data comprising a plurality of target frames;
converting, by the computing system, the target tracking data to a plurality of target graph-based representations, wherein each graph-based representation correspond to a target frame of the plurality of target frames; and
generating, by the computing system via the trained graph neural network, an action prediction for each player in each target frame.
2 . The method of claim 1 , wherein the graph neural network comprises a spatial dynamic graph generation network configured to update the graph-based representation with spatial interaction data among players.
3 . The method of claim 2 , wherein the spatial dynamic graph generation network comprises a multi-head self-attention module comprising a plurality of heads, wherein each head corresponds to a respective action of a plurality of actions for classification.
4 . The method of claim 3 , wherein each head of the plurality of heads is configured to generate an adjacency matrix.
5 . The method of claim 2 , where learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
learning spatial relationships between each player in each frame of the tracking data; and
learning neural network weights.
6 . The method of claim 2 , wherein learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
extracting temporal features from the tracking data.
7 . The method of claim 1 , wherein learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
learning to generate a probability distribution across all possible action classes for each player in each frame.
8 . A system, comprising:
a processor; and
a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform one or more operations, comprising:
retrieving tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;
converting the tracking data into a plurality of graph-based representations;
learning, by a graph neural network, to generate an action prediction for each player in each frame of the tracking data;
generating a trained graph neural network based on the learning;
receiving target tracking data for a target event, the target tracking data comprising a plurality of target frames;
converting the target tracking data to a plurality of target graph-based representations, wherein each graph-based representation corresponds to a target frame of the plurality of target frames; and
generating, via the trained graph neural network, an action prediction for each player in each target frame.
9 . The system of claim 8 , wherein the graph neural network comprises a spatial dynamic graph generation network configured to update the graph-based representation with spatial interaction data among players.
10 . The system of claim 9 , wherein the spatial dynamic graph generation network comprises a multi-head self-attention module comprising a plurality of heads, wherein each head corresponds to a respective action of a plurality of actions for classification.
11 . The system of claim 10 , wherein each head of the plurality of heads is configured to generate an adjacency matrix.
12 . The system of claim 9 , where learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
learning spatial relationships between each player in each frame of the tracking data; and
learning neural network weights.
13 . The system of claim 9 , wherein learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
extracting temporal features from the tracking data.
14 . The system of claim 8 , wherein learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
learning to generate a probability distribution across all possible action classes for each player in each frame.
15 . A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, causes a computing system to perform operations, comprising:
retrieving, by the computing system, tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;
converting, by the computing system, the tracking data into a plurality of graph-based representations;
learning, by a graph neural network, to generate an action prediction for each player in each frame of the tracking data;
generating, by the computing system, a trained graph neural network based on the learning;
receiving, by the computing system, target tracking data for a target event, the target tracking data comprising a plurality of target frames;
converting, by the computing system, the target tracking data to a plurality of target graph-based representations, wherein each graph-based representation corresponds to a target frame of the plurality of target frames; and
generating, by the computing system via the trained graph neural network, an action prediction for each player in each target frame.
16 . The non-transitory computer readable medium of claim 15 , wherein the graph neural network comprises a spatial dynamic graph generation network configured to update the graph-graph based representation with spatial interaction data among players.
17 . The non-transitory computer readable medium of claim 16 , wherein the spatial dynamic graph generation network comprises a multi-head self-attention module comprising a plurality of heads, wherein each head corresponds to a respective action of a plurality of actions for classification.
18 . The non-transitory computer readable medium of claim 16 , where learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
learning to spatial relationships between each player in each frame of the tracking data; and
learning neural network weights.
19 . The non-transitory computer readable medium of claim 16 , wherein learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
extracting temporal features from the tracking data.
20 . The non-transitory computer readable medium of claim 15 , wherein learning, by the graph neural network, to generate the action prediction for each player in each frame of the tracking data, comprises:
learning to generate a probability distribution across all possible action classes for each player in each frame.