Action detection using machine learning models

1 . A computer-implemented method comprising:

receiving video data representing a video that captures movements of a subject in a plurality of subjects;

identifying a first set of frames from the video data;

determining a rotated set of frames by rotating the first set of frames;

processing the first set of frames using a first trained model configured to identify a likelihood of the subject exhibiting a predetermined behavioral action;

based on the processing of the first set of frames by the first trained model, determining a first probability of the subject exhibiting the predetermined behavioral action in a first frame of the first set of frames, the first frame corresponding to a time duration of the video data;

processing the rotated set of frames using the first trained model;

determining a reflected set of frames by reflecting the first set of frames;

processing the reflected set of frames using the first trained model;

identifying the first label using the first probability, the second probability, and the third probability;

based on the processing of the rotated set of frames by the first trained model, determining a second probability of the subject exhibiting the predetermined behavioral action in a second frame of the rotated set of frames, the second frame corresponding to the time duration of the first frame;

based on the processing of the reflected set of frames by the first trained model, determining a third probability of the subject exhibiting the predetermined behavioral action a third frame of the reflected set of frames, the third frame corresponding to the first frame; and

using the first probability, the second probability, and the third probability, identifying a first label for the first frame, the first label indicating that the subject exhibits the predetermined behavioral action.

2 . The computer-implemented method of claim 1 , further comprising:

processing the first set of frames using a second trained model configured to identify a likelihood of the subject exhibiting the predetermined behavioral action;

based on the processing of the first set of frames by the second trained model, determining a third probability of the subject exhibiting the predetermined behavioral action in the first frame;

processing the rotated set of frames using the second trained model;

based on the processing of the rotated set of frames by the second trained model, determining a fourth probability of the subject exhibiting the predetermined behavioral action in the second frame; and

identifying the first label using the first probability, the second probability, the third probability and the fourth probability.

3 . The computer-implemented method of claim 1 , wherein the predetermined behavioral action comprises a grooming behavior, wherein the grooming behavior comprises at least one of paw licking, unilateral face wash, bilateral face wash, and flank licking.

4 . The computer-implemented method of claim 1 , wherein the first set of frames represent a portion of the video data during a time period, and the first frame is a last temporal frame of the time period.

5 . The computer-implemented method of claim 1 , further comprising:

identifying a second set of frames from the video data;

determining a second rotated set of frames by rotating the second set of frames;

processing the second set of frames using the first trained model;

based on the processing of the second set of frames by the first trained model, determining a third probability of the subject exhibiting the predetermined behavioral action in a third frame of the second set of frames;

processing the second rotated set of frames using the first trained model;

based on the processing of the second rotated set of frames by the first trained model, determining a fourth probability of the subject exhibiting the predetermined behavioral action in a fourth frame of the rotated set of frames, the fourth frame corresponding to the third frame; and

using the third probability and the fourth probability, identifying a second label for the fourth frame, the first label indicating that the subject exhibits the predetermined behavioral action.

6 . The computer-implemented method of claim 5 , further comprising:

using at least the first label and the second label, generating an ethogram representing the predetermined behavioral action of the subject during a time period.

7 . The computer-implemented method of claim 1 , further comprising prior to receiving the video data:

receiving training data including a first plurality of video frames and a second plurality of video frames, each of the first plurality of video frames associated with a positive label indicating that the subject is exhibiting the predetermined behavioral action and each of the second plurality of video frames associated with a negative label indicating that the subject is exhibiting a behavioral action that is not the predetermined behavior action; and

processing the training data using a first set of model parameters and first classifier model data to determine the first trained model.

8 . The computer-implemented method of claim 7 , wherein the first plurality of video frames and the second plurality of video frames represent movements of the plurality of subjects, wherein the subject in the plurality of subjects comprises one or more pre-identified physical characteristic(s).

9 . The computer-implemented method of claim 8 , wherein the pre-identified physical characteristic is one or more of a body shape, a body size, a coat color, a gender, an age, or a phenotype of a disease or disorder.

10 . The computer-implemented method of claim 9 , wherein the disease or disorder is a heritable disease, an injury, or a contagious disease.

11 . The computer-implemented method of claim 7 , wherein the first plurality of video frames and the second plurality of video frames represent movements of the plurality of subjects that are mice, wherein a mouse in the plurality of subjects that are mice comprises a coat color, a gender, a body shape and a size.

12 . A computer-implemented method comprising:

receiving video data representing a video that captures movements of a subject in a plurality of subjects;

identifying a first set of frames from the video data;

processing the first set of frames using a first trained model configured to identify a likelihood of the subject exhibiting a predetermined behavioral action;

determining a reflected set of frames by reflecting the first set of frames;

processing the reflected set of frames using the first trained model;

based on the processing of the first set of frames by the first trained model, determining a first probability of the subject exhibiting the predetermined behavioral action in a first frame of the first set of frames;

processing the first set of frames using a second trained model configured to identify a likelihood of the subject exhibiting the predetermined behavioral action;

based on the processing of the first set of frames by the second trained model, determining a second probability of the subject exhibiting the predetermined behavioral action in the first frame;

based on the processing of the reflected set of frames by the first trained model, determining a third probability of the subject exhibiting the predetermined behavioral action in a second frame of the reflected set of frames, the second frame corresponding to the first frame;

processing the reflected set of frames using the second trained model;

based on the processing of the reflected set of frames by the second trained model, determining a fourth probability of the subject exhibiting the predetermined behavioral action in the second frame; and

using the first probability, the second probability, the third probability, and the fourth probability, identifying a first label for the first frame, the first label indicating that the subject exhibits the predetermined behavioral action.

13 . The computer-implemented method of claim 12 , further comprising:

determining a rotated set of frames by rotating the first set of frames;

processing the rotated set of frames using the first trained model;

based on the processing of the rotated set of frames by the first trained model, determining a third probability of the subject exhibiting the predetermined behavioral action in a second frame of the rotated set of frames, the second frame corresponding to the first frame;

processing the rotated set of frames using the second trained model;

based on the processing of the rotated set of frames by the second trained model, determining a fourth probability of the subject exhibiting the predetermined behavioral action in the second frame; and

identifying the first label using the first probability, the second probability, the third probability and the fourth probability.

14 . The computer-implemented method of claim 12 , wherein the first trained model and the second trained model are neural network models, the first trained model is initialized using a first set of parameters, and the second trained model is initialized using a second set of parameters different than the first set of parameters.

15 . The computer-implemented method of claim 12 , further comprising:

processing the first set of frames using a third trained model configured to identify a likelihood of the subject exhibiting a predetermined behavioral action;

based on the processing of the first set of frames by the third trained model, determining a third probability of the subject exhibiting the predetermined behavioral action in the first frame;

processing the first set of frames using a fourth trained model configured to identify a likelihood of the subject exhibiting the predetermined behavioral action;

based on the processing of the first set of frames by the fourth trained model, determining a fourth probability of the subject exhibiting the predetermined behavioral action in the first frame; and

identifying the first label using the first probability, the second probability, the third probability, and the fourth probability.

16 . The computer-implemented method of claim 12 , further comprising:

identifying a second set of frames from the video data;

processing the second set of frames using the first trained model;

based on the processing of the second set of frames by the first trained model, determining a third probability of the subject exhibiting the predetermined behavioral action in a third frame of the second set of frames;

processing the second set of frames using the second trained model;

based on the processing of the second set of frames by the second trained model, determining a fourth probability of the subject exhibiting the predetermined behavioral action in the third frame; and

using the third probability and the fourth probability, identifying a second label for the third frame, the second label indicating that the subject exhibits the predetermined behavioral action.

17 . The computer-implemented method of claim 16 , further comprising:

using at least the first label and the second label, generating an ethogram representing the predetermined behavioral action of the subject during a time period.

18 . The computer-implemented method of claim 12 , further comprising prior to receiving the video data:

receiving training data including a first plurality of video frames and a second plurality of video frames, each of the first plurality of video frames associated with a positive label indicating that the subject is exhibiting the predetermined behavioral action, and each of the second plurality of video frames associated with a negative label indicating that the subject is exhibiting a behavioral action that is not the predetermined behavioral action;

processing the training data using a first set of model parameters and first classifier model data to determine the first trained model; and

processing the training data using a second set of model parameters and second classifier model data to determine the second trained model.