Animal visual identification, tracking, monitoring and assessment systems and methods thereof
An animal management system has one or more imaging devices, and a computing device coupled to the one or more image devices for receiving one or more images captured by the one or more imaging devices, processing at least one image using an artificial intelligence (AI) pipeline for: (i) detecting and locating in the image one or more animals, (ii) for each detected animal: (a) generating at least one section of the detected animal, (b) determining a plurality of key points in each section, (c) generating an embedding for each section based on the plurality of key points in the section, and (d) combining the embeddings for generating an identification of the detected animal with a confidence score. Key points and bounding boxes may also have associated confidence scores.
1. As non-transitory computer readable medium storing thereon program instructions, which when executed by at least one processor of a computing device, configure the at least one processor for performing a method for animal management, the method comprising:
receiving one or more images captured by one or more imaging devices;
processing the one or more images using an artificial intelligence (AI) pipeline having a plurality of AI models for:
detecting and locating in the one or more images at least one animal; and
for one of the detected animals in an image of the one or more images:
processing the image of the detected animal to generate one or more bounding boxes corresponding to one or more sections of the detected animal;
generating embeddings using at least one AI model for at least one of the bounding boxes corresponding to at least one section of the detected animal; and
combining the embeddings for generating an identification or classification of an activity of the detected animal,
wherein the at least one AI model comprises a Transformer Network that is used to identify the detected animal based on embeddings from feature-dense vectors with high dimensionality that are generated from the one or more images which show an appearance of the detected animal obtained from one or more angles.
2. The non-transitory computer readable medium of claim 1 , wherein after generating one or more bounding boxes and before generating the embeddings, the method comprises:
determining, in a bounding box of each section using one of the AI models, a plurality of key points corresponding to anatomical features of the detected animal used to identify the detected animal, wherein the plurality of key points is determined in response to a type of the detected animal and a part of the detected animal that the section corresponds to; and
normalizing each of the one or more sections, based on the determined key points for the one or more sections, by projecting the determined key points of each of the one or more sections to a common plane based on a reference template indicating where key points are expected to be located based on the section and species of the detected animal.
3. The non-transitory computer readable medium of claim 1 , wherein a given animal is detected and located in the image by processing the image using a backbone and a classifier to detect the given animal and define a bounding box and optionally a segmentation mask that indicates a location of the detected given animal in the image where the backbone is provided by DetectNet, Resnet, or DarkNet.
4. The non-transitory computer readable medium of claim 1 , wherein the one or more sections comprise a front-side head section, a left-side head section, a right-side head section, a top body section, a left-side body section, a right-side body section, a rear-side section, a drop tag section, a tail section, an udder section, a teat section, one or more knee sections, one or more hoof sections, a scrotum section, and/or one or more leg sections.
5. The non-transitory computer readable medium of claim 1 , wherein each section is defined by a bounding box having a confidence score indicating an accuracy of the bounding box.
6. The non-transitory computer readable medium of claim 2 , wherein the plurality of key points corresponds to features of one or more sections of the detected animal including one or more eyes, one or more ears, a nose, a mouth, a top of head, a bottom of head, a jaw, a hip, a lip, a neck, one or more joints or any combination thereof.
7. The non-transitory computer readable medium of claim 1 , wherein the generated identification of the detected animal is a face identification, and the one or more sections are one or more of a front-side head section, a left-side head section and a right-side head section.
8. The non-transitory computer readable medium of claim 6 , wherein defining the bounding box for each of the front-side head section, the left-side head section and the right-side head section includes one or more of cropping the head section, scaling the head section, and adding a margin to the head section if the bounding box for the head section is not square.
9. The non-transitory computer readable medium of claim 6 , wherein the AI models comprise a Face ID model to process the front-side head section, the left-side head section and/or the right-side head section to determine the identification for the detected animal.
10. The non-transitory computer readable medium of claim 1 , wherein the plurality of AI models further comprise one or more of a Pre-Trained Neural Network (PTNN), Transfer Learning, Convolutional Neural Network (CNN), Deep Neural Networks (DNN), a Deep Convolutional Neural Network (DCNN), a Fully Connected Network (FCN), a Recurrent Neural Network (RNN), Long Term Short Term (LSTM), and/or Pyramid Network.
11. The non-transitory computer readable medium of claim 1 , wherein the plurality of AI models further comprise a Triplet Loss Siamese Network for generating the embeddings for each section and the Triplet Loss Siamese is trained using a triplet loss method for each section that it is provided with.
12. The non-transitory computer readable medium of claim 11 , wherein the triplet loss method includes training the Siamese Network using tuples of anchor, positive and negative pre-processed images.
13. The non-transitory computer readable medium of claim 12 , wherein the pre-processed images are generated by normalizing and aligning the one or more sections by matching a determined plurality of key points for the one or more sections to one or more corresponding key point templates.
14. The non-transitory computer readable medium of claim 12 , wherein the pre-processed images are generated by additionally blocking portions of the section corresponding to a drop tag of the detected animal when the one or more sections include a front head section, a right side head section and/or a left side head section where the drop tag is visible.
15. The non-transitory computer readable medium of claim 1 , wherein the method further comprises masking portions of a section of the image corresponding to a drop-tag of the animal when the image section is a front head, left head or right head section that includes the drop-tag.
16. The non-transitory computer readable medium of claim 1 , wherein the one or more sections comprise a drop tag section that is processed by a drop tag ID model to produce a ground truth identification; and wherein the computing device is further configured for comparing the generated identification with the ground truth identification for verification.
17. The non-transitory computer readable medium of claim 1 , wherein at least one of the AI models are re-trained when the confidence level from the generated identifications trend lower than a threshold accuracy rate.
18. The non-transitory computer readable medium of claim 2 , wherein the method further comprises generating one or more animal assessments for a characteristic of the detected animal and the determined key points in the one or more sections of the detected animal, where the determined key points have locations that are physically related to the characteristic being assessed.
19. The non-transitory computer readable medium of claim 1 , wherein the generated identification of the detected animal is: (a) based on a tail identification, and the one or more sections includes a tail section of the detected animal, (b) based a scrotum identification, and the one or more sections includes a scrotum section of the detected animal, and/or (c) based on a hide identification, and the one or more sections are one or more of a top body section, a left-side body section and a right-side body section of the detected animal.
20. The non-transitory computer readable medium of claim 2 , wherein: (a) the plurality of key points is generated with a corresponding confidence score, links between successive adjacent key points that are approximately linear are formed and angles between adjacent links are determined to form key angles with a confidence score; and/or (b) the computing device is configured to generate a given bounding box for a given detected animal in the image and a confidence score for the given bounding box.
21. The non-transitory computer readable medium of claim 2 , wherein the computing device is further configured to annotate the image with the bounding box for the detected animal, the sections for the detected animal the plurality of key points for the detected animal, and/or confidence levels obtained for the bounding box, the sections and/or the plurality of key points for the detected animal.
22. The non-transitory computer readable medium of claim 2 , wherein the detected animal is a cow and the AI pipeline comprises a classification layer having a BRD AI model that is trained to detect cows that have a likelihood of developing BRD, wherein the BRD AI model is adapted to receive at least one head section and the key points thereof for the detected animal as inputs and generate a binary classification result to indicate whether the cow has BRD or not.
23. The non-transitory computer readable medium of claim 2 , wherein the AI pipeline comprises a classification layer having a stress AI model that is trained to determine a level of stress that the detected animal is incurring, wherein the stress AI model is adapted to receive at least one head section and/or a tail section of the detected animal and the key points of these sections and generate an ordinal output result for the level of stress being incurred by the detected animal.
24. The non-transitory computer readable medium of claim 2 , wherein the AI pipeline comprises a classification layer having a lameness AI model that is trained to determine a level of lameness that the detected animal is incurring, wherein the lameness AI model is adapted to receive at least one leg section and key points thereof of the detected animal and generate an ordinal output result for the level of lameness being incurred by the detected animal.
25. The non-transitory computer readable medium of claim 2 , wherein the AI pipeline comprises a classification layer having an activity AI model that is trained to determine when the detected animal is engaging in an activity, wherein the activity AI model is adapted to receive at least one leg section and key points thereof of the detected animal and generate an output result indicating whether the detected animal is engaging in the activity.
26. A non-transitory computer readable medium storing thereon program instructions, which when executed by at least one processor of a computing device, configure the at least one processor for performing a method for animal management animal management, the method comprising:
receiving one or more images captured by the one or more imaging devices;
processing at least one image from the one or more images using an artificial intelligence (AI) pipeline having several layers that each have a unique purpose and include one or more AI models, the AI pipeline including:
a detection level that uses a backbone as an object detector configured to detect, and locate in the at least one image one or more animals and a classifier to classify the one or more detected animals to indicate the species thereof, wherein the backbone is implemented using DetectNet, Resnet, or DarkNet and optionally a segmentation mask is used to indicate a location of the at least one image;
a bounding box level having one or more one or more Bounding Box AI models that are configured to generate one or more bounding boxes corresponding to one or more sections for the one or more detected animals based on the classification of the one or more detected animals where the bounding boxes include bounding boxes that are not square; and
an identification layer having one or more Identification AI models that are configured to generate embeddings in higher dimensional space from pixel values for each of the one or more sections of the image; and combine the embeddings for generating an identification of the one or more detected animals,
wherein the plurality of AI models comprise a Transformer Network that is used to associate a sound made by a given detected animal with an identification of the given detected animal.
27. The non-transitory computer readable medium of claim 26 , wherein the AI pipeline further includes a classification level having one or more Classification AI models to generate one or more animal assessments for the detected one or more animals based on input from pixel values from the one or more images.
28. An animal identification system for identifying an animal in an image, wherein the system comprises:
a memory configured to store one or more AI models; and
a computing device coupled to the memory and configured to:
receive at least one section of an animal, the at least one section being defined by pixel values in a bounding box that corresponds to the least one section of an animal determined from an image including the animal;
generate embeddings for the at least section of the animal using the one or more AI models; and
combine the embeddings to generate an identification or a classification of the animal using the one or more AI models,
wherein the one or more AI models comprise a Transformer Network that is used to classify a motion of the detected animal over time.
29. A non-transitory computer readable medium storing thereon program instructions, which when executed by at least one processor, configure the at least one processor for performing a method for animal management comprising:
identifying a visually observable animal behavioral trait of interest including lameness;
receiving, at a computing device, one or more images captured by one or more imaging devices;
processing the one or more images using an artificial intelligence (AI) pipeline having a plurality of AI models for:
detecting, locating and classifying one or more animals in the one or more images; and
for at least one detected animal:
processing an image of the at least one detected animal from the one or more images to generate one or more bounding boxes corresponding to one or more sections based on classification of a species of the detected animal;
generating embeddings from pixel values for the one or more bounding boxes of the image;
combining the embeddings for generating an identification of the detected animal, optionally with a first confidence score, and/or determining if the detected animal displays the visually observable trait, optionally with a second confidence score,
wherein the plurality of AI models comprise a Transformer Network that is used to determine a level of lameness for the detected animal.