System and method for automatically discovering, characterizing, classifying and semi-automatically labeling animal behavior and quantitative phenotyping of behaviors in animals

1. A method for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area, comprising:

(a) using a 3D depth camera to obtain a video stream having a plurality of images of the experimental area with the animal in the experimental area, the images having both area and depth information;

(b) removing background noise from each of the plurality of images to generate processed images having light and dark areas;

(c) determining contours of the light areas in the plurality of processed images;

(d) extracting parameters from both area and depth image information within the contours to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time and identifying the animal's head and tail based on a derivative of the contour curvature;

(e) clustering the data points at each specific time to output a set of clusters that are segmented form each other so that each cluster represents an animal behavior;

(f) assigning each cluster a label that represents an animal behavior; and

(g) outputting a visual representation of the set of clusters and corresponding labels.

2. The method of claim 1 wherein step (b) comprises:

(b1) using the 3D depth camera to obtain a video stream having a plurality of baseline images of the experimental area without an animal present;

(b2) generating a median of the baseline images to form a baseline depth image;

(b3) subtracting the baseline depth image from each of the plurality of images obtained in step (a) to produce a plurality of difference images;

(b4) performing a median filtering operation on each difference image to generate a filtered difference image; and

(b5) removing image data that is less than a predetermined threshold from each of the plurality of filtered difference images to generate the processed images.

3. The method of claim 1 , wherein step (c) comprises determining contours of all light regions in each processed image with a contour detection algorithm and tracking each contour with a Kalman filter.

4. The method of claim 1 , wherein the label for each cluster is requested by a user interface after displaying video data representing each cluster.

5. The method of claim 1 , wherein the behavior comprises a quantitative behavior primitive.

6. An apparatus for automatically discovering, characterizing and classifying the behavior of an animal in an experimental area, comprising:

a 3D depth camera that generates a video stream having a plurality of images of the experimental area with the animal in the experimental area, the images having both area and depth information;

a data processing system having a processor and a memory containing program code which, when executed:

(a) removes background noise from each of the plurality of images to generate processed images having light and dark areas;

(b) determines contours of the light areas in the plurality of processed images;

(c) extracts parameters from both area and depth image information within the contours to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time and identifies the animal's head and tail based on a derivative of the contour curvature; and

(d) clusters the data points to output a set of clusters that each represent an animal behavior;

(e) assigns each cluster a label that represents an animal behavior; and

(f) outputs a visual representation of the set of clusters and corresponding labels.

7. The apparatus of claim 6 , wherein the 3D depth camera obtains a video stream having a plurality of baseline images of the experimental area without an animal present and wherein step (a) comprises:

(a1) generating a median of the baseline images to form a baseline depth image;

(a2) subtracting the baseline depth image from each of the plurality of images obtained in step (a) to produce a plurality of difference images;

(a3) performing a median filtering operation on each difference image to generate a filtered difference image; and

(a4) removing image data that is less than a predetermined threshold from each of the plurality of filtered difference images to generate the processed images.

8. The apparatus of claim 6 , wherein the label for each cluster is requested after displaying video data representing each cluster.

9. The apparatus of claim 6 , wherein the behavior comprises a quantitative behavior primitive.

10. The apparatus of claim 6 , wherein step (b) comprises determining contours of all light regions in each processed image with a contour detection algorithm and tracking each contour with a Kalman filter.

11. The apparatus of claim 6 wherein in step (c) parameters extracted from area information with each contour include at least one of perimeter, surface area rotation angle and length.

12. The apparatus of claim 6 , wherein in step (c) parameters extracted from depth information with each contour include at least one of height, width, depth, velocity spine curvature and limb position.

13. The apparatus of claim 6 , wherein step (d) comprises reducing covariance between data points and clustering the reduced covariance data points with a clustering method.

14. The apparatus of claim 6 , wherein covariance between data points is reduced by reducing dimensionality of each data point.

15. The apparatus of claim 14 , wherein the dimensionality of each data point is reduced by applying at least one of principal components analysis, singular value decomposition, independent components analysis and locally linear embedding to the points.