Context-based speaker counter for a speaker diarization system
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining the number of speakers in a video and a corresponding audio using visual context. In one aspect, a method includes detecting within the video multiple speakers, determining a bounding box for each detected speaker that includes the detected person and objects within a threshold distance of the detected person in an image frame, determining a unique descriptor for that person based in part on image information depicting the objects within the bounding box, determining a cardinality of unique speakers in the video, providing to the speaker diarization system the cardinality of unique speakers.
1 . A method performed by data processing apparatus, the method comprising:
obtaining a video that includes a plurality of image frames and corresponding audio;
detecting, within the video, multiple persons speaking depicted in the video without using biometric information to identify the persons speaking, the detection of each person based on the face of the person speaking detected in a face detection bounding box;
in response to detecting the multiple persons speaking depicted in the video, determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame, the bounding box larger than the face detection bounding box and including the face detection bounding box;
determining, for each detected person, from image information included within the bounding box for the detected person and without using biometric data that identifies the detected person, a unique descriptor for that person, the unique descriptor based in part on image information depicting the objects within the bounding box, each unique descriptor for the detected person not including biometric data that identifies the person, including determining a histogram of pixel activations for each of multiple frames, the histogram of pixel activations determined from local factors from the image information included within the bounding box;
determining a cardinality of unique descriptors determined for the video, including:
generating histogram clusters of the histograms, wherein each histogram cluster includes only histogram that are within a threshold distance of each other; and
determining the cardinality of unique descriptors as the number of histogram clusters;
providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors.
2 . The method of claim 1 , wherein:
detecting, within the video, multiple persons depicted in the video, comprises detecting faces within the video; and
determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame comprises determining a bounding box that is a multiple of a face detection bounding box that is inclusive of a minimum portion of the image frame required to detect the face of the detected person.
3 . The method of claim 1 , wherein:
detecting, within the video, multiple persons depicted in the video, comprises detecting a body that includes detecting the position of a head and torso of each person; and
determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame comprises determining a bounding box that is a multiple of a body detection bounding box that is inclusive of a minimum portion of the image frame required to detect the at least head and torso of the detected person.
4 . The method of claim 1 , wherein determining a cardinality of unique descriptors comprises:
providing, to a cardinality estimating model, the unique descriptors; and
receiving, from the cardinality estimating model, an estimate of the cardinality unique descriptors.
5 . The method of claim 1 , wherein determining a cardinality of unique descriptors determined for the video comprises:
determining multiple cardinalities of unique descriptors; and
for each cardinality of the multiple cardinalities, determining a confidence value that is indicative of a confidence that the cardinality is correct.
6 . The method of claim 1 , wherein providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors comprises providing the multiple cardinalities of unique descriptors, and for each unique descriptor, a confidence value for that unique descriptor.
7 . The method of claim 1 , wherein providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors comprises providing cardinality of unique descriptors that has a highest confidence value relative to the confidence values of all other unique descriptors.
8 . A system, comprising:
a data processing apparatus; and
a non-transitory computer readable medium storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
obtaining a video that includes a plurality of image frames and corresponding audio;
detecting, within the video, multiple persons speaking depicted in the video without using biometric information to identify the persons speaking, the detection of each person based on face of the person speaking detected in a face detection bounding box;
in response to detecting the multiple persons speaking depicted in the video, determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame, the bounding box larger than the face detection bounding box and including the face detection bounding box;
determining, for each detected person, from image information included within the bounding box for the detected person and without using biometric data that identifies the detected person, a unique descriptor for that person, the unique descriptor based in part on image information depicting the objects within the bounding box, each unique descriptor for the detected person not including biometric data that identifies the person, including determining a histogram of pixel activations for each of multiple frames, the histogram of pixel activations determined from local factors from the image information included within the bounding box;
determining a cardinality of unique descriptors determined for the video, including:
generating histogram clusters of the histograms, wherein each histogram cluster includes only histogram that are within a threshold distance of each other; and
determining the cardinality of unique descriptors as the number of histogram clusters;
providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors.
9 . The system of claim 8 , wherein:
detecting, within the video, multiple persons depicted in the video, comprises detecting faces within the video; and
determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame comprises determining a bounding box that is a multiple of a face detection bounding box that is inclusive of a minimum portion of the image frame required to detect the face of the detected person.
10 . The system of claim 8 , wherein:
detecting, within the video, multiple persons depicted in the video, comprises detecting a body that includes detecting the position of a head and torso of each person; and
determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame comprises determining a bounding box that is a multiple of a body detection bounding box that is inclusive of a minimum portion of the image frame required to detect the at least head and torso of the detected person.
11 . The system of claim 8 , wherein determining a cardinality of unique descriptors comprises:
providing, to a cardinality estimating model, the unique descriptors; and
receiving, from the cardinality estimating model, an estimate of the cardinality unique descriptors.
12 . The system of claim 8 , wherein determining a cardinality of unique descriptors determined for the video comprises:
determining multiple cardinalities of unique descriptors; and
for each cardinality of the multiple cardinalities, determining a confidence value that is indicative of a confidence that the cardinality is correct.
13 . The system of claim 8 , wherein providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors comprises providing the multiple cardinalities of unique descriptors, and for each unique descriptor, a confidence value for that unique descriptor.
14 . The system of claim 8 , wherein providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors comprises providing cardinality of unique descriptors that has a highest confidence value relative to the confidence values of all other unique descriptors.
15 . The system of claim 8 , wherein the speaker diarization system performs speech detection, uses the cardinality of unique to determine the number of speakers, and associates portions of the audio with a corresponding cardinality of speakers.
16 . A non-transitory computer readable medium storing instructions executable by a data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
obtaining a video that includes a plurality of image frames and corresponding audio;
obtaining a video that includes a plurality of image frames and corresponding audio;
detecting, within the video, multiple persons speaking depicted in the video without using biometric information to identify the persons speaking, the detection of each person based on face of the person speaking detected in a face detection bounding box;
in response to detecting the multiple persons speaking depicted in the video, determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame, the bounding box larger than the face detection bounding box and including the face detection bounding box;
determining, for each detected person, from image information included within the bounding box for the detected person and without using biometric data that identifies the detected person, a unique descriptor for that person, the unique descriptor based in part on image information depicting the objects within the bounding box, each unique descriptor for the detected person not including biometric data that identifies the person, including determining a histogram of pixel activations for each of multiple frames, the histogram of pixel activations determined from local factors from the image information included within the bounding box;
determining a cardinality of unique descriptors determined for the video, including:
generating histogram clusters of the histograms, wherein each histogram cluster includes only histogram that are within a threshold distance of each other; and
determining the cardinality of unique descriptors as the number of histogram clusters;
providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors.