IP Library › Granted Patent US 11,777,754

Granted Patent B1

US 11,777,754 · App. 18/049,190 · Granted Oct 3, 2023

Multi-person tracking and identification for robust framing experience

Inventors: Kui Zhang (Austin, TX); Raghavendra Balavalikar Krishnamurthy (Austin, TX); Rajen Bhatt (Pittsburgh, PA)

Assignee: Plantronics, Inc.

H04L12/1822 G06T7/248 G06V10/25 G06V20/46 G06V40/167 G06V40/172 G06T2207/30201

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,777,754
App. No.: 18/049,190
Filed: Oct 24, 2022
Granted: Oct 3, 2023
Kind: B1
Examiner: NGUYEN, ANGELA
Art Unit: 2442
USPC: 709/204

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

The number of persons in a still image acquired during a videoconference may change over time due to movement or persons entering or exiting a conference room. Minimizing the number of person identifiers that are tracked is beneficial for framing purposes. Typically, a stream of frames is sent to a far side, such as another endpoint device, for viewing. If the number of persons identified keeps changing, the composition of the stream of frames will be constantly changing as well. By using person identification and the use of timers, movement by the same person or a temporary change in a number or set of persons may be detected without making changes to the number of identifiers and persons being framed. It is only when a change has persisted past a period of time that changes are made to the frames, thereby improving the overall viewing experience.

Claims (120)

1. A system comprising:

a first endpoint device ( 910 ) comprising:

one or more cameras ( 942 );

a first memory ( 950 ) storing first computer-executable instructions; and

a first hardware processor ( 920 ) coupled to the one or more cameras ( 942 ) and the first memory ( 950 ), the first hardware processor ( 920 ) configured to execute the first computer-executable instructions to:

acquire, at a first time, a first image from the one or more cameras ( 710 );

determine a first set of persons depicted in the first image ( 720 );

determine an identifier associated with each person in the first set of persons ( 740 );

add the identifier associated with the each person in the first set of persons to an identification pool ( 745 );

acquire, at a second time, a second image from the one or more cameras ( 755 );

determine a second set of persons depicted in the second image ( 760 );

detect a change between the first set of persons depicted in the first image and the second set of persons depicted in the second image ( 765 );

determine a first threshold of time ( 860 ); and

maintain an identifier of a person associated with the change in the identification pool at least until an elapsed time since the second time exceeds the first threshold of time ( 865 , 870 , 880 ).

2. The system of claim 1 , the first computer-executable instructions to determine the first set of persons depicted in the first image further comprise instructions to:

determine a bounding box associated with at least a portion of the each person in the first set of persons ( 725 , 730 ), wherein the at least a portion of the each person comprises a head, a face, or a body;

determine one or more features associated with each bounding box ( 735 ); and

use the one or more features to determine the identifiers associated with the each person in the first set of persons ( 740 ).

3. The system of claim 1 , the first hardware processor to further execute the first computer-executable instructions to:

determine the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises a decrease in a total number of persons detected from the first time to the second time ( 845 );

determine an identifier associated with a person detected in the first image but not detected in the second image ( 870 ); and

remove the identifier associated with the person detected in the first image but not detected in the second image from the identification pool when a difference between the second time and the first time exceeds the first threshold of time ( 870 ).

4. The system of claim 3 , the first hardware processor to further execute the first computer-executable instructions to:

at the second time, set a tracking status associated with the person detected in the first image but not detected in the second image as lost ( 850 );

determine a second threshold of time ( 860 );

determine the elapsed time since the second time exceeds the second threshold of time ( 865 ); and

set the tracking status associated with the person detected in the first image but not detected in the second image as inactive ( 870 ).

5. The system of claim 3 , the first hardware processor to further execute the first computer-executable instructions to:

determine a first frame for the each person in the first set of persons, wherein the first frame for the each person in the first set of persons comprises a portion of the first image that depicts a face of that person ( 750 );

determine a first layout of the first frames ( 750 ); and

after the elapsed time since the second time exceeds the first threshold of time, remove the first frame associated with the person detected in the first image but not detected in the second image from the first layout ( 870 ).

6. The system of claim 1 , the first hardware processor to further execute the first computer-executable instructions to:

determine the change between the first set of persons in the first image and the second set of persons in the second image comprises a decrease in a number of persons detected from the first time to the second time ( 845 );

acquire, at a third time before the elapsed time since the second time exceeds the first threshold of time, a third image from the one or more cameras ( 875 );

determine a third set of persons depicted in the third image ( 875 );

determine the first set of persons depicted in the first image is the same as the third set of persons depicted in the third image ( 875 );

maintain the identifiers in the identification pool at the third time ( 880 ); and

maintain a layout of frames associated with the each person in the first set of persons from the first time to the third time ( 880 ).

7. The system of claim 6 , the first hardware processor to further execute the first computer-executable instructions to:

set, at the second time, a tracking status associated with a person detected in the first image but not detected in the second image as lost ( 850 );

determine a second threshold of time ( 860 );

determine that the person that was detected in the first image but not detected in the second image is again detected in the third image, prior to the elapsed time since the second time exceeding the second threshold of time ( 875 ); and

set the tracking status associated with the person that was detected in first image but not detected in the second image and again detected in the third image as active ( 880 ).

8. The system of claim 1 , wherein the first threshold of time is between 1 minute and 10 minutes ( 860 ).

9. The system of claim 1 , the first hardware processor to further execute the first computer-executable instructions to:

determine the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises an increase in a total number of persons detected from the first time to the second time ( 845 );

determine a tracking status associated with a person that was not detected in the first image but was detected in the second image was previously set to lost or inactive ( 820 , 825 );

retrieve an identifier associated with the person that was not detected in the first image but was detected in the second image ( 830 ); and

set the tracking status associated with the person that was not detected in the first image but was detected in the second image as active ( 830 ).

10. The system of claim 1 , the first hardware processor to further execute the first computer-executable instructions to:

determine the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises an increase in a number of persons detected from the first time to the second time ( 845 , 820 );

determine a tracking status associated with a person not detected in the first image but detected in the second image was not previously set to lost or inactive ( 825 );

generate an identifier for the person not detected in the first image but detected in the second image ( 840 );

add the identifier for the person not detected in the first image but detected in the second image to the identification pool ( 840 ); and

set the tracking status associated with the person not detected in the first image but detected in the second image as active ( 840 ).

11. The system of claim 1 , the first hardware processor to further execute the first computer-executable instructions to:

determine the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises a change in persons being tracked ( 810 );

determine if a tracking status set to inactive matches any of the persons depicted in the second image ( 820 , 825 );

if a match exists, retrieve an identifier associated with that person and change the track status for that person to active ( 830 ); and

if no match exists, generate a new identifier associated with that person, add the new identifier to the identification pool, and set a tracking status for that person to active ( 840 ).

12. A method comprising:

acquiring, at a first time, a first image from one or more cameras of an endpoint device ( 710 );

determining a first set of persons depicted in the first image ( 720 );

determining an identifier associated with each person in the first set of persons ( 740 );

adding, to an identification pool, the identifier associated with the each person in the first set of persons ( 740 );

acquiring, at a second time, a second image from the one or more cameras ( 755 );

determining a second set of persons depicted in the second image ( 760 );

detecting a change between the first set of persons depicted in the first image and the second set of persons depicted in the second image ( 765 ); and

maintaining an identifier of a person associated with the change in the identification pool at least until an elapsed time since the second time exceeds a first threshold of time ( 865 , 870 , 880 ).

13. The method of claim 12 , further comprising:

determining the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises a decrease in a total number of persons detected from the first time to the second time ( 845 );

determining an identifier associated with a person detected in the first image but not detected in the second image ( 870 ); and

removing the identifier associated with the person detected in the first image but not detected in the second image from the identification pool when a difference between the second time and the first time exceeds the first threshold of time ( 870 ).

14. The method of claim 12 , further comprising:

determining the change between the first set of persons in the first image and the second set of persons in the second image comprises a decrease in a number of persons detected from the first time to the second time ( 845 );

acquiring, at a third time before the elapsed time since the second time exceeds the first threshold of time, a third image from the one or more cameras ( 875 );

determining a third set of persons depicted in the third image ( 875 );

determining the first set of persons depicted in the first image is the same as the third set of persons depicted in the third image ( 875 ); and

maintaining the identifiers in the identification pool from the first time to the second time to the third time ( 880 ).

15. The method of claim 12 , further comprising:

determining the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises an increase in a total number of persons detected from the first time to the second time ( 845 );

determining whether a tracking status associated with a person that was not detected in the first image but detected in the second image was previously set to lost or inactive ( 820 , 825 );

if the tracking status associated with the person that was not detected in the first image but was detected in the second image was previously set to lost or inactive, retrieving an identifier associated with that person and setting a tracking status associated with that person as active ( 830 ); and

if the tracking status associated with the person that was not detected in the first image but was detected in the second image was not previously set to lost or inactive, generating an identifier for that person, adding the identifier for that person to the identification pool, and setting the tracking status associated with that person to active ( 840 ).

16. The method of claim 12 , further comprising:

determining the change between the first set of persons depicted in the first image and the second set of persons depicted in the second image comprises a change in persons being tracked ( 810 );

determining if a tracking status set to lost or inactive matches any of the persons depicted in the second image ( 820 , 825 );

if a match exists, retrieving an identifier associated with that person and changing the track status for that person to active ( 830 ); and

if no match exists, generating a new identifier associated with that person, adding the new identifier to the identification pool, and setting the track status for that person to active ( 840 ).

17. A system comprising:

a first endpoint device ( 910 ) comprising:

one or more cameras ( 942 );

a first memory ( 950 ) storing first computer-executable instructions; and

acquire, at a first time, a first image from the one or more cameras ( 710 );

determine a first set of persons depicted in the first image ( 720 );

determine an identifier associated with each person in the first set of persons ( 740 );

add the identifiers associated with the each person in the first set of persons to an identification pool ( 745 );

set a tracking status for the each person in the first set of persons to active ( 745 );

acquire, at a second time, a second image from the one or more cameras ( 755 );

determine a second set of persons depicted in the second image ( 760 );

detect a change between the first set of persons depicted in the first image and the second set of persons depicted in the second image ( 765 ); and

maintain an identifier of a person associated with the change in the identification pool for a duration of time after the change is detected ( 865 , 870 , 880 ).

18. The system of claim 17 , the first hardware processor to further execute the first computer-executable instructions to:

set the tracking status associated with a person detected in the first image but not detected in the second image as lost ( 850 );

determine an identifier associated with the person detected in the first image but not detected in the second image ( 870 );

set the tracking status associated with the person detected in the first image but not detected in the second image as inactive when the difference between the second time and the first time exceeds the first threshold of time ( 870 ).

19. The system of claim 17 , the first hardware processor to further execute the first computer-executable instructions to:

acquire, at a third time before an elapsed time since the second time exceeds a first threshold of time, a third image from the one or more cameras ( 875 );

determine a third set of persons depicted in the third image ( 875 );

determine the first set of persons depicted in the first image is the same as the third set of persons depicted in the third image ( 875 ); and

maintain the identifiers in the identification pool from the first time to the second time to the third time ( 880 ).

20. The system of claim 17 , the first hardware processor to further execute the first computer-executable instructions to:

determine whether a tracking status associated with a person that was not detected in the first image but was detected in the second image was previously set to lost or inactive ( 820 , 825 );

if the tracking status associated with the person that was not detected in the first image but was detected in the second image was previously set to lost or inactive, retrieve an identifier associated with that person and set the tracking status associated with that person as active ( 830 ); and

if the tracking status associated with the person that was not detected in the first image but was detected in the second image was not previously set to lost or inactive, generate an identifier for that person, add the identifier for that person to the identification pool, and set the tracking status associated with that person to active ( 840 ).

Assignments (2)

NUNC PRO TUNC ASSIGNMENT Recorded Nov 13, 2023

From: PLANTRONICS, INC.

To: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.

Reel/Frame 065549/0065 →

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2022

From: KRISHNAMURTHY, RAGHAVENDRA BALAVALIKAR; BHATT, RAJEN; ZHANG, KUI

To: PLANTRONICS, INC.

Reel/Frame 061530/0116 →

Cited By (2)

US 12,333,854 US 12,726,374