IP Library Granted Patent US 10,255,483
Granted Patent B2
US 10,255,483 · App. 16/100,878 · Granted Apr 9, 2019

Computer systems and computer-implemented methods specialized in tracking faces across visual representations

Inventors: Yury Hushchyn (Vilnius, LT); Aliaksei Sakolski (Minsk, BY); Alexander Poplavsky (Minsk, BY)
Assignee: Banuba Limited
G06K9/00255G06K9/00201G06K9/00268G06K9/2063
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,255,483
App. No.
16/100,878
Granted
Apr 9, 2019
Kind
B2
Abstract

Embodiments directed towards systems and methods for tracking a human face present within a video stream are described herein. In some embodiments, the exemplary illustrative methods and the exemplary illustrative systems of the present invention are specifically configured to process image data to identify and align the presence of a face in a particular frame.

Claims (78)

1. A method, comprising:

obtaining, by at least one processor, a plurality of sequential visual representations having a face of at least one subject;

applying, by the at least one processor, a face detection algorithm to detect an initial presence of the face of the at least one subject within an initial visual representation of the plurality of sequential visual representations;

wherein the initial visual representation is a first visual representation in which the initial presence of the face of the at least one subject has been detected for a first time in the plurality of sequential visual representations;

wherein the initial presence of the face of the at least one subject within the initial visual representation is defined by a first plurality of initial conditions;

constructing, by the at least one processor, a first face model of the face of the at least one subject based, at least in part, on the first plurality of initial conditions;

wherein the initial presence of the face of the at least one subject is detected once prior to the constructing of the first face model;

tracking, by the at least one processor, the face of the at least one subject in at least one subsequent visual representation of the plurality of sequential visual representations, by executing one or more times at least the following:

i) modifying one or more initial conditions of the first plurality of initial conditions to generate a plurality of subsequent face models where each subsequent face model is a prediction of how the face of the at least one subject would appear in the at least one subsequent visual representation; and

ii) applying at least one special filter to each subsequent face model of the plurality of subsequent face models to generate a plurality of updated subsequent face models;

determining, by the at least one processor, one of:

i) a refined subsequent face model from the plurality of updated subsequent face models or

ii) a lack of the refined subsequent face model;

wherein the refined subsequent face model is a predictive face model that adequately represents a respective presence of the face of the at least one subject in the at least one subsequent visual representation of the plurality of sequential visual representations;

outputting, by the at least one processor, a second plurality of initial conditions to be utilized for tracking the face of the at least one subject in at least one next subsequent visual representation of the plurality of sequential visual representation;

wherein the second plurality of initial conditions is:

i) a third plurality of initial conditions that is representative of the refined subsequent face model within the at least one subsequent visual representation; or

ii) a fourth plurality of initial conditions that is representative of a second face model that has been constructed, based, at least in part, on the fourth plurality of initial conditions determined after the lack of the refined subsequent face model has been determined; and

wherein the fourth plurality of initial conditions has been determined by applying the face detection algorithm to the at least one subsequent visual representation to detect the respective presence of the face of the at least one subject in the at least one subsequent visual representation of the plurality of sequential visual representations.

2. The method of claim 1 , wherein each plurality of initial conditions comprises:

i) respective face positional data, being representative of a respective position of the face of the at least one subject within a respective visual representation, and

ii) respective face model data, comprising at least one of:

1) a plurality of initial latent variables or

2) a plurality of initial multi-dimensional points.

3. The method of claim 1 , wherein the respective face positional data comprises one or more initial regions-of-interest (ROI).

4. The method of claim 1 , wherein the one or more times is between 2 and 25 times.

5. The method of claim 1 , wherein the plurality of subsequent face models are uncorrelated subsequent face models.

6. The method of claim 1 , wherein the at least one subject is a person.

7. The method of claim 1 , wherein the plurality of sequential visual representations comprises at least one of:

i) a plurality of frames of a video input,

ii) a plurality of images, or

iii) a combination of one or more frames of the video input and one or more images.

8. The method of claim 7 , wherein the video input is a real-time video stream.

9. The method of claim 8 , wherein the real-time video stream is a live video stream.

10. The method of claim 1 , wherein the plurality of sequential visual representations is obtained by utilizing a camera of a portable electronic device.

11. The method of claim 10 , wherein the at least one processor is a processor of the portable electronic device.

12. The method of claim 1 , wherein the modifying one or more initial conditions of the first plurality of initial conditions is modifying one or more initial conditions of the first plurality of initial conditions in a pre-determined manner.

13. The method of claim 1 , wherein the determining that the refined subsequent face model adequately represents the respective presence of the face of the at least one subject in the at least one subsequent visual representation of the plurality of sequential visual representations is based, at least in part, on testing a stability of the refined subsequent face model over a number of landmark points.

14. The method of claim 13 , wherein the number of landmark points is based on a particular landmark methodology.

15. A system comprising:

a portable electronic device having a camera, wherein the camera is configured to acquire a plurality of sequential visual representations having a face of at least one subject;

a non-transient computer memory storing software instructions; and

at least one processor configured, when executing one or more of the software instructions, to perform at least the following:

obtaining a plurality of sequential visual representations having a face of at least one subject;

applying a face detection algorithm to detect an initial presence of the face of the at least one subject within an initial visual representation of the plurality of sequential visual representations;

wherein the initial visual representation is a first visual representation in which the initial presence of the face of the at least one subject has been detected for a first time in the plurality of sequential visual representations;

wherein the initial presence of the face of the at least one subject within the initial visual representation is defined by a first plurality of initial conditions;

constructing a first face model of the face of the at least one subject based, at least in part, on the first plurality of initial conditions;

wherein the initial presence of the face of the at least one subject is detected once prior to the constructing of the first face model;

tracking the face of the at least one subject in at least one subsequent visual representation of the plurality of sequential visual representations, by executing one or more times at least the following:

i) modifying one or more initial conditions of the first plurality of initial conditions to generate a plurality of subsequent face models where each subsequent face model is a prediction of how the face of the at least one subject would appear in the at least one subsequent visual representation; and

ii) applying at least one special filter to each subsequent face model of the plurality of subsequent face models to generate a plurality of updated subsequent face models;

determining one of:

i) a refined subsequent face model from the plurality of updated subsequent face models or

ii) a lack of the refined subsequent face model;

wherein the refined subsequent face model is a predictive face model that adequately represents a respective presence of the face of the at least one subject in the at least one subsequent visual representation of the plurality of sequential visual representations;

outputting a second plurality of initial conditions to be utilized for tracking the face of the at least one subject in at least one next subsequent visual representation of the plurality of sequential visual representation;

wherein the second plurality of initial conditions is:

i) a third plurality of initial conditions that is representative of the refined subsequent face model within the at least one subsequent visual representation; or

ii) a fourth plurality of initial conditions that is representative of a second face model that has been constructed, based, at least in part, on the fourth plurality of initial conditions determined after the lack of the refined subsequent face model has been determined; and

wherein the fourth plurality of initial conditions has been determined by applying the face detection algorithm to the at least one subsequent visual representation to detect the respective presence of the face of the at least one subject in the at least one subsequent visual representation of the plurality of sequential visual representations.

16. The system of claim 15 , wherein each plurality of initial conditions comprises:

i) respective face positional data, being representative of a respective position of the face of the at least one subject within a respective visual representation, and

ii) respective face model data, comprising at least one of:

1) a plurality of initial latent variables or

2) a plurality of initial multi-dimensional points.

17. The system of claim 15 , wherein the respective face positional data comprises one or more initial regions-of-interest (ROI).

18. The system of claim 15 , wherein the one or more times is between 2 and 25 times.

19. The system of claim 15 , wherein the plurality of subsequent face models are uncorrelated subsequent face models.

20. The system of claim 15 , wherein the at least one subject is a person.

21. The system of claim 15 , wherein the plurality of sequential visual representations comprises at least one of:

i) a plurality of frames of a video input,

ii) a plurality of images, or

iii) a combination of one or more frames of the video input and one or more images.

22. The system of claim 21 , wherein the video input is a real-time video stream.

23. The system of claim 15 , wherein the modifying one or more initial conditions of the first plurality of initial conditions is modifying one or more initial conditions of the first plurality of initial conditions in a pre-determined manner.

24. The system of claim 15 , wherein the determining that the refined subsequent face model adequately represents the respective presence of the face of the at least one subject in the at least one subsequent visual representation of the plurality of sequential visual representations is based, at least in part, on testing a stability of the refined subsequent face model over a number of landmark points.

25. The system of claim 24 , wherein the number of landmark points is based on a particular landmark methodology.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2025
From: BANUBA LIMITED
To: BANUBA FZCO
Reel/Frame 070195/0149 →
Continuity (8)
Continuation 15881353 · Jan 26, 2018
Provisional Application 62451450 · Jan 27, 2017
Provisional Application 62451404 · Jan 27, 2017
Provisional Application 62451382 · Jan 27, 2017
Provisional Application 62451328 · Jan 27, 2017
Provisional Application 62451357 · Jan 27, 2017
Provisional Application 62451281 · Jan 27, 2017
Related Publication 20180349680A1 · Dec 6, 2018
Cited By (1)
US 12,645,290