Granted Patent B2

US 12,175,751 · App. 17/729,729 · Granted Dec 24, 2024

Inventors: Shaomin Xiong (Newark, CA); Toshiki Hirano (San Jose, CA)

Assignee: Sandisk Technologies, Inc.

G06V20/41G06T7/80G06V10/764G06V20/46G06V40/172

Abstract

Claims (141)

1. An end user device comprising:

at least one processor;

at least one memory;

a storage medium configured to store user video data;

an object tracker, stored in the at least one memory for execution by the at least one processor, configured to:

determine a first image object in a first frame of a video data stream from the user video data;

assign, to the first image object in the first frame, a first tracker identifier;

process subsequent frames of the video data stream to track movement of the first image object; and

associate each frame of the video data stream corresponding to the movement of the first image object with the first tracker identifier; and

a video group engine, stored in the at least one memory for execution by the at least one processor, configured to:

determine, using a group classifier and a target frame assigned the first tracking identifier, a first group identifier associated with the first image object to generate a bonding key configured to map the first tracker identifier to the first group identifier;

assign, based on the first tracker identifier and the bonding key, the first group identifier to a first plurality of frames of the video data stream, wherein:

the first plurality of frames of the video data stream are not processed by the group classifier;

the first group identifier corresponds to a first classifier identifier from the group classifier; and

the group classifier is trained using machine learning to determine at least one classifier identifier, including the first classifier identifier.

2. The end user device of claim 1 , further comprising:

a face detector, stored in the at least one memory for execution by the at least one processor, configured to:

process each frame of the video data stream to detect at least one human face; and

calculate, for the at least one human face, embeddings quantifying a plurality of features of the at least one human face, wherein detection of the at least one human face by the face detector initiates the object tracker.

3. The end user device of claim 1 , further comprising:

a bonding map, stored in the at least one memory, comprising:

a plurality of frame identifiers corresponding to a plurality of frames in the video data stream; and

a plurality of group identifiers, including the first group identifier, associated with at least one frame identifier of the plurality of frame identifiers, wherein the plurality of group identifiers is assigned to the plurality of frame identifiers based on a plurality of tracker identifiers from the object tracker.

4. The end user device of claim 3 , wherein the video group engine is further configured to:

process the plurality of frames using the object tracker to determine the plurality of frame identifiers; and

for each frame identifier and corresponding tracker identifier:

store, in the bonding map, that frame identifier;

determine a group identifier from the plurality of group identifiers that corresponds to that corresponding tracker identifier; and

store, in the bonding map, the determined group identifier associated with that frame identifier.

5. The end user device of claim 4 , wherein:

the bonding key is further configured to map the plurality of tracker identifiers to the plurality of group identifiers; and

determining the group identifier for each corresponding tracker identifier includes using that corresponding tracker identifier to index the corresponding group identifier in the bonding key.

6. The end user device of claim 4 , wherein the video group engine is further configured to:

determine tracker conditions for initiating the group classifier for a target frame of the plurality of frames;

determine, using the group classifier, a group null value corresponding to a detected image object not meeting a classification threshold for groups in the group classifier;

evaluate, in the bonding map and following processing the plurality of frames to generate the bonding map, tracker identifiers for frame identifiers of adjacent frames to determine a continuous set of a same tracker identifier value; and

assign, to the frame identifier for the target frame, a corresponding group classifier for the same tracker identifier value.

7. The end user device of claim 1 , further comprising:

the group classifier, stored in the at least one memory for execution by the at least one processor, configured to:

determine embeddings for at least one image object in the target frame from a calibration set of frames from the video data stream;

process the embeddings for the at least one image object through at least one set of machine learning parameters, wherein each set of the at least one set of machine learning parameters is assigned to a different classifier identifier; and

determine, based on processing the embeddings for the at least one image object, the first classifier identifier for the at least one image object; and

a video calibration engine, stored in the at least one memory for execution by the at least one processor, configured to:

determine the calibration set of frames from the video data stream, wherein the calibration set of frames includes fewer frames than all frames in the video data stream;

for each frame in the calibration set of frames:

select a corresponding target frame from the training set of frames;

initiate the object tracker to process the corresponding target frame to generate at least one tracker identifier for the corresponding target frame; and

initiate the group classifier to process the corresponding target frame to generate at least one classifier identifier for the corresponding target frame; and

determine at least one correlation between the at least one tracker identifier and the at least one classifier identifier, wherein generating the bonding key is based on the at least one correlation.

8. The end user device of claim 7 , wherein:

the object tracker is further configured to determine at least one tracker confidence parameter for the at least one tracker identifier corresponding to the corresponding target frame;

the group classifier is further configured to determine at least one classifier confidence parameter for the first classifier identifier corresponding to the corresponding target frame; and

the video calibration engine is further configured to:

determine at least one tracker threshold;

determine at least one classifier threshold;

map, for each corresponding target frame, the at least one tracker identifier to the at least one classifier identifier; and

execute a voting function to determine a set of valid correlations between the at least one tracker identifier and the at least one classifier identifier to determine the at least one correlation, wherein the voting function is based on:

evaluating the at least one tracker confidence parameter based on the at least one tracker threshold; and

evaluating the at least one classifier confidence parameter based on the at least one classifier threshold.

9. The end user device of claim 7 , wherein the video calibration engine is further configured to determine the calibration set of frames from the video data stream by:

determining a calibration threshold of a number of frames with correlations between the at least one tracker identifier and the at least one classifier identifier;

determining, using the object tracker, at least one unique tracker identifier in the video data stream; and

sequentially selecting, until the calibration threshold is reached for each unique tracker identifier of the at least one unique tracker identifier, frames from the video data stream to include in the calibration set of frames.

10. The end user device of claim 7 , wherein the video calibration engine is further configured to:

determine, for each frame in the calibration set of frames, at least one initial tracker identifier;

determine, for each frame in the calibration set of frames, at least one initial classifier identifier;

modify, for selected frames and to determine the at least one correlation, at least one of:

the at least one initial tracker identifier responsive to tracker correction logic; and

the at least one initial classifier identifier responsive to classifier correction logic; and

generate, based on the at least one correlation, the bonding key comprising a plurality of different tracker identifiers from the calibration set of frames and corresponding group identifiers.

11. A computer-implemented method, comprising:

determining, by an end user device using an object tracker, a first image object in a first frame of a video data stream from user video data;

assigning, by the object tracker and to the first image object in the first frame, a first tracker identifier;

processing, by the object tracker, subsequent frames of the video data stream to track movement of the first image object;

associating, by the object tracker, each frame of the video data stream corresponding to the movement of the first image object with the first tracker identifier;

determining, using a group classifier and a target frame assigned the first tracking identifier, a first group identifier associated with the first image object to generate a bonding key configured to map the first tracker identifier to the first group identifier;

assigning, by the end user device and based on the first tracker identifier and the bonding key, the first group identifier to a first plurality of frames of the video data stream, wherein:

the first plurality of frames of the video data stream are not processed by the group classifier;

the first group identifier corresponds to a first classifier identifier from the group classifier; and

the group classifier is trained using machine learning to determine at least one classifier identifier, including the first classifier identifier.

12. The computer-implemented method of claim 11 , further comprising:

processing each frame of the video data stream to detect at least one human face; and

calculating, for the at least one human face, embeddings quantifying a plurality of features of the at least one human face, wherein detection of the at least one human face initiates the object tracker.

13. The computer-implemented method of claim 11 , further comprising:

determining a plurality of frame identifiers for a plurality of frames in the video data stream;

processing the plurality of frames using the object tracker to determine a plurality of tracker identifiers; and

for each frame identifier and corresponding tracker identifier:

storing, in a bonding map, that frame identifier;

determining, from a plurality of group identifiers, the group identifier that corresponds to that corresponding tracker identifier; and

storing, in the bonding map, the determined group identifier associated with that frame identifier.

14. The computer-implemented method of claim 13 , wherein:

the bonding key is further configured to map the plurality of tracker identifiers to the plurality of group identifiers; and

determining the group identifier for each corresponding tracker identifier includes using that corresponding tracker identifier to index the corresponding group identifier in the bonding key.

15. The computer-implemented method of claim 13 , further comprising:

determining tracker conditions for initiating the group classifier for a target frame of the plurality of frames;

determine, from the group classifier, a group null value corresponding to a detected image object not meeting a classification threshold for groups in the group classifier;

evaluating, in the bonding map and following processing the plurality of frames to generate the bonding map, tracker identifiers for frame identifiers of adjacent frames to determine a continuous set of a same tracker identifier value; and

assigning, to the frame identifier for the target frame, a corresponding group classifier for the same tracker identifier value.

16. The computer-implemented method of claim 11 , further comprising, during a calibration process:

determining a calibration set of frames from the video data stream, wherein the calibration set of frames includes fewer frames than all frames in the video data stream;

for each frame in the calibration set of frames:

selecting a corresponding target frame from the calibration set of frames;

determining embeddings for at least one image object in the corresponding target frame;

initiating the object tracker to process the corresponding target frame to generate at least one tracker identifier for the corresponding target frame; and

initiating the group classifier to process the corresponding target frame to generate at least one classifier identifier for the corresponding target frame, wherein the group classifier is configured to:

process the embeddings for the at least one image object through at least one set of machine learning parameters; and

determine, based on processing the embeddings for the at least one image object, at least one category identifier for the at least one image object; and

determining at least one correlation between the at least one tracker identifier and the at least one classifier identifier, wherein generating the bonding key is based on the at least one correlation.

17. The computer-implemented method of claim 16 , further comprising:

determining at least one tracker threshold;

determining at least one classifier threshold;

determining, by the object tracker, at least one tracker confidence parameter for the at least one tracker identifier corresponding to the corresponding target frame;

determining, by the group classifier, at least one classifier confidence parameter for the first classifier identifier corresponding to the corresponding target frame;

mapping, for each corresponding target frame, the at least one tracker identifier to the at least one classifier identifier; and

executing a voting function to determine a set of valid correlations between the at least one tracker identifier and the at least one classifier identifier to determine the at least one correlation, wherein the voting function is based on:

evaluating the at least one tracker confidence parameter based on the at least one tracker threshold; and

evaluating the at least one classifier confidence parameter based on the at least one classifier threshold.

18. The computer-implemented method of claim 16 , wherein determining the calibration set of frames from the video data stream comprises:

determining a calibration threshold of a number of frames with correlations between the at least one tracker identifier and the at least one classifier identifier;

determining, using the object tracker, at least one unique tracker identifier in the video data stream; and

sequentially selecting, until the calibration threshold is reached for each unique tracker identifier of the at least one unique tracker identifier, frames from the video data stream to include in the calibration set of frames.

19. The computer-implemented method of claim 18 , further comprising:

determining, for each frame in the calibration set of frames, at least one initial tracker identifier;

determining, for each frame in the calibration set of frames, at least one initial classifier identifier;

modifying, for selected frames and to determine the at least one correlation, at least one of:

the at least one initial tracker identifier responsive to tracker correction logic; and

the at least one initial classifier identifier responsive to classifier correction logic; and

generating, based on the at least one correlation, the bonding key comprising a plurality of different tracker identifiers from the training set of frames and corresponding group identifiers.

20. An end user computing device, comprising:

at least one processor;

at least one memory;

a storage medium configured to store user video data;

means for determining, using an object tracker, a first image object in a first frame of a video data stream from the user video data;

means for assigning, by the object tracker and to the first image object in the first frame, a first tracker identifier;

means for processing, by the object tracker, subsequent frames of the video data stream to track movement of the first image object;

means for associating, by the object tracker, each frame of the video data stream corresponding to the movement of the first image object with the first tracker identifier;

means for determining, using a group classifier and a target frame assigned the first tracking identifier, a first group identifier associated with the first image object to generate a bonding key configured to map the first tracker identifier to the first group identifier;

means for assigning, based on the first tracker identifier and the bonding key, the first group identifier to a first plurality of frames of the video data stream, wherein:

the first plurality of frames of the video data stream are not processed by the group classifier;

the first group identifier corresponds to a first classifier identifier from the group classifier; and

the group classifier is trained using machine learning to determine at least one classifier identifier, including the first classifier identifier.

Assignments (8)

Continuity (1)