Granted Patent B2

US 12,452,490 · App. 17/968,684 · Granted Oct 21, 2025

Inventor: Frank Maggiore (Jersey City, NJ)

Assignee: Apprentice FS, Inc.

H04N21/4542G06T7/20G06V10/225G06V10/25G06V10/757G06V10/761G06V20/41G06V20/46G06V20/49G06V20/52H04N21/23418H04N21/4396H04N21/84G06V2201/07

Abstract

Claims (142)

1. A method for distributing censored videos of manufacturing procedures performed within a facility comprising:

at a computing device associated with an operator, accessing a procedure containing:

a first instructional block comprising a first instruction; and

a second instructional block comprising a second instruction;

in response to initiation of the first instructional block at a first time, accessing a first video feed captured by the computing device interfacing with the operator during performance of a first instance of the first instruction at a make line within the facility;

generating a first instruction profile representing performance of the first instruction based on a first set of instruction features extracted from the first video feed;

detecting a first deviation exceeding a first threshold deviation between the first instruction profile of the first instruction and a target instruction profile of the first instruction in the first instructional block;

in response to detecting the first deviation exceeding the first threshold deviation:

extracting a first set of objects depicted in the first video feed from the first instruction profile;

accessing a manifest of objects specified in the first instructional block;

identifying a first subset of objects, in the first set of objects, unrelated to the first instance of the first instruction based on the manifest of objects;

censoring the first subset of objects in the first video feed to generate a censored video feed; and

serving the censored video feed to a remote viewer portal accessed by a remote viewer;

in response to initiation of the second instructional block at a second time, accessing a second video feed captured by the computing device interfacing with the operator during performance of a second instance of the second instruction at the make line within the facility;

generating a second instruction profile representing performance of the second instruction based on a second set of instruction features extracted from the second video feed;

detecting a second deviation falling below a second threshold deviation between the second instruction profile of the second instruction and a second target instruction profile of the second instruction in the second instructional block;

in response to detecting the second deviation falling below the second threshold deviation, flagging the second video feed as an uncensored video feed; and

serving the uncensored video feed to the remote viewer portal accessed by the remote viewer.

2. The method of claim 1 , wherein generating the first instruction profile representing performance of the first instruction based on the first set of instruction features extracted from the first video feed comprises:

initializing a template instruction profile;

identifying the first set of objects depicted in the first video feed based on the first set of instruction features;

identifying a first set of object paths for the first set of objects in the first video feed based on the first set of instruction features;

identifying a first set of object types for the first set of objects; and

storing the first set of objects, the first set of object paths, and the first set of object types in the template instruction profile to define the first instruction profile.

3. The method of claim 1 :

wherein generating the first instruction profile representing performance of the first instruction based on the first set of instruction features extracted from the first video feed comprises:

detecting a first constellation of objects in the first video feed based on the first set of instruction features; and

for each object in the constellation of objects:

identifying an object type in response to a constellation of the object matching a target object constellation of a particular object type within a threshold degree of confidence; and

storing the object of the particular object type in the first instruction profile to define the first set of objects depicted in the first video feed; and

wherein detecting the first deviation exceeding the first threshold deviation between the first instruction profile of the first instruction and the target instruction profile of the first instruction in the first instructional block comprises:

extracting a particular object from the target instruction profile related to performance of the first instruction;

querying the first set of objects from the first instruction profile for the particular object; and

in response to detecting absence of the particular object in the first set of objects, detecting the first deviation between the first instruction profile and the target instruction profile.

4. The method of claim 1 :

wherein accessing the first video feed captured by the computing device interfacing with the operator during performance of the first instance of the first instruction at the make line within the facility comprises:

extracting a first frame from the first video feed; and

extracting a second frame, following the first frame, from the first video feed;

wherein generating the first instruction profile representing performance of the first instruction based on the first set of instruction features extracted from the first video feed comprises:

extracting the first set of features from the first frame of the first video feed;

interpreting the first set of objects depicted in the first frame of the first video feed based on the first set of features;

extracting a second set of features from the second frame of the first video feed;

deriving a first set of object paths for the first set of objects from the first frame to the second frame based on differences between the second set of features extracted from the second frame and the first set of features extracted from the first frame; and

storing the first set of object paths in the first instruction profile; and

wherein detecting the first deviation exceeding the first threshold deviation between the first instruction profile of the first instruction and the target instruction profile of the first instruction in the first instructional block comprises:

extracting a target set of object paths from the target instruction profile related to performance of the first instruction; and

in response to the first set of object paths deviating from the target set of object paths, detecting the first deviation between the first instruction profile and the target instruction profile.

5. The method of claim 1 :

wherein generating the first instruction profile representing performance of the first instruction based on the first set of instruction features extracted from the first video feed comprises:

detecting a first equipment unit in the first video feed based on the first set of instruction features;

generating a bounding box in the first video feed enclosing the first equipment unit;

interpreting a first value output from a display on the equipment unit based on a first subset of instruction features, in the first set of instruction features, extracted from an area within the bounding box; and

storing the first value in the first instruction profile; and

wherein detecting the first deviation exceeding the first threshold deviation between the first instruction profile of the first instruction and the target instruction profile of the first instruction in the first instructional block comprises:

extracting a target value for the first equipment unit from the target instruction profile; and

in response to the first value deviating from the target value, detecting the first deviation between the first instruction profile and the target instruction profile.

6. The method of claim 1 :

wherein generating the first instruction profile representing performance of the first instruction based on the first set of instruction features extracted from the first video feed comprises:

detecting a first object in the first video feed corresponding to a hand of the operator based on the first set of instruction features;

detecting a second object in the first video feed corresponding to an instrument handled by the operator;

during a first time period of the first video feed, tracking a set of positions of the first object and the second object depicted in the first video feed;

interpreting a first action performed by the operator based on the set of positions of the first object and the second object; and

storing the first action in the first instruction profile; and

wherein detecting the first deviation exceeding the first threshold deviation between the first instruction profile of the first instruction and the target instruction profile of the first instruction in the first instructional block comprises:

extracting a target action for the operator from the target instruction profile; and

in response to the first action deviating from the target action, detecting the first deviation between the first instruction profile and the target instruction profile.

7. The method of claim 1 :

wherein accessing the first video feed captured by the computing device interfacing with the operator during performance of the instance of the first instruction at the make line within the facility comprises:

extracting a first video segment from the first video feed in response to initializing the first instructional block; and

extracting a second video segment, following the first video segment, from the first video feed following completion of the first instructional block;

wherein generating the first instruction profile representing performance of the first instruction comprises generating the first instruction profile based on the first set of instruction features extracted from the first video segment in the first video feed; and

wherein censoring the first subset of objects in the first video feed comprises censoring the first subset of objects in the first video segment of the first video feed to generate a first censored video segment.

8. The method of claim 1 , further comprising:

at a second time preceding the first time, accessing a third video feed captured by the computing device interfacing with the operator during performance of a prior instance of the procedure at the make line within the facility;

extracting a third set of instruction features from the third video feed;

initializing a template instruction profile;

identifying a second set of objects depicted in the third video feed based on the third set of instruction features;

identifying a second set of object paths for the second set of objects depicted in the third video feed based on the third set of instruction features;

identifying a second set of object types for the second set of objects;

storing the second set of objects, the second set of object paths, and the second set of object types in the template instruction profile to define the target instruction profile; and

storing the target instruction profile in the first instructional block.

9. The method of claim 1 :

further comprising:

accessing a minimum censorship specification associated with performance of the first instance of the procedure, the minimum censorship specification defining object types related to the procedure and corresponding to a first degree of censorship; and

identifying a second subset of objects, in the first set of objects, depicted in the first video feed related to the procedure based on the object types defined in the minimum censorship specification; and

wherein censoring the first subset of objects in the first video feed to generate the censored video feed comprises censoring the first subset of objects and the second subset of objects in the first video feed to generate the censored video feed.

10. The method of claim 9 , wherein censoring the first subset of objects in the first video feed to generate the censored video feed comprises:

censoring the first subset of objects in the first video feed at a first opacity level defined in the minimum censorship specification; and

censoring the second subset of objects in the first video feed at a second opacity level, greater than the first opacity level.

11. The method of claim 1 :

wherein identifying the first subset of objects, in the first set of objects, depicted in the first video feed comprises:

identifying the first subset of objects occupying a first region within the first video feed; and

generating a first bounding box about the first region in the first video feed containing the first subset of objects; and

wherein censoring the first subset of objects in the first video feed to generate the censored video feed comprises:

censoring a first area in the first video feed enclosed by the first bounding box and containing the first subset of objects; and

generating the censored video feed depicting the fogged first area.

12. The method of claim 1 , wherein serving the censored video feed to the remote viewer portal accessed by the remote viewer comprises:

at a second time following the first time, receiving selection to view the censored video feed at the remote viewer portal accessed by the remote viewer;

accessing a remote viewer profile associated with the remote viewer and specifying a permission to access an uncensored video feed; and

in response to the remote viewer profile specifying the permission to access the uncensored video feed:

serving the censored video feed censoring the first subset of objects to the remote viewer portal; and

serving the first video feed depicting the first subset of objects to the remote viewer portal.

13. The method of claim 1 , further comprising, in response to detecting the first deviation:

extracting a first audio feed from the first video feed captured by the computing device interfacing with the operator during performance the first instruction;

detecting a first set of audio phrases in the first audio feed based on a first set of audio features extracted from the first audio feed;

accessing a manifest of audio phrases from the first instructional block;

identifying a first subset of audio phrases, in the first set of audio phrases, unrelated to the procedure based on the manifest of audio phrases for the procedure;

muting the first subset of audio phrases in the first audio feed to generate a muted audio feed; and

appending the muted audio feed to the censored video feed.

14. A method comprising:

at a computing device associated with an operator, accessing a procedure containing a first instructional block comprising a first instruction;

in response to initiation of the first instructional block, accessing a first video feed captured by the computing device interfacing with the operator during performance of an instance of the first instruction at a make line within the facility;

extracting a first set of features from the first video feed;

based on the first set of features:

identifying a first set of objects depicted in the first video feed; and

tracking a first set of object paths for the first set of objects;

detecting a deviation exceeding a threshold deviation between the first set of object paths for the first set of objects and a target set of object paths related to performance of the first instruction; and

in response to detecting the deviation:

accessing a minimum censorship specification associated with performance of the instance of the procedure, the minimum censorship specification defining object types related to the procedure and corresponding to a first degree of censorship;

identifying a first subset of objects, in the first set of objects, depicted in the first video feed related to the procedure based on the object types defined in the minimum censorship specification;

censoring the first subset of objects in the first video feed to generate a censored video feed; and

serving the censored video feed to a remote viewer portal accessed by a remote viewer.

15. The method of claim 14 , further comprising generating a first instruction profile representing performance of the first instruction based on the first set of features extracted from the first video feed by:

initializing a template instruction profile;

identifying the first set of objects depicted in the first video feed based on the first set of features;

tracking the first set of object paths for the first set of objects in the first video feed based on the first set of features;

identifying a first set of object types for the first set of objects; and

aggregating the first set of objects, the first set of object paths, and the first set of object types in the template instruction profile to define the first instruction profile.

16. The method of claim 15 :

wherein identifying the first set of object types comprises:

detecting a first constellation of objects in the first video feed based on the first set of features; and

for each object in the first constellation of objects:

identifying an object type in response to a constellation of the object matching a target object constellation of a particular object type within a threshold degree of confidence; and

defining the object as the particular object type in the first instruction profile to define the first set of objects depicted in the first video feed.

17. The method of claim 14 :

further comprising:

accessing a manifest of objects specified in the first instructional block; and

identifying a second subset of objects, in the first set of objects, unrelated to the instance of the first instruction based on the manifest of objects; and

wherein censoring the first subset of objects in the first video feed to generate the censored video feed comprises, censoring the first subset of objects and the second subset of objects in the first video feed to generate the censored video feed.

18. The method of claim 17 , wherein censoring the first subset of objects in the first video feed to generate the censored video feed comprises:

censoring the first subset of objects in the first video feed at a first opacity level defined in the minimum censorship specification; and

censoring the second subset of objects in the first video feed at a second opacity level, greater than the first opacity level.

Assignments (1)

Continuity (3)

References Cited (30)