IP Library › Granted Patent US 11,516,438
Granted Patent B2
US 11,516,438 · App. 17/378,296 · Granted Nov 29, 2022

System and method for remote observation in a non-networked production facility

Inventors: Frank Maggiore (Port Jefferson Station, NY); Angelo Stracquatanio (Warren, NJ); Ali Jaafar (Jersey City, NJ); John Zimmerman (Jersey City, NJ)
Assignee: Apprentice FS, Inc.
H04N7/181G06F21/31G06F21/84G06T7/73G06V20/40G08B5/22H04N5/04H04N5/2624H04W76/30G06T2207/10016G06T2207/10028G06T2207/20132
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,516,438
App. No.
17/378,296
Granted
Nov 29, 2022
Kind
B2
Abstract

A method includes: via a user interface of a hub device, receiving a first set of user credentials associated with a first operator, the hub device connected to a set of mobile devices; accessing a first operator schedule defining a first scheduled manufacturing operation and a first observer of the first scheduled manufacturing operation, the first observer characterized by a first set of observer credentials; detecting disconnection of a first mobile device in the set of mobile devices from the hub device, the first mobile device associated with a first device ID; in response to detecting disconnection of the first mobile device from the hub device, associating the first device ID with the first operator; and routing a first video feed from the first mobile device to the first observer based on the first device ID and the first set of observer credentials.

Claims (111)

1. A method for remotely observing a manufacturing operation in a production facility comprising:

via a user interface of a hub device, receiving a first set of user credentials associated with a first operator, the hub device connected to a set of mobile devices;

accessing a first operator schedule defining:

a first scheduled manufacturing operation; and

a first observer of the first scheduled manufacturing operation, the first observer characterized by a first set of observer credentials;

detecting disconnection of a first mobile device in the set of mobile devices from the hub device, the first mobile device associated with a first device ID;

in response to detecting disconnection of the first mobile device from the hub device, associating the first device ID with the first operator; and

routing a first video feed from the first mobile device to the first observer based on the first device ID and the first set of observer credentials.

2. The method of claim 1 , wherein routing the first video feed from the first mobile device to the first observer based on the first device ID and the first set of observer credentials comprises, in response to detecting a direct connection of the first mobile device to the hub device based on the first device ID:

uploading the first video feed from the first mobile device to the hub device; and

transmitting the first video feed from the hub device to the first observer based on the first set of observer credentials.

3. The method of claim 1 :

wherein accessing the first operator schedule comprises accessing the first operator schedule defining:

the first scheduled manufacturing operation; and

the first observer of the first scheduled manufacturing operation, the first observer characterized by the first set of observer credentials comprising a first set of restricted regions of the production facility; and

wherein routing the first video feed from the first mobile device to the first observer comprises:

receiving the first video feed from the first mobile device;

identifying a first set of restricted regions depicted in the first video feed;

obscuring the first set of restricted regions in the first video feed to generate a first modified video feed; and

transmitting the first video feed to the first observer.

4. The method of claim 3 , wherein receiving the first video feed from the first mobile device comprises, in response to detecting a direct connection of the first mobile device to the hub device based on the first device ID, uploading the first video feed from the first mobile device to the hub device.

5. The method of claim 3 , wherein receiving the first video feed from the first mobile device comprises receiving the first video feed from the first mobile device via a wireless connection between the first mobile device and the hub device.

6. The method of claim 3 , wherein identifying the first set of restricted regions depicted in the first video feed comprises:

generating a first 3D point cloud representation of the production facility based on the first video feed;

localizing the first mobile device in the production facility based on the first 3D point cloud to calculate a first series of locations of the first mobile device in the production facility during the first video feed; and

identifying the first set of restricted regions depicted in the first video feed based on the first series of locations of the first mobile device and a field of view of the first mobile device.

7. The method of claim 3 :

further comprising receiving a secondary video feed from a second device, the secondary video feed concurrent with the first video feed and depicting the first operator and the first mobile device; and

wherein identifying the first set of restricted regions depicted in the first video feed comprises:

localizing the first mobile device in the production facility based on the first video feed and the secondary video feed to calculate a first series of locations of the first mobile device in the production facility during the first video feed; and

identifying the first set of restricted regions depicted in the first video feed based on the first series of locations of the first mobile device and a field of view of the first mobile device.

8. The method of claim 7 , further comprising:

identifying the first set of restricted regions depicted in the secondary video feed;

obscuring the first set of restricted regions in the secondary video feed to generate a modified secondary video feed; and

transmitting the secondary video feed to the first observer.

9. The method of claim 1 :

wherein accessing the first operator schedule comprises accessing the first operator schedule defining:

the first scheduled manufacturing operation; and

the first observer of the first scheduled manufacturing operation, the first observer characterized by the first set of observer credentials comprising a first set of restricted regions of the production facility; and

wherein routing the first video feed from the first mobile device to the first observer comprises:

receiving the first video feed from the first mobile device;

identifying a first set of restricted regions depicted in the first video feed;

cropping the first video feed to remove the first set of restricted regions depicted in the video feed; and

transmitting the first video feed to the first observer.

10. The method of claim 1 :

wherein accessing the first operator schedule comprises accessing the first operator schedule defining:

the first scheduled manufacturing operation; and

the first observer of the first scheduled manufacturing operation, the first observer characterized by the first set of observer credentials comprising a first set of viewable regions of the production facility; and

wherein routing the first video feed from the first mobile device to the first observer comprises:

receiving the first video feed from the first mobile device;

identifying a first set of restricted regions of the production facility outside of the first set of viewable regions depicted in the first video feed;

obscuring the first set of restricted regions to generate a first modified video feed; and

transmitting the first modified video feed to the first observer.

11. The method of claim 1 :

further comprising:

via the user interface of the hub device, receiving a second set of user credentials associated with a second operator;

accessing a second operator schedule defining:

the first scheduled manufacturing operation; and

the first observer of the first scheduled manufacturing operation;

detecting disconnection of a second mobile device in the set of mobile devices from the hub device, the second mobile device associated with a second device ID; and

in response to detecting disconnection of the second mobile device from the hub device, associating the second device ID with the second operator; and

wherein routing the first video feed from the first mobile device to the first observer comprises:

receiving the first video feed depicting the first scheduled manufacturing operation from the first mobile device;

receiving a second video feed depicting the first scheduled manufacturing operation from the second mobile device;

synchronizing the first video feed and the second video feed; and

transmitting a split-screen video feed to the first observer based on the first device ID, the second device ID, and the first set of observer credentials.

12. The method of claim 1 :

wherein accessing the first operator schedule comprises accessing the first operator schedule defining:

the first scheduled manufacturing operation;

a second scheduled manufacturing operation;

the first observer of the first scheduled manufacturing operation; and

a second observer of the second schedule manufacturing operation, the second observer characterized by a second set of observer credentials;

wherein routing a first video feed from the first mobile device to the first observer comprises:

receiving a video feed from the first mobile device based on the first device ID;

identifying a first subsegment of the video feed depicting the first scheduled manufacturing operation; and

transmitting the first subsegment of the video feed to the first observer based on the first set of observer credentials; and

further comprising:

identifying a second subsegment of the video feed depicting the second scheduled manufacturing operation; and

transmitting the second subsegment of the video feed to the second observer based on the second set of observer credentials.

13. The method of claim 1 :

further comprising receiving a second video feed from a second device, the second video feed concurrent with the first video feed and depicting the first operator and the first mobile device;

localizing the first mobile device in the production facility based on the first video feed and the second video feed to calculate a first series of locations of the first mobile device in the production facility during the first video feed; and

identifying the first set of restricted regions depicted in the first video feed based on the first series of locations of the first mobile device and a field of view of the first mobile device.

14. The method of claim 1 , further comprising:

in response to detecting disconnection of the first mobile device from the hub device, appending the first device ID to a list of in-use devices; and

in response to not detecting reconnection of the first mobile device with the hub device and in response to expiration of the first scheduled manufacturing operation, rendering a prompt to reconnect the first mobile device to the hub device at the user interface of the hub device.

15. The method of claim 1 , further comprising, in response to detecting a network security vulnerability:

halting the routing of the first video feed to the first observer; and

storing the first video feed in a local cache of the hub device.

16. The method of claim 1 , further comprising:

receiving a safety data feed from the production facility; and

in response to the safety data indicating a safety concern, rendering a safety warning at the user interface of hub device.

17. The method of claim 1 , wherein the first mobile device is air-gapped from the hub device during the first scheduled manufacturing operation.

18. A hub device for coordinating mobile devices within a production facility, comprising:

a user interface;

a set of charging locations configured to charge a set of mobile devices;

a chassis configured to mount to a wall within the production facility; and

an integrated computational device configured to:

via the user interface, receive a first set of user credentials associated with a first operator, the hub device charging the set of mobile devices;

access a first operator schedule defining:

a first scheduled manufacturing operation; and

a first observer of the first scheduled manufacturing operation, the first observer characterized by a first set of observer credentials;

detect disconnection of a first mobile device in the set of mobile devices from a first charging location in the set of charging locations, the first mobile device associated with a first device ID;

at a first time, in response to detecting disconnection of the first mobile device from the first charging location, associate the first device ID with the first operator; and

at a second time after the scheduled manufacturing operation, in response to detecting a direct connection of the first mobile device to the hub device based on the first device ID:

upload a first video feed from the first mobile device to the hub device; and

transmit the first video feed from the hub device to the first observer based on the first set of observer credentials.

19. The hub device of claim 18 , further comprising an emergency shut-off switch configured to disconnect the hub device from an external network.

20. The hub device of claim 18 , wherein the integrated computational device is further configured to, prior to transmitting the first video feed from the hub device to the first observer:

identify a first set of restricted regions depicted in the first video feed; and

obscure the first set of restricted regions in the first video feed to generate a first modified video feed.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 23, 2021
From: MAGGIORE, FRANK; STRACQUATANIO, ANGELO; JAAFAR, ALI; ZIMMERMAN, JOHN
To: APPRENTICE FS, INC.
Reel/Frame 057585/0235 →
Continuity (2)
Provisional Application 63052913 · Jul 16, 2020
Related Publication 20220021847A1 · Jan 20, 2022