Configuring remote devices through volumetric video telepresence interfaces
A configuration system uses multiple depth cameras to create a volumetric capture space around an electronically controllable industrial machine or system, referred to as a target system. The output of the cameras is processed to create a live 3D model of everything within the space. A remote operator can then navigate within this 3D model, for example from a desktop application, in order to view the target system from various perspectives in a live 3D telepresence. In addition to the live 3D model, a configuration system generates a 3D user interface for programming and configuring machines or target systems within the space in a spatially coherent way. Local operators can interact with the target system using mobile phones which track the target system in augmented reality. Any number of local operators can interact with a remote operator to simultaneously program and configure the target system.
1. A method performed by one or more computers, each computer having at least one processor and a memory, the method comprising:
receiving a stream of live image data from each of a plurality of cameras capturing images of a portion of a physical environment containing an electronically configurable target system;
processing the streams of live image data to create a live 3D model of the environment;
providing a remote user interface through which a remote user can view a visual perspective of the live 3D model, wherein the remote user can freely navigate a six dimensional pose of the visual perspective about the live 3D model;
augmenting the remote user interface with a configuration user interface for configuring the electronically configurable target system, wherein the configuration user interface is visually depicted in a constant location relative to an aspect of the environment as the visual perspective of the remote user interface changes;
receiving user input to configure the electronically configurable target system through the configuration user interface by way of the remote user interface; and
in response to receiving the user input, providing visual feedback through the configuration user interface.
2. The method of claim 1 , wherein the aspect of the environment is a stationary coordinate system fixed relative to the environment.
3. The method of claim 1 , wherein the aspect of the environment is an object that is moveable about the environment.
4. The method of claim 3 , wherein the object is a portion of the configurable target system.
5. The method of claim 1 , wherein the user input to configure the electronically configurable target system comprises an instruction to modify an operational property of the configurable target system.
6. The method of claim 1 , further comprising receiving visual programming input through the remote user interface for visually programming the configurable target system, wherein the visual programming input comprises the user input to configure the electronically configurable target system.
7. The method of claim 6 , wherein the visually programming the configurable target system comprises configuring the configurable target system to display a display element in the configuration user interface, wherein the display element visualizes dynamically updated information of the configurable target system.
8. The method of claim 7 , wherein the display element is selected from the group consisting of: a graph, a numerical value display, an image, a downloadable file, and a document.
9. The method of claim 6 , wherein the visually programming the configurable target system comprises configuring the configurable target system to display a control element in the configuration user interface, wherein the control element operates to receive user input that affects control of the configurable target system.
10. The method of claim 9 , wherein the control element is selected from the group consisting of: a switch, a numerical input, a file upload input, a data upload input, and a text input.
11. The method of claim 1 , wherein the electronically configurable target system is an industrial machine.
12. The method of claim 1 , wherein the electronically configurable target system is an industrial system.
13. The method of claim 1 , wherein the electronically configurable target system is a residential home automation system.
14. The method of claim 1 , wherein the electronically configurable target system is a commercial building automation system.
15. The method of claim 1 , wherein each of the plurality of cameras is a depth camera configured to measure and record distances of physical surfaces from the depth camera for pixels representing physical surfaces in the images.
16. A system comprising the one or more computers of the method of claim 1 , wherein the one or more computers:
perform a first portion of the method, and
cause one or more other computers to perform a second portion of the method,
such that the method is performed jointly by the one or more computers and the one or more other computers.
17. One or more non-transitory computer-readable media encoded with computer code that is executed by the one or more computers of the method of claim 1 to cause the one or more computers to perform the method.