Virtual presence for telerobotics in a dynamic scene
Described herein are methods and systems for providing virtual presence for telerobotics in a dynamic scene. A sensor captures frames of a scene comprising one or more objects. A computing device generates a set of feature points corresponding to objects in the scene and matches the set of feature points to 3D points in a map of the scene. The computing device generates a dense mesh of the scene and the objects using the matched feature points and transmits the dense mesh the frame to a remote viewing device. The remote viewing device generates a 3D representation of the scene and the objects for display to a user and receives commands from the user corresponding to interaction with the 3D representation of the scene. The remote viewing device transmits the commands to a robot device that executes the commands to perform operations on the objects in the scene.
1 . A system for providing virtual presence for telerobotics in a dynamic scene, the system comprising:
a remote viewing device and a remote controller coupled to the remote viewing device;
a sensor device that captures one or more frames of a scene comprising one or more objects, each frame comprising (i) one or more color images of the scene and the one or more objects and (ii) one or more depth maps of the scene and the one or more objects;
a computing device coupled to the sensor device, the computing device comprising a memory that stores computer-executable instructions and a processor that executes the instructions to:
receive a first set of frames from the sensor device;
for each frame in the first set of frames:
generate a set of feature points corresponding to the one or more objects in the scene,
match the set of feature points to one or more corresponding 3D points in a depth map from the frame, and
construct a dense mesh of the scene and the one or more objects using the matched feature points;
receive a second set of frames from the sensor device;
for each frame in the second set of frames:
a) calculate a geometric error between one or more segments of the dense mesh and corresponding 3D points in a depth map from the frame,
b) classify one or more segments of the dense mesh as one or more dynamic segments based on the geometric error calculated for the one or more segments,
c) non-rigidly deform only the one or more segments of the dense mesh that are classified as the one or more dynamic segments using data associated with the corresponding 3D points in the depth map from the frame,
d) calculate a second geometric error between the deformed one or more dynamic segments and corresponding 3D points in the depth map from the frame,
e) modify the deformed dense mesh using a non-linear optimization if the second geometric error does not converge,
f) repeat steps d) and e) until the second geometric error converges, and
g) update a texture of the deformed dense mesh by aligning a color image from the frame to the deformed dense mesh; and
determine net changes to the dense mesh resulting from the non-rigid deformation and transmit (i) the net changes to the dense mesh of the scene and the one or more objects and (ii) the frame to the remote viewing device,
wherein the remote viewing device configured to:
update an existing 3D representation of the scene and the one or more objects using the net changes to the dense mesh and the frame for display to a user;
receive one or more commands from the user via the remote controller, the one or more commands corresponding to interaction with the one or more objects in the updated 3D representation of the scene, and
transmit the one or more commands to a robot device, and
wherein the robot device is configured to:
execute the one or more commands received from the remote viewing device to perform one or more operations.
2 . The system of claim 1 , wherein generating the set of feature points corresponding to the one or more objects in the scene comprises detecting one or more feature points in the frame using a corner detection algorithm.
3 . The system of claim 2 , wherein matching the set of feature points to the one or more corresponding 3D points in the depth map from the frame comprises using a feature descriptor to match the set of feature points to the one or more corresponding 3D points.
4 . The system of claim 1 , wherein matching the set of feature points to the one or more corresponding 3D points in the depth map from the frame comprises minimizing a projection error between each feature point and one or more corresponding 3D points.
5 . The system of claim 4 , wherein the minimizing the projection error is performed using a nonlinear optimization algorithm.
6 . The system of claim 1 , wherein the updating the existing 3D representation of the scene and the one or more objects using the net changes to the dense mesh and the frame comprises:
detecting one or more keypoints of one or more objects in the scene using the received frame;
matching the detected one or more keypoints to one or more 3D points in a stored map to generate a point cloud;
updating the generated point cloud using the net changes to the dense mesh received from the computing device; and
mapping the frame onto a surface of the generated point cloud to update the 3D representation.
7 . The system of claim 6 , wherein updating the generated point cloud using the net changes to the dense mesh is performed using an Iterative Closest Point (ICP) algorithm.
8 . The system of claim 6 , wherein the 3D representation comprises a textured mesh of the scene and the one or more objects in the scene.
9 . The system of claim 1 , wherein the remote viewing device comprises an augmented reality (AR) viewing apparatus, a virtual reality (VR) viewing apparatus, or a mixed reality (MR) viewing apparatus.
10 . The system of claim 9 , wherein the remote viewing device is worn by the user.
11 . A computerized method for providing virtual presence for telerobotics in a dynamic scene, the method comprising:
capturing, by a sensor device, one or more frames of a scene comprising one or more objects, each frame comprising (i) one or more color images of the scene and the one or more objects and (ii) one or more depth maps of the scene and the one or more objects;
receiving, by a computing device coupled to the sensor device, a first set of frames from the sensor device;
for each frame in the first set of frames:
generating, by the computing device coupled to the sensor device, a set of feature points corresponding to the one or more objects in the scene,
matching, by the computing device, the set of feature points to one or more corresponding 3D points in a depth map from the frame, and
constructing, by the computing device, a dense mesh of the scene and the one or more objects using the matched feature points;
receiving, by the computing device, a second set of frames from the sensor device;
for each frame in the second set of frames:
a) calculating, by the computing device, a first geometric error between one or more segments of the dense mesh and corresponding 3D points in a depth map from the frame,
b) classifying, by the computing device, one or more segments of the dense mesh as one or more dynamic segments based on the first geometric error calculated for the one or more segments,
c) non-rigidly deforming, by the computing device, only the one or more segments of the dense mesh that are classified as the one or more dynamic segments using data associated with the corresponding 3D points in the depth map from the frame,
d) calculate a second geometric error between the deformed one or more dynamic segments and corresponding 3D points in the depth map from the frame,
e) modify the deformed dense mesh using a non-linear optimization if the second geometric error does not converge,
f) repeating, by the computing device, steps d) and e) until the second geometric error converges, and
g) updating, by the computing device, a texture of the deformed dense mesh by aligning a color image from the frame to the deformed dense mesh;
determining, by the computing device, net changes to the dense mesh resulting from the non-rigid deformation and transmitting (i) the net changes to the dense mesh of the scene and the one or more objects and (ii) the frame to a remote viewing device, the remote viewing device coupled to a remote controller;
updating, by the remote viewing device, an existing 3D representation of the scene and the one or more objects using the net changes to the dense mesh and the frame for display to a user;
receiving, by the remote viewing device, one or more commands from the user via the remote controller, the one or more commands corresponding to interaction with the one or more objects in the updated 3D representation of the scene;
transmitting, by the remote viewing device, the one or more commands to a robot device that interacts with the one or more objects in the scene; and
executing, by the robot device, the one or more commands received from the remote viewing device to perform one or more operations.
12 . The method of claim 11 , wherein the generating the set of feature points corresponding to the one or more objects in the scene comprises detecting one or more feature points in the frame using a corner detection algorithm.
13 . The method of claim 12 , wherein the matching the set of feature points to the one or more corresponding 3D points in the depth map from the frame comprises using a feature descriptor to match the set of feature points to the one or more corresponding 3D points.
14 . The method of claim 11 , wherein the matching the set of feature points to the one or more corresponding 3D points in the depth map from the frame comprises minimizing a projection error between each feature point and one or more corresponding 3D points.
15 . The method of claim 14 , wherein the minimizing the projection error is performed using a nonlinear optimization algorithm.
16 . The method of claim 11 , wherein the updating the existing 3D representation of the scene and the one or more objects using the net changes to the dense mesh and the frame comprises:
detecting one or more keypoints of one or more objects in the scene using the received frame;
matching the detected one or more keypoints to one or more 3D points in a stored map to generate a point cloud;
updating the generated point cloud using the net changes to the dense mesh received from the computing device; and
mapping the frame onto a surface of the generated point cloud to update the 3D representation.
17 . The method of claim 16 , wherein updating the generated point cloud using the net changes to the dense mesh is performed using an Iterative Closest Point (ICP) algorithm.
18 . The method of claim 16 , where in the 3D representation comprises a textured mesh of the scene and the one or more objects in the scene.
19 . The method of claim 11 , wherein the remote viewing device comprises an augmented reality (AR) viewing apparatus, a virtual reality (VR) viewing apparatus, or a mixed reality (MR) viewing apparatus.
20 . The method of claim 19 , wherein the remote viewing device is worn by the user.