SYSTEMS, METHODS, AND COMPUTER PROGRAM PRODUCTS FOR IMPLEMENTING OBJECT PERMANENCE IN A SIMULATED ENVIRONMENT
Systems, methods, and computer program products for managing simulated environments are described. A simulated environment is accessed which represents a physical environment, and representations of objects are included, maintained, or removed in the simulated environment based on whether objects are represented in image data of the physical environment, and based on whether objects are occluded by other objects in the image data of the physical environment. Future occlusion can also be predicted based on motion paths of objects.
1 . A system comprising:
at least one image sensor;
at least one processor; and
at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing processor executable instructions and/or data that, when executed by the at least one processor, cause the system to:
access, by the at least one processor, a simulated environment, wherein the simulated environment comprises an environment model representing a physical environment;
capture, by the at least one image sensor, first image data representing the physical environment at a first time, the first image data further representing a first object at a first position;
include, in the environment model, a first representation of the first object based on the first image data;
capture, by the at least one image sensor, second image data representing the physical environment at a second time after the first time;
determine, by the at least one processor, whether the first object is represented in the second image data;
if the first object is represented in the second image data, maintain representation of the first object in the environment model;
if the first object is not represented in the second image data:
determine, by the at least one processor, whether the first object is occluded from the second image data by at least one second object at the second time;
if the first object is occluded from the second image data by the at least one second object at the second time: maintain representation of the first object in the environment model;
if the first object is not occluded from the second image data by the at least one second object at the second time: remove representation of the first object from the environment model.
2 . The system of claim 1 wherein the processor-executable instructions and/or data which cause the system to include, in the environment model, a first representation of the first object based on the first image data cause the system to:
populate, by the at least one processor, the environment model with the first representation of the first object based on the first image data.
3 . The system of claim 1 wherein the processor-executable instructions and/or data which cause the system to include, in the environment model, a first representation of the first object based on the first image data cause the system to:
determine, by the at least one processor, whether the environment model includes the first representation of the first object;
if the environment model includes the first representation of the first object, preserve the first representation of the first object as included in the environment model; and
if the environment model does not include the first representation of the first object, populate the environment model with the first representation of the first object based on the first image data.
4 . The system of claim 1 wherein the processor-executable instructions and/or data which cause the system to determine whether the first object is occluded from the second image data by the at least one second object at the second time cause the system to:
identify a second position of the at least one second object at the second time; and
determine, by the at least one processor, whether the second position of the at least one second object is between the first position of the first object at the first time and a position of the image sensor at the second time.
5 . The system of claim 1 , further comprising a robot body, wherein the at least one image sensor is carried by the robot body.
6 . The system of claim 5 wherein the at least one second object includes at least one member of the robot body.
7 . The system of claim 6 wherein the at least one member of the robot body includes at least one end-effector of the robot body.
8 . The system of claim 7 wherein the at least one end-effector of the robot body includes at least one hand member of the robot body.
9 . The system of claim 6 , wherein the processor-executable instructions and/or data which cause the system to determine whether the first object is occluded from the second image data by the at least one second object at the second time cause the system to:
identify, at a third time before the second time, a motion path of the at least one member of the robot body;
determine, by the at least one processor before the second time, a predicted position of the at least one member of the robot body at the second time based on the identified motion path; and
determine whether the predicted position of the at least one member of the robot body is between the first position of the first object at the first time and a position of the image sensor.
10 . The system of claim 9 wherein the processor-executable instructions and/or data which cause the system to identify the motion path of the at least one member of the robot body cause the system to: identify the motion path of the at least one member of the robot body by simulating, by the at least one processor, a trajectory of the at least one member of the robot body based on speed and direction of travel of the at least one member of the robot body.
11 . The system of claim 10 , further comprising at least one haptic sensor carried by the robot body, wherein the speed and direction of travel of the at least one member of the robot body are identified based on haptic data captured by the at least one haptic sensor.
12 . The system of claim 10 wherein the speed and direction of travel of the at least one member of the robot body are identified based on image data from the at least one image sensor.
13 . The system of claim 9 wherein the processor-executable instructions and/or data which cause the system to identify the motion path of the at least one member of the robot body cause the system to: identify the motion path of the at least one member of the robot body based on at least one motion instruction upon which movement of the at least one member of the robot body is based.
14 . The system of claim 5 , wherein the at least one processor is carried by the robot body.
15 . The system of claim 5 , wherein the at least one non-transitory processor-readable storage medium is carried by the robot body, and the simulated environment including the environment model is stored on the at least one non-transitory processor-readable storage medium carried by the robot body.
16 . The system of claim 5 , further comprising at least one device remote from the robot body, wherein:
the at least one non-transitory processor-readable storage medium includes at least one first non-transitory processor-readable storage medium at the remote device; and
the simulated environment including the environment model is stored on the at least one first non-transitory processor-readable storage medium at the remote device.
17 . The system of claim 1 , wherein the processor-executable instructions and/or data which cause the system to determine, by the at least one processor, whether the first object is represented in the second image data cause the system to: determine that the first object is not represented in the second image data.