Systems and methods for detecting occluded objects
Systems and methods for detecting occluded objects are disclosed. In one embodiment, a method of determining a shape and pose of an object occluded by an occlusion object includes receiving, by a generative model, a latent vector, and iteratively performing an optimization routine until a loss is less than a loss threshold. The optimization routine includes generating, by the generative model, a predicted object having a shape and a pose from the latent vector, generating a predicted shadow cast by the predicted object, calculating the loss by comparing the predicted shadow with an observed shadow, and modifying the latent vector when the loss is greater than the loss threshold. The method further includes selecting the predicted object as the object when the loss is less than the loss threshold.
1 . A method of determining a shape and pose of an object occluded by an occlusion object, the method comprising:
receiving, by a generative model, a latent vector of an observed shadow of the object occluded by the occlusion object;
iteratively performing an optimization routine until a loss is less than a loss threshold, the optimization routine comprising:
generating, by the generative model from the latent vector of the observed shadow of the object occluded by the occlusion object, a predicted object having a shape and a pose;
generating a predicted shadow cast by the predicted object;
calculating the loss by comparing the predicted shadow with the observed shadow, wherein the loss corresponds to a generation accuracy of the predicted shadow and the observed shadow;
modifying the latent vector when the loss is greater than the loss threshold; and
selecting the predicted object as the object when the loss is less than the loss threshold.
2 . The method of claim 1 , further comprising receiving a known light location and a known pose of the object, wherein the known light location and the known pose of the object affect the predicted shadow.
3 . The method of claim 1 , further comprising receiving an initial light location and an initial pose of the object, wherein the optimization routine further comprises modifying the initial light location and the initial pose of the object when the loss is greater than the loss threshold.
4 . The method of claim 1 , wherein generating the predicted shadow cast by the object is performed by differential shadow rendering.
5 . The method of claim 1 , wherein the generative model is trained to generate one class of object.
6 . The method of claim 1 , wherein the loss is a binary cross-entropy loss.
7 . A system for determining a shape and pose of an object occluded by an occlusion object, the method comprising:
one or more processors; and
a non-transitory, computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a latent vector of an observed shadow of the object occluded by the occlusion object;
iteratively perform an optimization routine until a loss is less than a loss threshold, the optimization routine comprising:
generating, by a generative model from the latent vector of the observed shadow of the object occluded by the occlusion object, a predicted object having a shape and a pose;
generating a predicted shadow cast by the predicted object;
calculating the loss by comparing the predicted shadow with the observed shadow, wherein the loss corresponds to a generation accuracy of the predicted shadow and the observed shadow;
modifying the latent vector when the loss is greater than the loss threshold; and
selecting the predicted object as the object when the loss is less than the loss threshold.
8 . The system of claim 7 , wherein the instructions further cause the one or more processors to receive a known light location and a known pose of the object, wherein the known light location and the known pose of the object affect the predicted shadow.
9 . The system of claim 7 , wherein the instructions further cause the one or more processors to receive an initial light location and an initial pose of the object, wherein the optimization routine further comprises modifying the initial light location and the initial pose of the object when the loss is greater than the loss threshold.
10 . The system of claim 7 , wherein generating the predicted shadow cast by the object is performed by differential shadow rendering.
11 . The system of claim 7 , wherein the generative model is trained to generate one class of object.
12 . The system of claim 7 , wherein the loss is a binary cross-entropy loss.
13 . A robot operable to determine a shape and pose of an object occluded by an occlusion object, the robot comprising:
one or more processors; and
a non-transitory, computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a latent vector of an observed shadow of the object occluded by the occlusion object;
iteratively perform an optimization routine until a loss is less than a loss threshold, the optimization routine comprising:
generating, by a generative model from the latent vector of the observed shadow of the object occluded by the occlusion object, a predicted object having a shape and a pose;
generating a predicted shadow cast by the predicted object;
calculating the loss by comparing the predicted shadow with the observed shadow, wherein the loss corresponds to a generation accuracy of the predicted shadow and the observed shadow;
modifying the latent vector when the loss is greater than the loss threshold; and
select the predicted object as the object when the loss is less than the loss threshold; and
control a movement of the robot based on the predicted object.
14 . The robot of claim 13 , wherein the instructions further cause the one or more processors to receive a known light location and a known pose of the object, wherein the known light location and the known pose of the object affect the predicted shadow.
15 . The robot of claim 13 , wherein the instructions further cause the one or more processors to receive an initial light location and an initial pose of the object, wherein the optimization routine further comprises modifying the initial light location and the initial pose of the object when the loss is greater than the loss threshold.
16 . The robot of claim 13 , wherein generating the predicted shadow cast by the object is performed by differential shadow rendering.
17 . The robot of claim 13 , wherein the generative model is trained to generate one class of object.
18 . The robot of claim 13 , wherein the loss is a binary cross-entropy loss.
19 . The robot of claim 13 , further comprising an arm and end effector, wherein the movement of the robot comprises moving one or more of the arm and the end effector.
20 . The robot of claim 13 , further comprising an image sensor, wherein the observed shadow is generated by the image sensor.