Generating a model for an object encountered by a robot
Methods and apparatus related to generating a model for an object encountered by a robot in its environment, where the object is one that the robot is unable to recognize utilizing existing models associated with the robot. The model is generated based on vision sensor data that captures the object from multiple vantages and that is captured by a vision sensor associated with the robot, such as a vision sensor coupled to the robot. The model may be provided for use by the robot in detecting the object and/or for use in estimating the pose of the object.
1. A method implemented by one or more processors, the method comprising:
identifying a three-dimensional (3D) object model of an object;
generating a plurality of rendered images based on the 3D object model, wherein the rendered images capture the 3D object model at a plurality of different poses relative to viewpoints of the rendered images, and wherein generating the rendered images based on the object model comprises:
rendering a first image that renders the 3D object model at a first pose of the different poses and that includes first additional content; and
rendering a second image that renders the 3D object model at a second pose of the different poses and that includes second additional content that is distinct from the first additional content:
generating training examples that each include a corresponding one of the rendered images as training example input and that each include training example output that is based on a feature of the object in the corresponding one of the rendered images;
training a machine learning model based on the training examples; and
providing the machine learning model for use in control of a robot.
2. The method of claim 1 , wherein rendering the first image with the first additional content comprises rendering the 3D object model onto a first background, and wherein rendering the second image with second additional content comprises rendering the 3D object model onto a second background that is distinct from the first background.
3. The method of claim 2 , further comprising:
selecting the first background based on the environment of the robot.
4. The method of claim 3 , wherein selecting the first background based on the environment of the robot comprises:
selecting an additional image based on the additional image being captured in the environment of the robot; and
generating the first background based on the additional image.
5. The method of claim 4 , wherein the additional image is captured by a vision sensor of the robot.
6. The method of claim 4 , further comprising:
selecting the second background based on the environment of the robot.
7. The method of claim 1 , further comprising:
generating a first scene that includes the 3D object model and that includes a first additional 3D object model of a first additional object,
generating a second scene that includes the 3D object model, that includes a second additional 3D object model of a second additional object, and that excludes the first additional 3D object model;
wherein rendering the first image with the first additional content comprises rendering the first image using the first scene; and
wherein rendering the second image with the second additional content comprises rendering the second image using the second scene.
8. The method of claim 7 , further comprising selecting the first additional 3D object model based on an environment of the robot.
9. The method of claim 1 , wherein the training example output of each of the training examples includes a corresponding pose of the object in the corresponding one of the rendered images.
10. The method of claim 1 , wherein the rendered images each include a plurality of color channels and a depth channel.
11. A system comprising:
one or more processors executing instructions stored in memory to cause the one or more processors to:
identify a three-dimensional (3D) object model of an object;
generate a plurality of rendered images based on the 3D object model, wherein the rendered images capture the 3D object model at a plurality of different poses relative to viewpoints of the rendered images, and wherein in generating the rendered images based on the object model one or more of the processors are to:
render a first image that renders the 3D object model at a first pose of the different poses and that includes first additional content; and
render a second image that renders the 3D object model at a second pose of the different poses and that includes second additional content that is distinct from the first additional content:
generate training examples that each include a corresponding one of the rendered images as training example input and that each include training example output that is based on a feature of the object in the corresponding one of the rendered images;
train a machine learning model based on the training examples; and
provide the machine learning model for use in control of a robot.
12. The system of claim 11 , wherein in rendering the first image with the first additional content one or more of the processors are to render the 3D object model onto a first background, and wherein in rendering the second image with second additional content one or more of the processors are to render the 3D object model onto a second background that is distinct from the first background.
13. The system of claim 12 , wherein in executing the instructions one or more of the processors are further to:
select the first background based on the environment of the robot.
14. The system of claim 13 , wherein in selecting the first background based on the environment of the robot one or more of the processors are to:
select an additional image based on the additional image being captured in the environment of the robot; and
generate the first background based on the additional image.
15. The system of claim 14 , wherein the additional image is captured by a vision sensor of the robot.
16. The system of claim 14 , wherein in executing the instructions one or more of the processors are further to:
select the second background based on the environment of the robot.
17. The system of claim 11 , wherein in executing the instructions one or more of the processors are further to:
generate a first scene that includes the 3D object model and that includes a first additional 3D object model of a first additional object,
generate a second scene that includes the 3D object model, that includes a second additional 3D object model of a second additional object, and that excludes the first additional 3D object model;
wherein in rendering the first image with the first additional content one or more of the processors are to render the first image using the first scene; and
wherein in rendering the second image with the second additional content or more of the processors are to render the second image using the second scene.
18. The system of claim 17 , wherein in executing the instructions one or more of the processors are further to select the first additional 3D object model based on an environment of the robot.
19. The system of claim 11 , wherein the training example output of each of the training examples includes a corresponding pose of the object in the corresponding one of the rendered images.
20. The system of claim 11 , wherein the rendered images each include a plurality of color channels and a depth channel.