Source-agnostic image processing

1. A method comprising:

receiving an input image that is captured by a sensor and that is from a real domain, the input image showing an object to be manipulated by a robot in a robotic process;

processing the input image to generate an intermediate representation of the input image, comprising:

generating (i) a gradient orientation representation that comprises, for each pixel in the input image, a first sub-representation that includes a gradient orientation of the pixel along a horizontal direction and a second sub-representation that includes a gradient orientation of the pixel along a vertical direction and (ii) a gradient magnitude representation of the input image that comprises, for each pixel in the input image, a gradient magnitude of the pixel; and

generating the intermediate representation of the input image based on, for each pixel in the input image, computing a first product between (i) the gradient orientation of the pixel included in the first sub-representation included in the gradient orientation representation and (ii) the gradient magnitude of the pixel included in the gradient magnitude representation and computing a second product between (i) the gradient orientation of the pixel included in the second sub-representation included in the gradient orientation representation and (ii) the gradient magnitude of the pixel included in the gradient magnitude representation;

processing the intermediate representation of the input image using a neural network that has been trained to make predictions about geometry-related characteristics of objects in training images from a synthetic domain to generate a network output that represents a prediction about geometry-related characteristics of the object in the input image that is from the real domain;

generating, based on the network output, a motion plan for the robot to manipulate the object in the robotic process; and

controlling the robot to execute the robotic process to manipulate the object in accordance with the motion plan.

2. The method of claim 1 , wherein the neural network is trained to learn the geometry of the object in the image.

3. The method of claim 2 , wherein the network does not allocate capacity to model changes in illumination, light color, specularities, or reflections.

4. The method of claim 1 , further comprising:

receiving a second input image from the real domain that is different from the synthetic domain, the second input image showing a second object to be manipulated by the robot in the robotic process;

processing the second input image to generate a second intermediate representation of the second input image; and

processing the second intermediate representation of the second input image using the neural network to generate a second network output.

5. The method of claim 1 , wherein the network output is an object detection or classification output.

6. The method of claim 1 , wherein the network output is a pose estimation output.

7. The method of claim 1 , wherein the network output is a semantic segmentation output.

8. The method of claim 1 , wherein neural network has been trained on labelled images from the synthetic domain.

9. The method of claim 1 , wherein generating the gradient orientation representation and the gradient magnitude representation of the input image comprises:

computing a respective derivative of each of some or all pixels of the input image; and

generating, from the respective derivatives, the gradient orientation representation and the gradient magnitude representation of the input image.

10. The method of claim 1 , wherein generating the intermediate representation of the input image from the gradient orientation representation and the gradient magnitude representation comprises:

generating a hue, saturation, value (HSV) color encoded representation of the input image from the gradient orientation representation and the gradient magnitude representation of the input image.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving an input image that is captured by a sensor and that is from a real domain, the input image showing an object to be manipulated by a robot in a robotic process;

processing the input image to generate an intermediate representation of the input image, comprising:

generating (i) a gradient orientation representation that comprises, for each pixel in the input image, a first sub-representation that includes a gradient orientation of the pixel along a horizontal direction and a second sub-representation that includes a gradient orientation of the pixel along a vertical direction and (ii) a gradient magnitude representation of the input image that comprises, for each pixel in the input image, a gradient magnitude of the pixel; and

generating the intermediate representation of the input image based on, for each pixel in the input image, computing a first product between (i) the gradient orientation of the pixel included in the first sub-representation included in the gradient orientation representation and (ii) the gradient magnitude of the pixel included in the gradient magnitude representation and computing a second product between (i) the gradient orientation of the pixel included in the second sub-representation included in the gradient orientation representation and (ii) the gradient magnitude of the pixel included in the gradient magnitude representation;

processing the intermediate representation of the input image using a neural network that has been trained to make predictions about geometry-related characteristics of objects in training images from a synthetic domain to generate a network output that represents a prediction about geometry-related characteristics of the object in the input image that is from the real domain;

generating, based on the network output, a motion plan for the robot to manipulate the object in the robotic process; and

controlling the robot to execute the robotic process to manipulate the object in accordance with the motion plan.

12. The system of claim 11 , wherein the neural network is trained to learn the geometry of the object in the image.

13. The system of claim 12 , wherein the network does not allocate capacity to model changes in illumination, light color, specularities, or reflections.

14. The system of claim 11 , wherein the operations further comprise:

receiving a second input image from the real domain that is different from the synthetic domain, the second input image showing a second object to be manipulated by the robot in the robotic process;

processing the second input image to generate a second intermediate representation of the second input image; and

processing the second intermediate representation of the second input image using the neural network to generate a second network output.

15. The system of claim 11 , wherein generating the gradient orientation representation and the gradient magnitude representation of the input image comprises:

computing a respective derivative of each of some or all pixels of the input image; and

generating, from the respective derivatives, the gradient orientation representation and the gradient magnitude representation of the input image.

16. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an input image that is captured by a sensor and that is from a real domain, the input image showing an object to be manipulated by a robot in a robotic process;

processing the input image to generate an intermediate representation of the input image, comprising:

generating (i) a gradient orientation representation that comprises, for each pixel in the input image, a first sub-representation that includes a gradient orientation of the pixel along a horizontal direction and a second sub-representation that includes a gradient orientation of the pixel along a vertical direction and (ii) a gradient magnitude representation of the input image that comprises, for each pixel in the input image, a gradient magnitude of the pixel; and

generating the intermediate representation of the input image based on, for each pixel in the input image, computing a first product between (i) the gradient orientation of the pixel included in the first sub-representation included in the gradient orientation representation and (ii) the gradient magnitude of the pixel included in the gradient magnitude representation and computing a second product between (i) the gradient orientation of the pixel included in the second sub-representation included in the gradient orientation representation and (ii) the gradient magnitude of the pixel included in the gradient magnitude representation;

processing the intermediate representation of the input image using a neural network that has been trained to make predictions about geometry-related characteristics of objects in training images from a synthetic domain to generate a network output that represents a prediction about geometry-related characteristics of the object in the input image that is from the real domain;

generating, based on the network output, a motion plan for the robot to manipulate the object in the robotic process; and

controlling the robot to execute the robotic process to manipulate the object in accordance with the motion plan.

17. The non-transitory computer-readable storage media of claim 16 , wherein the neural network is trained to learn the geometry of the object in the image.

18. The non-transitory computer-readable storage media of claim 16 , wherein the network does not allocate capacity to model changes in illumination, light color, specularities, or reflections.

19. The non-transitory computer-readable storage media of claim 16 , wherein the network output is an object detection or classification output.

20. The non-transitory computer-readable storage media of claim 16 , wherein the network output is a pose estimation output.