Identifying and avoiding obstructions using depth information in a single image
A farming machine includes one or more image sensors for capturing an image as the farming machine moves through the field. A control system accesses an image captured by the one or more sensors and identifies a distance value associated with each pixel of the image. The distance value corresponds to a distance between a point and an object that the pixel represents. The control system classifies pixels in the image as crop, plant, ground, etc. based on depth information in in the pixels. The control system generates a labelled point cloud using the labels and depth information, and identifies features about the crops, plants, ground, etc. in the point cloud. The control system generates treatment actions based on any of the depth information, visual information, point cloud, and feature values. The control system actuates a treatment mechanism based on the classified pixels.
1. A method for avoiding an obstruction by a machine that moves through an operational environment, the machine including a plurality of mechanisms for performing machine actions:
accessing a single image of the operational environment from an image sensor as the machine moves through the operational environment, the single image comprising one or more pixels representing at least a substrate and an obstruction;
applying a depth identification model to the single image, the depth identification model:
determining, for each pixel in the single image, a distance between the image sensor and the substrate or the obstruction represented by the pixel, a depth identification module including a plurality of layers in a convolutional neural network configured to identify distances between sensors and representative pixels in single images, wherein the single image is encoded onto a first neural network layer as an encoded image and transformed to a reduced image with latent features classified as distances corresponding to pixels on a second neural network layer,
classifying, based on the determined distance for each pixel, a first set of pixels in the single image as the substrate, and
classifying, based on the determined distance for each pixel, a second set of pixels in the single image as the obstruction; and
actuating a mechanism of the plurality of mechanisms to perform a machine action that changes a direction of the machine to avoid the classified obstruction, the machine action selected based on the determined distance for pixels in the first set of pixels representing the substrate and pixels in the second set of pixels representing the obstruction.
2. The method of claim 1 , further comprising modifying an operating parameter of the machine based on the determined distances of the pixels in the single image.
3. The method of claim 2 , wherein modifying the operating parameter includes modifying any of:
a speed of the machine;
a height of a mechanism of the plurality of mechanisms of the machine relative to the substrate; and
a position of a mechanism of the plurality of mechanisms of the machine.
4. The method of claim 1 , further comprising modifying a sensor parameter of the image sensor based on the determined distances of the pixels in the single image.
5. The method of claim 1 , further comprising determining a separation between the substrate and a mechanism of the plurality of mechanisms of the machine based on a known position of the mechanism on the machine and the sensor.
6. The method of claim 1 , wherein the distance of the obstruction is closer to the machine than the distance of the substrate.
7. The method of claim 1 , wherein the distance of the obstruction is farther from the machine than the distance of the substrate.
8. The method of claim 1 , wherein encoding the single image further comprises applying one or more transformation functions including a set of weights and parameters to transform data in the encoded image to the reduced image.
9. The method of claim 1 , further comprising generating a depth map comprising the determined distances of the pixels in the single image and wherein the depth map comprises the distances of substrate and the obstruction.
10. A machine configured to perform machine actions and avoid an obstruction as the machine moves through an operational environment, the machine comprising:
a plurality of mechanisms configured to perform machine actions as the machine travels through the operational environment;
an image sensor to capture a single image of the operational environment as the machine moves through the operational environment, the single image comprising one or more pixels representing at least a substrate and an obstruction; and
a processor; and
a non-transitory computer readable storage medium comprising computer program instructions that, when executed by the processor, cause the processor to:
access the single image of the operational environment from the image sensor;
apply a depth identification model to the single image, the depth identification model to:
determine, for each pixel in the single image, a distance between the image sensor and the substrate or the obstruction represented by the pixel, a depth identification module including a plurality of layers in a convolutional neural network configured to identify distances between sensors and representative pixels in single images, wherein the single image is encoded onto a first neural network layer as an encoded image and transformed to a reduced image with latent features classified as distances corresponding to pixels on a second neural network layer,
classify, based on the determined distance for each pixel, a first set of pixels in the single image as the substrate, and
classify, based on the determined distance for each pixel, a second set of pixels in the single image as the obstruction; and
actuate a mechanism of the plurality of mechanisms to perform a machine action that changes a direction of the machine to avoid the classified obstruction, the machine action selected based on the determined distance for pixels in the first set of pixels representing the substrate and pixels in the second set of pixels representing the obstruction.
11. The machine of claim 10 , wherein the computer program instructions, when executed by the processor, further cause the processor to:
modify an operating parameter of the machine based on the determined distances of the pixels in the single image.
12. The machine of claim 11 , wherein modifying the operating parameter causes the processor to modify any of:
a speed of the machine;
a height of a mechanism of the plurality of mechanisms of the machine relative to the substrate;
a height of a mechanism of the plurality of mechanisms of the machine relative to the obstruction; and
a position of a mechanism of the plurality of mechanisms of the machine.
13. The machine of claim 10 , wherein the computer program instructions, when executed by the processor, further causes the processor to:
modify a sensor parameter of the image sensor based on the determined distances of the pixels in the single image.
14. The machine of claim 10 , wherein the computer program instructions, when executed by the processor, further causes the processor to:
determine a separation between the substrate and a mechanism of the plurality of mechanisms of the machine based on a known position of the mechanism on the machine and the sensor.
15. The machine of claim 10 , wherein the distance of the obstruction is closer to the machine than the distance of the substrate.
16. The machine of claim 10 , wherein the distance of the obstruction is farther from the machine than the distance of the substrate.
17. The machine of claim 10 , wherein encoding the single image further causes the computer program instructions, when executed by the processor, to apply one or more transformation functions including a set of weights and parameters to transform data in the encoded image to the reduced image.
18. The machine of claim 10 , wherein the computer program instructions, when executed by the processor, further causes the processor to generate a depth map comprising the determined distances of the pixels in the single image and wherein the depth map comprises the distances of substrate and the obstruction.
19. A non-transitory computer readable storage medium comprising computer program instructions for avoiding an obstruction by a machine that moves through an operational environment, the machine including a plurality of mechanisms for performing machine actions, computer program instructions when executed by a processor causing the processor to:
access a single image of the operational environment from an image sensor as the machine moves through the operational environment, the single image comprising one or more pixels representing at least a substrate and an obstruction;
apply a depth identification model to the single image, the depth identification model causing the processor to:
determine, for each pixel in the single image, a distance between the image sensor and the substrate or the obstruction represented by the pixel, a depth identification module including a plurality of layers in a convolutional neural network configured to identify distances between sensors and representative pixels in single images, wherein the single image is encoded onto a first neural network layer as an encoded image and transformed to a reduced image with latent features classified as distances corresponding to pixels on a second neural network layer,
classify, based on the determined distance for each pixel, a first set of pixels in the single image as the substrate, and
classify, based on the determined distance for each pixel, a second set of pixels in the single image as the obstruction; and
actuate a mechanism of the plurality of mechanisms to perform a machine action that changes a direction of the machine to avoid the classified obstruction, the machine action selected based on the determined distance for pixels in the first set of pixels representing the substrate and pixels in the second set of pixels representing the obstruction.