Systems and methods for machine vision robotic processing
Systems, methods, and media for machine vision-guided robotic loading are provided. Such systems and methods can allow for more accurate object detection so as to allow for robotic picking or loading of an object from a pile or group of objects. In some embodiments, a camera is controlled to capture a background image of the object or group of objects. One or more illumination sources (such as, e.g., laser illuminators) are controlled to project immunization toward the object or group of objects within the field-of-view of the camera. The camera may be controlled to capture a scan image comprising the projected illumination. Based on the scan image, a depth map can be determined, which is provided to a machine learning model along with the background image. The output of the machine learning model can then provide information associated with one or more objects, such as location information, orientation information, and the like.
1 . A system for detecting an object on a surface, comprising:
a camera oriented to include the surface in a field-of-view (FOV) of the camera;
a plurality of aimable illumination sources configured to project corresponding illuminations toward the surface within the FOV of the camera; and
a controller comprising a processor that is configured:
to control the camera to capture, with the camera, a first background image of the object on the surface;
to control the plurality of aimable illumination sources to project the corresponding illuminations simultaneously toward the surface to form a projected illumination on the surface;
to control the camera to capture a scan image comprising the projected illumination formed by simultaneously projected corresponding illuminations that intersect the surface at respective positions;
to determine a first depth map based on separating the scan image into a plurality of object scan mask sets, each object scan mask set representing a corresponding image formed only by an illumination of an aimable illumination source chosen from the plurality of aimable illumination sources;
to provide the first depth map and the first background image to a trained machine learning model executed by the processor; and
to receive an output from the machine learning model, wherein the output comprises information associated with the object;
wherein the plurality of aimable illumination sources comprises a first aimable illumination source and a second aimable illumination source, and wherein the first aimable illumination source and the second aimable illumination source have different spectra, and
wherein the controller is further configured to separate the scan image into the plurality of scan image sets each of which is formed only by light having a spectrum of an illumination of said aimable illumination source chosen from the plurality of aimable illumination sources.
2 . The system of claim 1 , wherein each of the aimable illumination sources comprises a line laser.
3 . The system of claim 2 , wherein each of the aimable illumination sources further comprises:
a motor; and
a mirror coupled to the motor,
wherein the mirror is positioned to reflect illumination from the line laser;
wherein the motor is controllable to rotate the mirror based on a received control input.
4 . The system of claim 1 , wherein the controller is further configured:
to controllably scan the projected illumination across the surface within the FOV in a plurality of scan steps by changing a position of a corresponding illumination of each of the plurality of aimable illumination sources on the surface;
to determine an object scan set, comprising a plurality of scan image sets, at each scan step of the plurality of scan steps; and
to determine the first depth map based on the object scan set.
5 . The system of claim 4 , wherein the controller is further configured:
to determine a baseline scan set for each of the plurality of aimable illumination sources; and
to determine the first depth map based on the object scan set by subtracting, pixel by pixel, the baseline scan set from the object scan set to identify a pixel shift in the projected illumination captured in a given object scan mask set as compared to a corresponding baseline scan mask set, for each scan step, wherein the shift is due to the projected illumination intersecting with the object.
6 . The system of claim 1 , wherein the controller is further configured to control a robotic picking device to acquire the object according to the output of the machine learning model.
7 . The system of claim 1 , wherein the controller is further configured:
to determine, based on the output of the machine learning model, a first object among a plurality of objects to prioritize; and
to provide a first signal to a robotic picking device to aid in acquiring the first object.
8 . The system of claim 7 , wherein the controller is further configured, after the first object has been removed from the surface:
to provide a second depth map and a second background image to the machine learning model;
to receive a second output from the machine learning model and, based on the second output, determine a second object among the plurality of objects to prioritize; and
to provide a second signal to the robotic picking device to aid in acquiring the second object.
9 . The system of claim 1 , wherein the machine learning model is configured:
to identify the object; and
wherein the information associated with the object comprises at least one of a mask, a key a point, a class, and a bounding box for the object identified by the machine learning model.
10 . A method for detecting an object, the method comprising:
capturing, with a camera, a first background image of the object on a surface that is positioned in a field-of-view (FOV) of the camera;
controlling a plurality of aimable illumination sources to simultaneously project corresponding illuminations toward the surface within the FOV of the camera to form a projected illumination on the surface;
capturing, with the camera, a scan image comprising the projected illumination formed by simultaneously projected corresponding illuminations that intersect the surface at respective positions;
determining a first depth map based on separating the scan image into plurality of object scan mask sets, each object scan mask set representing a corresponding image formed only by an illumination of an aimable illumination source chosen from the plurality of aimable illumination sources; providing the first depth map and the first background image to a machine learning model; and receiving an output from the machine learning model, wherein the output comprises information associated with the object,
wherein the plurality of aimable illumination sources comprises a first aimable illumination source and a second aimable illumination source, and wherein the first aimable illumination source and the second aimable illumination source have different spectra, and
wherein the controller is configured to separate the scan image into the plurality of scan image sets each of which is formed only by light having a spectrum of an illumination of said aimable illumination source chosen from the plurality of aimable illumination sources.
11 . The method of claim 10 , wherein each of the aimable illumination sources comprises a line laser.
12 . The method of claim 11 , wherein each of the aimable illumination sources further comprises:
a motor; and
a mirror coupled to the motor and positioned to reflect illumination from the line laser; and
wherein the controlling the plurality of aimable illumination sources comprises providing a control input to the motor of each aimable illumination source.
13 . The method of claim 10 , wherein the method further comprises:
controllably scanning the projected illumination across the surface within the FOV in a plurality of scan steps by changing a position, on the surface, of the corresponding illumination of an aimable illumination source of the plurality of aimable illumination sources,
determining an object scan set, comprising a plurality of scan image sets, at each scan step of the plurality of scan steps, and
determining the first depth map based on the object scan set.
14 . The method of claim 13 , wherein the method further comprises:
determining a baseline scan set for each of the plurality of aimable illumination sources; and
wherein the determining the first depth map based on the object scan set comprises subtracting, pixel by pixel, the baseline scan set from the object scan set to identify a pixel shift in the projected illumination captured in a given object scan mask set as compared to a corresponding baseline scan mask set, for each scan step, wherein the shift is due to the projected illumination intersecting with the object.
15 . The method of claim 10 , wherein the method further comprises generating a control signal for a robotic picking device to acquire the object according to the output of the machine learning model.
16 . The method of claim 10 , wherein the method further comprises:
determining, based on the output of the machine learning model, a first object among a plurality of objects on the surface to prioritize; and
controlling a robotic picking device to acquire the first object.
17 . The method of claim 16 , wherein the method further comprises determining a second object among the plurality of objects to prioritize, after the first object has been removed from the surface.
18 . The method of claim 10 , wherein the machine learning model is configured to identify the object; and wherein the information associated with the object comprises at least one of a mask, a key point, a class, and a bounding box for the object identified by the machine learning model.
19 . A system for detecting an object on a surface, comprising:
a camera oriented to include the surface in a field-of-view (FOV) of the camera;
a plurality of aimable illumination sources configured to project corresponding illuminations toward the surface within the FOV of the camera; and
a controller comprising a processor that is configured:
to control the camera to capture, with the camera, a first background image of the object on the surface;
to control the plurality of aimable illumination sources to project the corresponding illuminations simultaneously toward the surface to form a projected illumination on the surface;
to control the camera to capture a scan image comprising the projected illumination formed by simultaneously projected corresponding illuminations that intersect the surface at respective positions;
to determine a first depth map based on separating the scan image into a plurality of object scan mask sets, each object scan mask set representing an image formed only by an illumination of an aimable illumination source chosen from the plurality of aimable illumination sources;
to provide the first depth map and the first background image to a trained machine learning model executed by the processor; and
to receive an output from the machine learning model, wherein the output comprises information associated with the object;
wherein the controller is further configured:
to controllably scan the projected illumination across the surface within the FOV in a plurality of scan steps by changing a position of the corresponding illumination of each of the plurality aimable illumination source on the surface;
to determine an object scan set, comprising a plurality of scan image sets, at each scan step of the plurality of scan steps; and
to determine the first depth map based on the object scan set.
20 . The system of claim 19 , wherein the machine learning model is configured:
to identify the object; and
wherein the information associated with the object comprises at least one of a mask, a key point, a class, and a bounding box for the object identified by the machine learning model.