Granted Patent B2

US 12,468,279 · App. 17/933,323 · Granted Nov 11, 2025

Inventors: Matthew C. Gombolay (Atlanta, GA); Michael J. Johnson (Atlanta, GA); Ruisen Liu (Atlanta, GA); Nakul Gopalan (Atlanta, GA)

Assignee: Georgia Tech Research Corporation

G05B19/401B23P21/00G05B2219/36412G05B2219/39001G05B2219/40126

Abstract

Claims (71)

1 . A method to manufacture or assemble a workpiece comprising landmarks or holes, the method comprising:

while a camera mounted on a robot arm is swept through a workspace, capturing, by a processor, a set of images of a first workpiece, wherein the first workpiece comprises one or more holes;

determining, by the processor using the captured set of images, a surface normal orientation for the first workpiece, wherein the surface normal orientation is determined by:

determining, by the processor, a point cloud data object of the captured set of images;

determining, by the processor, a plane intersecting a pre-defined portion of the point cloud data object; and

determining, by the processor, a normal vector or direction from the plane as the surface normal orientation;

directing, by the processor, the robot arm or a mechanized actuator of the camera, to orient the camera to an orientation to capture at least one image of the first workpiece at the surface normal orientation;

generating, by the processor, a composite image or a data model of the first workpiece by aggregating at least a portion of the set of images of the first workpiece, wherein the composite image or the data model includes an image of, or data object representing, the first workpiece having a workpiece region captured at a plurality of surface normal orientations; and

determining, by the processor, using the composite image or a portion thereof, coordinates of landmark locations of the one or more holes of the first workpiece and/or characteristics of the one or more holes;

wherein the determined coordinates of landmark locations and/or characteristics of the one or more landmarks used by a robotic system to perform manufacturing action on, or to, a second workpiece to make it resemble the first workpiece.

2 . The method of claim 1 , wherein the determining of the characteristics of the one or more landmarks includes:

determining, via a classification operation, a classification value for at least one of the one or more landmarks.

3 . The method of claim 2 , wherein the classification operation is performed using a trained AI model or operation.

4 . The method of claim 3 , wherein the trained AI model or operation includes a trained neural network.

5 . The method of claim 1 , wherein generating the composite image or the data model of the first workpiece, including a first image and a second image, comprises:

determining candidate features in the first image and the second image, wherein the first image and the second image each has a workpiece region captured at a given surface normal orientation;

aligning the candidate features of the first image and the second image; and

generating, via a stitching operation, the composite image or the data model using the first image and the second image, or portions thereof.

6 . The method of claim 1 , further comprising:

determining, by the processor, a first dynamic motion primitive data object;

updating, by the processor, via an AI-learning operation, the first dynamic motion primitive data object using the coordinates of landmark locations of the one or more landmarks of the first workpiece as a target parameter in a reinforcement learning operation, wherein the first dynamic motion primitive data object, or instructions derived therefrom, is stored and subsequently used to direct a robotic system in the manufacturing of the second workpiece.

7 . The method of claim 6 , wherein the reinforcement learning operation employs a reward function that minimizes, from multiple guided learning examples, at least one of (i) difference or residual of measured or determined force in a direction of action to a pre-defined value (static or dynamic), (ii) difference or residual of measured or determined position to a pre-defined trajectory of movement, (iii) difference or residual of the measured or determined position of a target of action, and (iv) a combination thereof.

8 . The method of claim 7 , further comprising:

determining, by the processor, a second dynamic motion primitive data object;

updating, by the processor, via then AI-learning operation, the second dynamic motion primitive data object using the coordinates of landmark locations of the one or more landmarks of the first workpiece, wherein the second dynamic motion primitive data object, or an instruction derived therefrom, is stored and subsequently used in combination with the first dynamic motion primitive to direct the robotic system in the manufacturing of the second workpiece.

9 . The method of claim 6 , wherein the first dynamic motion primitive data object, or the instructions derived therefrom, includes at least one of:

first kinetic instructions to move from a first position to a second position to insert an attachment at a landmark defined as a target location;

second kinetic instructions to move an effector component from a third position to a fourth position to drill a landmark at a target location;

third kinetic instructions to move the effector component from a first position to a second position to spray or dispense a propellant over a set of target location of regions;

fourth kinetic instructions to move the effector component from a first position to a second position to cut a set of target location of regions; and

fifth kinetic instructions to move the effector component from a first position to a second position to rotate a wrap over a set of target location of regions.

10 . The method of claim 6 , wherein the first dynamic motion primitive data object, or the instructions derived therefrom, includes, at least:

a first movement primitive data object, the first movement primitive data object being associated with movement of the robot arm from a first position to a second position, wherein the second position is associated with a task movement; and

a second movement primitive data object associated with the task movement.

11 . The method of claim 10 , wherein the second movement primitive data object includes instructions to at least one of an insertion task, a drilling task, a holding task, a spraying or coating task, and a wrapping task.

12 . The method of claim 1 , further comprising:

defining, by the processor, a salient region of the workspace based on two or more coordinate received from measurements of the robotic arm established by a user.

13 . The method of claim 1 , further comprising:

defining, by the processor, a salient region of the workspace based on two or more coordinates received from a user via a computer user interface.

14 . The method of claim 12 , wherein the salient region, includes a first coordinate position and a second coordinate position, the first and second coordinate positions being interpolated to define a grid of waypoints parallel to a determined surface of the workpiece.

15 . The method of claim 1 , further comprising:

determining instructions to perform the manufacturing action on, or to, the second workpiece to make it resemble the first workpiece based on motive primitives generated from the determined coordinates of the landmark locations and/or the characteristics of the one or more landmarks.

16 . A system comprising:

a processor;

a memory having instructions stored thereon, wherein execution of the instructions by the processor causes the processor to:

receive (i) image data from a camera mounted on a robot arm and (ii) coordinate data of a landmark on the robot arm;

while the camera mounted on the robot arm is swept through a workspace, capture a set of images of a first workpiece and corresponding set of coordinate data of the landmark, wherein the first workpiece comprises one or more landmarks;

determine, from the captured set of images, a surface normal orientation for the first workpiece, wherein the surface normal orientation is determined by:

determining, by the processor, a point cloud data object of the captured set of images:

determining, by the processor, a plane intersecting a pre-defined portion of the point cloud data object; and

determining, by the processor, a normal vector or direction from the plane as the surface normal orientation;

direct the robot arm or a mechanized actuator of the camera, to orient the camera to an orientation to capture at least one image of the first workpiece at the surface normal orientation;

generate a composite image or a data model of the first workpiece by aggregating at least a portion of the set of images of the first workpiece, wherein the composite image or the data model includes an image of, or data object representing, the first workpiece having a workpiece region captured at a plurality of surface normal orientations; and

determine using the composite image or a portion thereof and the corresponding set of coordinate data, coordinates of landmark locations of the one or more landmarks of the first workpiece and/or characteristics of the one or more landmarks;

wherein the determined coordinates of landmark locations and/or characteristics of the one or more landmarks are used by a robotic system to perform manufacturing action on, or to, a second workpiece to make it resemble the first workpiece.

17 . The system of claim 16 , further comprising:

the robot arm;

the camera mounted to said robot arm; and

a robot controller, the robot controller being operatively connected to the processor and configured to receive the image data and the coordinate data.

18 . The system of claim 17 , wherein robot controller is configured to receive instructions to perform a manufacturing task based on a set of movement primitives derived from the determined coordinates of landmark locations and/or the characteristics of the one or more landmarks.

19 . A non-transitory compute readable medium having instructions stored thereon, wherein execution of the instructions by a processor, causes the processor to:

receive (i) image data from a camera mounted on a robot arm and (ii) coordinate data of a landmark on the robot arm;

while the camera mounted on the robot arm is swept through a workspace, capture a set of images of a first workpiece and corresponding set of coordinate data of the landmark, wherein the first workpiece comprises one or more landmarks;

determine, using the captured set of images, a surface normal orientation for the first workpiece, wherein the surface normal orientation is determined by:

determining, by the processor, a point cloud data object of the captured set of images;

determining, by the processor, a plane intersecting a pre-defined portion of the point cloud data object; and

determining, by the processor, a normal vector or direction from the plane as the surface normal orientation;

direct the robot arm or a mechanized actuator of the camera, to orient the camera to an orientation to capture at least one image of the first workpiece at the surface normal orientation;

generate a composite image or a data model of the first workpiece by aggregating at least a portion of the set of images of the first workpiece, wherein the composite image or the data model includes an image of, or data object representing, the first workpiece having a workpiece region captured at a plurality of surface normal orientations; and

determine using the composite image or a portion thereof and the corresponding set of coordinate data, coordinates of landmark locations of the one or more landmarks of the first workpiece and/or characteristics of the one or more landmarks;

wherein the determined coordinates of landmark locations and/or characteristics of the one or more landmarks are used by a robotic system to perform manufacturing action on, or to, a second workpiece to make it resemble the first workpiece.

Assignments (1)

Continuity (2)

References Cited (26)