Use of artificial intelligence models to identify fasteners and perform related operations
Aspects of the disclosure are directed towards artificial intelligence-based modeling of target objects, such as aircraft parts. In an example, a system initially trains a machine learning (ML) model based on synthetic images generated based on multi-dimensional representation of target objects. The same system or a different system subsequently further trains the ML model based on actual images generated by cameras positioned by robots relative to target objects. The ML model can be used to process an image generated by a camera positioned by a robot relative to a target object based on a multi-dimensional representation of the target object. The output of the ML model can indicate, for a detected target, position data, a target type, and/or a visual inspection property. This output can then be used to update the multi-dimensional representation, which is then used to perform robotics operations on the target object.
1. A computer-implemented method comprising:
receiving a multi-dimensional representation of a part of an object, the object including a craft or a vehicle, the part including a target;
causing a robot to position a camera relative to the target based on the multi-dimensional representation;
receiving an image generated by the camera while the camera is positioned relative to the target, the image showing at least the target;
determining, by using a machine learning model based on the image, a visual inspection property of the target; and
storing the visual inspection property of the target in a data store.
2. The computer-implemented method of claim 1 , wherein the part includes an aircraft part, and wherein the computer-implemented method further comprises:
receiving a plurality of images generated by the camera, each of the plurality of images corresponding to one of a plurality of targets of the aircraft part;
determining, by using the machine learning model and based on the plurality of images, a visual inspection property for each target of the plurality of targets; and
storing the visual inspection property of each target in the data store.
3. The computer-implemented method of claim 2 , wherein the target includes a fastener, and wherein the visual inspection property of the fastener indicates at least one of: a corrosion of the fastener, a corrosion of an area around the fastener, a fastener presence, a fastener absence, a fastener damage, or a surrounding area damage.
4. The computer-implemented method of claim 2 , wherein the data store stores the visual inspection property of the target in association with an identifier of: at least one of the aircraft part or an aircraft.
5. The computer-implemented method of claim 4 , wherein the data store further stores the multi-dimensional representation of the aircraft part in association with the visual inspection property of the target and the identifier of at least one of: the aircraft part or the aircraft.
6. The computer-implemented method of claim 1 , further comprising:
determining first pose data indicated by the multi-dimensional representation for the target in a first coordinate system of the part;
determining, based on one or more images, second pose data of the target in a second coordinate system of the robot; and
controlling an operation of the robot on the target based on the second pose data.
7. The computer-implemented method of claim 6 , wherein the second pose data is determined in the first coordinate system based on an inverse kinematics transformation from a third coordinate system of the camera to the second coordinate system of the robot and a forward kinematics transformation from the second coordinate system of the robot to the first coordinate system of the part.
8. The computer-implemented method of claim 6 , further comprising:
determining that an offset between the first pose data and the second pose data exists; and
storing an updated multi-dimensional representation of the part that includes the second pose data instead of the first pose data, wherein the updated multi-dimensional representation is usable to control the operation on the target.
9. The computer-implemented method of claim 8 , wherein the updated multi- dimensional representation includes the visual inspection property.
10. The computer-implemented method of claim 6 , wherein the camera is positioned relative to the target by setting the camera as a tool center point (TCP) in the second coordinate system of the robot and by expressing the first pose data in the second coordinate system based on a forward kinematics transformation.
11. A system comprising:
one or more processors; and
one or more memory storing instructions that, upon execution by the one or more processors, configure the system to:
receive a multi-dimensional representation of a part of an object, the object including a craft or a vehicle, the part including a target;
cause a robot to position a camera relative to the target based on the multi-dimensional representation;
receive an image generated by the camera while the camera is positioned relative to the target, the image showing at least the target;
determine, by using a machine learning model based on the image, a visual inspection property of the target; and
store the visual inspection property of the target in a data store.
12. The system of claim 11 , wherein the execution of the instructions further configures the system to:
receive a plurality of images, each of the plurality of images corresponding to one of a plurality of targets of the part;
determine, by using the machine learning model and based on the plurality of images, a visual inspection property for each target of the plurality of targets; and
store the visual inspection property of each target in the data store.
13. The system of claim 11 , wherein the visual inspection property indicates at least one of: a corrosion, a target presence, a target absence, a target damage, or a damage to an area surrounding the target.
14. The system of claim 12 , wherein the data store stores the visual inspection property of the target in association with an identifier of: at least one of the part or the object.
15. The system of claim 14 , wherein the data store further stores the multi- dimensional representation of the part in association with the visual inspection property of the target.
16. One or more non-transitory computer-readable storage media storing instructions that, upon execution on a system, cause the system to perform operations comprising:
receiving a multi-dimensional representation of a part of an object, the object including a craft or a vehicle, the part including a target;
causing a robot to position a camera relative to the target based on the multi-dimensional representation;
receiving an image generated by the camera while the camera is positioned relative to the target, the image showing at least the target;
determining, by using a machine learning model based on the image, a visual inspection property of the target; and
storing the visual inspection property of the target in a data store.
17. The one or more non-transitory computer-readable storage media of claim 16 , wherein the operations further comprise:
receiving a plurality of images generated by the camera, each of the plurality of images corresponding to one of a plurality of targets of the part;
determining, by using the machine learning model and based on the plurality of images, a classification for each target of the plurality of targets; and
storing the classification of each target in the data store.
18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the data store further stores the multi-dimensional representation of the part in association with the visual inspection property of the target and the corresponding classification of the target.
19. The one or more non-transitory computer-readable storage media of claim 16 , wherein the operations further comprise:
receiving a plurality of images generated by the camera, each of the plurality of images corresponding to one of a plurality of targets of the part;
determining, by using the machine learning model and based on the plurality of images, pose data for each target of the plurality of targets; and
storing the pose data of each target in the data store.
20. The one or more non-transitory computer-readable storage media of claim 16 , wherein the data store further stores the multi-dimensional representation of the part in association with the visual inspection property of the target and the corresponding pose data of the target.