Training of artificial intelligence model
Aspects of the disclosure are directed towards artificial intelligence-based modeling of target objects, such as aircraft parts. In an example, a system initially trains a machine learning (ML) model based on synthetic images generated based on multi-dimensional representation of target objects. The same system or a different system subsequently further trains the ML model based on actual images generated by cameras positioned by robots relative to target objects. The ML model can be used to process an image generated by a camera positioned by a robot relative to a target object based on a multi-dimensional representation of the target object. The output of the ML model can indicate, for a detected target, position data, a target type, and/or a visual inspection property. This output can then be used to update the multi-dimensional representation, which is then used to perform robotics operations on the target object.
1. A computer-implemented method comprising:
causing, based on a multi-dimensional representation of a part of an object, a robot to position a camera relative to a first target included in the part, the object including at least one of a craft or a vehicle;
receiving a first image generated by the camera while the camera is positioned relative to the first target, the first image showing at least the first target;
generating a first input to a machine learning model based on the first image;
determining a first output of the machine learning model based on the first input, the first output indicating first position data of a first detected target in the first image, the first detected target corresponding to the first target;
determining first corrected position data based on the first image;
generating first training data based on the first corrected position data and the first position data; and
storing the first training data for machine learning model training.
2. The computer-implemented method of claim 1 further comprising:
determining an offset between the first position data and the first corrected position data;
determining that the offset exceeds a threshold offset; and
including the first image and at least one of the offset or the first corrected position data in the first training data, wherein the machine learning model training is based on a loss function that minimizes the offset.
3. The computer-implemented method of claim 1 further comprising:
causing, based on the multi-dimensional representation of the part, the robot to position the camera relative to a second target included in the part;
receiving a second image generated by the camera while the camera is positioned relative to the second target, the second image showing at least the second target;
generating a second input to the machine learning model based on the second image;
determining a second output of the machine learning model based on the second input, the second output indicating second position data of a second detected target in the second image, the second detected target corresponding to the second target;
determining an offset between the second position data and second corrected position data;
determining that the offset is smaller than a threshold offset; and
excluding the second image, the offset, and the second corrected position data from the first training data.
4. The computer-implemented method of claim 1 further comprising:
causing, based on the multi-dimensional representation of the part, the robot to position the camera relative to a second target included in the part;
receiving a second image generated by the camera while the camera is positioned relative to the second target, the second image showing at least the second target;
generating a second input to the machine learning model based on the second image;
determining a second output of the machine learning model based on the second input, the second output indicating second position data of a second detected target in the second image, the second detected target corresponding to the second target;
generating pose data of the second target based on the second position data; and
generating an updated multi-dimensional representation of the part by at least updating the multi-dimensional representation based on the pose data.
5. The computer-implemented method of claim 4 further comprising:
causing, based on the updated multi-dimensional representation of the part, the robot to perform an operation on the second target.
6. The computer-implemented method of claim 1 further comprising:
generating, based on the multi-dimensional representation of the part, a synthetic image that shows at least one modeled target that corresponds to at least one target of the part;
generating second training data based on the synthetic image; and
storing the second training data for the machine learning model training.
7. The computer-implemented method of claim 6 , wherein the synthetic image is generated by using an image transformation applied to a modeled target.
8. The computer-implemented method of claim 6 further comprising:
determining, based on the multi-dimensional representation of the part, second position data of a modeled target; and
including, in the second training data, a synthetic image that shows the modeled target and the second position data.
9. The computer-implemented method of claim 1 further comprising:
causing, based on the multi-dimensional representation of the part, the robot to position the camera relative to a second target included in the part;
receiving a second image generated by the camera while the camera is positioned relative to the second target, the second image showing at least the second target;
generating a second input to the machine learning model based on the second image;
determining a second output of the machine learning model based on the second input, the second output indicating a first classification of a second detected target in the second image, the second detected target corresponding to the second target;
determining a corrected classification of the second target; and
including, in the first training data, the second image and the corrected classification.
10. The computer-implemented method of claim 1 further comprising:
causing, based on the multi-dimensional representation of the part, the robot to position the camera relative to a second target included in the part;
receiving a second image generated by the camera while the camera is positioned relative to the second target, the second image showing at least the second target;
generating a second input to the machine learning model based on the second image;
determining a second output of the machine learning model based on the second input, the second output indicating a first visual inspection property of a second detected target in the second image, the second detected target corresponding to the second target;
determining a corrected visual inspection property of the second target; and
including, in the first training data, the second image and the corrected visual inspection property.
11. The computer-implemented method of claim 1 , wherein the first target includes a fastener or a fastener hole, and wherein the machine learning model is trained to classify targets into fastener types, determine target positions, and determine target visual inspection properties.
12. The computer-implemented method of claim 1 , wherein the first position data indicates a bounding box that surrounds the first detected target.
13. A system comprising:
one or more processors; and
one or more memory storing instructions that, upon execution by the one or more processors, configure the system to:
cause, based on a multi-dimensional representation of a part of an object, a robot to position a camera relative to a first target included in the part, the object including at least one of a craft or a vehicle;
receive a first image generated by the camera while the camera is positioned relative to the first target, the first image showing at least the first target;
generate a first input to a machine learning model based on the first image;
determine a first output of the machine learning model based on the first input, the first output indicating first position data of a first detected target in the first image, the first detected target corresponding to the first target;
determine first corrected position data based on first image;
generate first training data based on the first corrected position data and the first position data; and
store the first training data for machine learning model training.
14. The system of claim 13 , wherein the first training data is stored in a data store in association with an identifier of at least: the part or the object.
15. The system of claim 13 , wherein the one or more memory store further instructions that, upon execution by the one or more processors, configure the system to:
cause, based on the multi-dimensional representation of the part, the robot to position the camera relative to a second target included in the part;
receive a second image generated by the camera while the camera is positioned relative to the second target, the second image showing at least the second target;
generate a second input to the machine learning model based on the second image;
determine a second output of the machine learning model based on the second input, the second output indicating a first classification of a second detected target in the second image, the second detected target corresponding to the second target;
determine a corrected classification of the second target; and
include, in the first training data, the second image and the corrected classification.
16. The system of claim 13 , wherein the one or more memory store further instructions that, upon execution by the one or more processors, configure the system to:
cause, based on the multi-dimensional representation of the part, the robot to position the camera relative to a second target included in the part;
receive a second image generated by the camera while the camera is positioned relative to the second target, the second image showing at least the second target;
generate a second input to the machine learning model based on the second image;
determine a second output of the machine learning model based on the second input, the second output indicating a first visual inspection property of a second detected target in the second image, the second detected target corresponding to the second target;
determine a corrected visual inspection property of the second target; and
include, in the first training data, the second image and the corrected visual inspection property.
17. The system of claim 13 , wherein the one or more memory store further instructions that, upon execution by the one or more processors, configure the system to:
determine an offset between the first position data and the first corrected position data;
determine that the offset exceeds a threshold offset; and
include the first image and at least one of the offset or the first corrected position data in the first training data.
18. One or more non-transitory computer-readable storage media storing instructions that, upon execution on a system, cause the system to perform operations comprising:
causing, based on a multi-dimensional representation of a part of an object, a robot to position a camera relative to a first target included in the part, the object including at least one of a craft or a vehicle;
receiving a first image generated by the camera while the camera is positioned relative to the first target, the first image showing at least the first target;
generating a first input to a machine learning model based on the first image;
determining a first output of the machine learning model based on the first input, the first output indicating first position data of a first detected target in the first image, the first detected target corresponding to the first target;
determining first corrected position data based on first image;
generating first training data based on the first corrected position data and the first position data; and
storing the first training data for machine learning model training.
19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the machine learning model is trained for at least one of: classifying targets into classification types associated with a type of the object, determining target positions, or determining target visual inspection properties.
20. The one or more non-transitory computer-readable storage media of claim 18 , wherein the part includes an aircraft part, and wherein the first training data is stored in a data store in association with an identifier of at least: the aircraft part or an aircraft.