Method of in-process detection and mapping of defects in a composite layup
A method of detecting defects in a composite layup includes capturing, using an infrared camera, reference images of a reference layup being laid up by a reference layup head. The method also includes manually reviewing the reference images for defects, and generating reference defect masks indicating defects in the reference images. The method further includes training, using the reference images and reference defect masks, a neural network, creating a machine learning model that, given a production image as input, outputs a production defect mask indicating the defect location and the defect type of each defect. The method also includes capturing, using an infrared camera, production images of a production layup being laid up by the production layup head, and applying the model to the production images to automatically generate a production defect masks indicating each defect in the production images.
1. A method of in-process detection of defects in a composite layup, comprising the steps of:
capturing, using an infrared camera of a reference layup head, a series of reference images of a course of a reference layup being laid up by the reference layup head;
manually reviewing the reference images for defects, and generating, for each reference image containing a defect, a reference defect mask indicating a defect location of each defect in the reference image;
training, using the reference images and corresponding reference defect masks, a convolutional neural network to perform segmentation in a manner creating a machine learning defect identification model that, given a production image as input, outputs a production defect mask indicating the defect location and the defect type of each defect detected by the model;
capturing, using an infrared camera of a production layup head, a series of production images of at least one course of a production layup being laid up by the production layup head; and
applying the model to the production images, and automatically generating, for each production image containing a defect, the production defect mask indicating the defect location and identifying the defect type of each defect detected by the model;
spatially aligning each production defect mask with a digital representation of a region of the course captured in the corresponding production image, and resulting in a defect map containing an on-part location and the defect type of each defect associated with the course;
inferring the on-part location of each defect associated with the course, based on at least one of the following:
the location of the production layup head relative to the production layup at an image capture time of the production image containing the defect; and
the location and orientation of the infrared camera relative to the production layup head.
2. The method of claim 1 , wherein the model parameters comprise at least one of the following:
the quantity of convolution layers and down-sampling layers in an encoder portion of the model architecture;
the quantity of convolution layers and up-sampling layers in a decoder portion of the model architecture;
the size of the reference images that are input into the convolutional neural network; and
the types of activation functions of the convolutional neural network.
3. The method of claim 1 , wherein
inferring the on-part location of each defect associated with the course is based on
a pixel-wise location of each defect detected by the model.
4. The method of claim 1 , further comprising:
physically locating, using the defect map, one or more of the defects associated with the course, and performing at least one of the following:
repairing one or more of the defects prior to applying subsequent courses of the production layup; and
adjusting at least one layup process parameter in a manner mitigating subsequent formation of defects of the same defect type at one or more of the defects indicated on the defect map.
5. The method of claim 4 , wherein adjusting at least one layup process parameter comprises adjusting at least one of the following:
a travel speed of the production layup head when applying a course onto a substrate;
a heat output of a heating device for heating the substrate prior to application of the course; and
a compaction pressure of a compaction device compacting the course against the substrate.
6. The method of claim 1 , wherein the course is laid up as either a single-width tape or as a plurality of side-by-side tows, the step of identifying the defect type of each defect comprises:
identifying a defect as one of the following: a twist, a fold, a gap, an overlap, bridging, puckers, wrinkles, a missing tape or tow, low-quality tack, foreign object debris, a resin ball, a fuzz ball.
7. The method of claim 1 , wherein capturing the reference images comprises:
capturing, using the infrared camera, the reference images at a location on the course immediately aft of a compaction device of the reference layup head, the compaction device configured to compact the course against a substrate.
8. The method of claim 7 , further comprising:
applying, using a heating device of the reference layup head, heat to a region of the substrate immediately forward of the compaction device.
9. The method of claim 1 , further comprising:
deleting references images that are at least one of the following: off-part images, obstructed-view images, and non-layup-head-movement images.
10. The method of claim 1 , wherein each reference image is comprised of pixels having intensity values, the method further comprising:
preprocessing the reference images in preparation for training the neural network, by performing at least one the following:
cropping each reference image to remove non-layup features from the reference image; and
normalizing each reference image by centering the intensity values of the pixels of the reference image at a mean pixel value, and rescaling the range of the intensity values of the pixels by a standard deviation.
11. The method of claim 1 , wherein generating the reference defect masks is performed using an image-annotation software program.
12. The method of claim 1 , wherein training the neural network to perform segmentation comprises:
training the neural network to perform one of semantic image segmentation or instance segmentation.
13. The method of claim 1 , further comprising:
manually reviewing the production images for defects, and generating, for each production image containing a defect detected manually, a reference defect mask indicating the defect location and the defect type of each defect in the production image; and
inputting the production images and corresponding reference defect masks into the neural network to further train the model for increasing the accuracy of the model in detecting defects in the production images of subsequent courses applied by the production layup head.
14. A method of in-process detection of defects in a composite layup, comprising:
capturing, using one or more infrared cameras of a reference layup head, a series of reference images of a course of a reference layup being laid up by the reference layup head, the course comprising a plurality of tows in side-by-side relation to each other;
manually reviewing the reference images for defects, and generating, for each reference image containing a defect, a reference defect mask indicating tow boundaries and a unique tow identification number for each tow in the course, and identifying a defect type of each defect in the reference image;
training, using the reference images and corresponding reference defect masks, a convolutional neural network to perform instance segmentation in a manner creating a machine learning defect identification model that, given a production image as input, outputs a production defect mask indicating the tow boundaries and tow identification number of each tow in the course, and the defect location and the defect type of each defect detected by the model;
capturing, using one or more infrared cameras of a production layup head, a series of production images of a course of tows of a production layup being laid up by the production layup head; and
applying the model to the production images, and automatically generating, for each production image containing a defect, the production defect mask indicating the defect type, identifying the defect location along a lengthwise direction of the course, and indicating the tow identification number of one or more of the tows containing the defect;
spatially aligning each production defect mask with a digital representation of a region of the course captured in the corresponding production image, and resulting in a defect map containing an on-part location and the defect type of each defect associated with the course;
inferring the on-part location of each defect associated with the course, based on at least one of the following:
the tow identification number of one or more of the tows containing the defect;
the location of the production layup head relative to the production layup at an image capture time of the production image containing the defect; and
the location and orientation of the infrared camera relative to the production layup head.
15. The method of claim 14 , wherein
inferring the on-part location of each defect associated with the course is based on
a pixel-wise location of each defect as detected by the model.
16. The method of claim 14 , further comprising:
physically locating, using the defect map, one or more of the defects associated with the course, and performing at least one of the following:
repairing one or more of the defects prior to applying subsequent courses of the production layup; and
adjusting at least one layup process parameter in a manner mitigating subsequent formation of defects of the same defect type at one or more of the defects indicated on the defect map.
17. The method of claim 16 , wherein adjusting at least one layup process parameter comprises adjusting at least one of the following:
a travel speed of the production layup head when applying a course onto a substrate;
a heat output of a heating device for heating the substrate prior to application of the course; and
a compaction pressure of a compaction device compacting the course against the substrate.
18. The method of claim 14 , wherein the course is laid up as either a single-width tape or as a plurality of side-by-side tows, the step of identifying the defect type of each defect comprises:
identifying a defect as one of the following: a twist, a fold, a gap, an overlap, bridging, puckers, wrinkles, a missing tape or tow, low-quality tack, foreign object debris, a resin ball, a fuzz ball.
19. A method of in-process mapping of defects in a composite layup, comprising:
manually generating, using an image annotation software program, a reference defect mask for each of a plurality of reference images of a course of a reference layup during application by a reference layup head onto a substrate, each reference image showing at least one defect in the course;
training, using the reference images and corresponding reference defect masks, a convolutional neural network to perform instance segmentation in a manner creating a machine learning defect identification model that, given a production image as input, outputs the production defect mask indicating a defect location and a defect type of each defect detected by the model;
capturing, using an infrared camera of a production layup head, a series of production images of one or more courses of a production layup being laid up by the production layup head; and
applying the model to the production images in real time to automatically generate, for each production image containing a defect, the production defect mask indicating the defect location of each defect detected by the model;
spatially aligning each production defect mask with a digital representation of a region of the course captured in the corresponding production image, and resulting in a defect map containing an on-part location of each defect associated with the course;
inferring the on-part location of each defect associated with the course, based on at least one of the following:
the location of the production layup head relative to the production layup at an image capture time of the production image containing the defect; and
the location and orientation of the infrared camera relative to the production layup head.
20. The method of claim 19 , wherein the model parameters comprise at least one of the following:
the quantity of convolution layers and down-sampling layers in an encoder portion of the model architecture;
the quantity of convolution layers and up-sampling layers in a decoder portion of the model architecture;
the size of the reference images that are input into the convolutional neural network; and
the types of activation functions of the convolutional neural network.