Deep learning based image enhancement for additive manufacturing
A method is provided for enhancing image resolution for sequences of 2-D images of additively manufactured products. For each of a plurality of additive manufacturing processes, the process obtains a respective plurality of sequenced low-resolution 2-D images of a respective product during the respective additive manufacturing process and obtains a respective high-resolution 3-D image of the respective product after completion of the respective additive manufacturing process. The process selects tiling maps that subdivide the low-resolution 2-D images and the high-resolution 3-D images into low-resolution tiles and high-resolution tiles, respectively. The process also builds an image enhancement generator iteratively in a generative adversarial network using training inputs that includes ordered pairs of low-resolution and high-resolution tiles. The process stores the image enhancement generator for subsequent use to enhance sequences of low-resolution 2-D images captured for products during additive manufacturing.
1. A method for enhancing image resolution for sequences of 2-D images of additively manufactured products, the method comprising:
for each of a plurality of additive manufacturing processes:
obtaining a respective plurality of sequenced low-resolution 2-D images of a respective product during the respective additive manufacturing process;
obtaining a respective high-resolution 3-D image of the respective product after completion of the respective additive manufacturing process, wherein the high-resolution 3-D image comprises a plurality of high-resolution 2-D images corresponding to the low resolution 2-D images;
selecting one or more tiling maps that subdivide each of the low-resolution 2-D images into a plurality of LR tiles and subdivide each of the corresponding high-resolution 2-D images into a plurality of corresponding HR tiles;
building an image enhancement generator iteratively in a generative adversarial network using training input comprising ordered pairs of corresponding LR tiles and HR tiles; and
storing the image enhancement generator for subsequent use to enhance sequences of low-resolution 2-D images captured for products during additive manufacturing,
wherein the method further comprises cropping and aligning the low-resolution 2-D images with the high-resolution 2-D images prior to subdividing into tiles.
2. The method of claim 1 , wherein each of the plurality of sequenced low-resolution 2-D images is a near-infrared (NIR) image of the respective product captured in a temporal sequence during the respective additive manufacturing process.
3. The method of claim 1 , wherein each of the high-resolution 3-D images is generated based on performing a micro-CT scan of the respective product after the respective additive manufacturing process is complete.
4. The method of claim 1 , wherein the generative adversarial network includes a first neural network comprising the image enhancement generator and a second neural network comprising a discriminator.
5. The method of claim 4 , wherein building the image enhancement generator iteratively comprises:
training the image enhancement generator to produce candidate high-resolution 2-D images based on low-resolution 2-D images; and
training the discriminator to distinguish between the candidate high-resolution 2-D images and 2-D slices of the obtained high-resolution 3-D images.
6. The method of claim 5 , wherein building the image enhancement generator ceases when output of the image enhancement generator is classified by the discriminator as a real high-resolution 3-D image for 50 percent of the candidate high-resolution 2-D images during multiple successive training iterations.
7. The method of claim 1 , further comprising augmenting the LR tiles and HR tiles in the training input by performing a warp transformation on some of the 2-D images.
8. The method of claim 1 , wherein the one or more tiling maps comprise a plurality of tiling maps, each subdividing according to a different pattern.
9. An electronic device for enhancing image resolution for sequences of 2-D images of additively manufactured products, comprising:
one or more processors; and
memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method of claim 1 .
10. A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the method of claim 1 .
11. A method for enhancing image resolution for sequences of 2-D images of additively manufactured products, the method comprising:
obtaining a plurality of temporally sequenced low-resolution 2-D images of a product during an in-progress additive manufacturing process;
obtaining an image enhancement generator previously trained as part of a generative adversarial network, wherein the image enhancement generator is configured to accept input images of a fixed 2-dimensional size;
selecting one or more tiling maps that subdivide each of the low-resolution 2-D images into a plurality of LR tiles;
for each of the LR tiles, applying the image enhancement generator to generate a high-resolution 2-D artificial image tile of the product;
stitching together the high-resolution 2-D artificial image tiles to form a set of high-resolution 2-D artificial layers corresponding to the low-resolution images; and
stacking together the high-resolution 2-D artificial layers to form a 3-D artificial volume of the product,
wherein the stitching comprises:
for each pixel included in two or more overlapping regions of the tiling map, generating a respective output image for the respective pixel by computing a respective weighted sum of values in the corresponding high-resolution 2-D artificial image tiles, and
wherein computing the respective weighted sum comprises:
associating, for each tile's contribution to the respective weighted sum, a respective weight that is linearly proportional to a distance from a center of the respective tile.
12. The method of claim 11 , wherein the generative adversarial network includes a first neural network comprising the image enhancement generator and a second neural network comprising a discriminator.
13. The method of claim 12 , wherein, during training:
the image enhancement generator is trained to generate candidate high-resolution 2-D images based on low-resolution 2-D images; and
the discriminator is trained to discriminate between the candidate high-resolution 2-D images and slices of real high-resolution 3-D images captured after additive manufacturing processes are complete.
14. The method of claim 11 , wherein obtaining the plurality of temporally sequenced low-resolution 2-D images comprises capturing a respective low-resolution 2-D image for each layer of the product during the in-progress additive manufacturing process.
15. The method of claim 11 , further comprising resizing the plurality of sequenced low-resolution 2-D images.
16. The method of claim 11 , wherein each tiling map subdivides each of the low-resolution 2-D images into non-overlapping tiles.
17. The method of claim 11 , wherein the one or more tiling maps comprise a plurality of tiling maps, each subdividing the low-resolution 2-D images according to a different pattern.
18. The method of claim 11 , further comprising:
converting the 3-D artificial volume of the product into a native CT-scan format.
19. The method of claim 11 , further comprising:
interpolating, using a trained neural network, between print layers of the in-progress additive manufacturing process.
20. The method of claim 11 , further comprising using the 3-D artificial volume to identify post-build effects or defects in the product.
21. An electronic device for enhancing image resolution for sequences of 2-D images of additively manufactured products, comprising:
one or more processors; and
memory storing one or more programs configured for execution by the one or more processors, the one or more programs including instructions for performing the method of claim 11 .
22. A non-transitory computer-readable storage medium storing one or more programs configured for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the method of claim 11 .