Deep photometric learning (DPL) systems, apparatus and methods
An imaging system is disclosed herein. The imaging system includes an imaging apparatus and a computing system. The imaging apparatus includes a plurality of light sources positioned at a plurality of positions and a plurality of angles relative to a stage configured to support a specimen. The imaging apparatus is configured to capture a plurality of images of a surface of the specimen. The computing system in communication with the imaging apparatus. The computing system configured to generate a 3D-reconstruction of the surface of the specimen by receiving, from the imaging apparatus, the plurality of images of the surface of the specimen, generating, by the imaging apparatus via a deep learning model, a height map of the surface of the specimen based on the plurality of images, and outputting a 3D-reconstruction of the surface of the specimen based on the height map generated by the deep learning model.
1 . An imaging system, comprising:
an imaging apparatus comprising a plurality of light sources positioned at a plurality of positions and a plurality of angles relative to a stage configured to support a substrate, wherein the imaging apparatus is configured to capture a plurality of images of a microscopic surface of the substrate; and
a computing system in communication with the imaging apparatus, the computing system configured to generate a 3D-reconstruction of the microscopic surface of the substrate by:
receiving, from the imaging apparatus, the plurality of images of the microscopic surface of the substrate;
generating, by a deep learning model, a prediction of a local surface of the substrate based on the plurality of images;
generating, by the deep learning model, a height map of the microscopic surface of the substrate based on the prediction of the local surface; and
generating the 3D-reconstruction of the microscopic surface of the substrate based on the height map generated by the deep learning model;
wherein the deep learning model comprises at least one upsampling layer that directly feeds into a max pooling layer.
2 . The imaging system of claim 1 , wherein the deep learning model comprises a convolutional neural network.
3 . The imaging system of claim 2 , wherein the convolutional neural network is configured to receive, as input, the plurality of images and output a single image.
4 . The imaging system of claim 1 , wherein generating the height map based on the prediction comprises:
integrating a surface normal of the substrate to generate the height map.
5 . The imaging system of claim 1 , further comprising:
training the deep learning model to generate the height map based on variations in noise and tilt.
6 . A method of generating a 3D-reconstruction of a microscopic surface of a substrate, comprising:
receiving, by a computing system from an imaging apparatus, a plurality of images of a microscopic surface of a substrate, the plurality of images comprising images taken using plurality of light sources positioned at a plurality of positions and a plurality of angles relative to a stage configured to support the substrate;
generating, by a deep learning model, a prediction of a local surface of the substrate based on the plurality of images;
generating, by the deep learning model of the computing system, a height map of the microscopic surface of the substrate based on the prediction of the local surface; and
generating, by the computing system, a 3D-reconstruction of the microscopic surface of the substrate based on the height map generated by the deep learning model;
wherein the deep learning model comprises at least one upsampling layer that directly feeds into a max pooling layer.
7 . The method of claim 6 , wherein the deep learning model comprises a convolutional neural network.
8 . The method of claim 7 , wherein the convolutional neural network is configured to receive, as input, the plurality of images and output a single image.
9 . The method of claim 6 , wherein generating the height map based on the prediction comprises:
integrating a surface normal of the substrate to generate the height map.
10 . The method of claim 6 , further comprising:
training the deep learning model to generate the height map based on variations in noise and tilt.
11 . A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, cause a computing system to perform operations, comprising:
receiving, by the computing system from an imaging apparatus, a plurality of images of a microscopic surface of a substrate, the plurality of images comprising images taken using plurality of light sources positioned at a plurality of positions and a plurality of angles relative to a stage configured to support the substrate;
generating, by a deep learning model, a prediction of a local surface of the substrate based on the plurality of images;
generating, by the deep learning model of the computing system, a height map of the microscopic surface of the substrate based on the prediction of the local surface; and
generating, by the computing system, a 3D-reconstruction of the microscopic surface of the substrate based on the height map generated by the deep learning model;
wherein the deep learning model comprises at least one upsampling layer that directly feeds into a max pooling layer.
12 . The non-transitory computer readable medium of claim 11 , wherein the deep learning model comprises a convolutional neural network configured to receive, as input, the plurality of images and output a single image.
13 . The non-transitory computer readable medium of claim 11 , wherein generating the height map based on the prediction comprises:
integrating a surface normal of the substrate to generate the height map.
14 . The non-transitory computer readable medium of claim 11 , further comprising:
training the deep learning model to generate the height map based on variations in noise and tilt.