Zero-shot referring segmentation for defect detection in visual inspection with LLM-generated prompts
Methods, systems, and computer-readable storage media for receiving a product image depicting a product that is to-be-inspected for defects, transmitting a request to a LLM system, the request including the product image and a reference image, receiving, from the LLM system, a textual response, the textual response being generated by the LLM system at least partially by processing the product image and the reference image, processing the textual response and the product image through a RIS model to provide an output image that includes one or more masks, each mask indicating a defect in a product depicted in the product image, and providing the output image with an indication that the product is defective.
1 . A computer-implemented method for automated visual inspection of products for defects, the method being executed by one or more processors and comprising:
receiving a product image depicting a product that is to-be-inspected for defects;
transmitting a request to a large language model (LLM) system, the request comprising the product image and a reference image;
receiving, from the LLM system, a textual response, the textual response being generated by the LLM system at least partially by processing the product image and the reference image;
processing the textual response and the product image through a referring image segmentation (RIS) model to provide an output image that includes one or more masks, each mask indicating a defect in a product depicted in the product image; and
providing the output image with an indication that the product is defective.
2 . The method of claim 1 , wherein the request to the LLM system further comprises a prompt requesting the LLM system to describe any differences between the product image and the reference image.
3 . The method of claim 1 , wherein the RIS model is provided as a pre-trained model.
4 . The method of claim 3 , further comprising executing zero-shot learning of the RIS model using one or more of labeled training images and unlabeled training images.
5 . The method of claim 1 , further comprising providing one or more labels with the output image, each label indicating a respective defect type.
6 . The method of claim 1 , wherein processing the textual response and the product image through a RIS model to provide an output image is executed in response to determining that the textual response from the LLM system indicates a difference between the product image and the reference image.
7 . The method of claim 1 , wherein the LLM system executes a LLM that is enabled to analyze visual inputs.
8 . A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for automated visual inspection of products for defects, the operations comprising:
receiving a product image depicting a product that is to-be-inspected for defects;
transmitting a request to a large language model (LLM) system, the request comprising the product image and a reference image;
receiving, from the LLM system, a textual response, the textual response being generated by the LLM system at least partially by processing the product image and the reference image;
processing the textual response and the product image through a referring image segmentation (RIS) model to provide an output image that includes one or more masks, each mask indicating a defect in a product depicted in the product image; and
providing the output image with an indication that the product is defective.
9 . The non-transitory computer-readable storage medium of claim 8 , wherein the request to the LLM system further comprises a prompt requesting the LLM system to describe any differences between the product image and the reference image.
10 . The non-transitory computer-readable storage medium of claim 8 , wherein the RIS model is provided as a pre-trained model.
11 . The non-transitory computer-readable storage medium of claim 10 , wherein operations further comprise executing zero-shot learning of the RIS model using one or more of labeled training images and unlabeled training images.
12 . The non-transitory computer-readable storage medium of claim 8 , wherein operations further comprise providing one or more labels with the output image, each label indicating a respective defect type.
13 . The non-transitory computer-readable storage medium of claim 8 , wherein processing the textual response and the product image through a RIS model to provide an output image is executed in response to determining that the textual response from the LLM system indicates a difference between the product image and the reference image.
14 . The non-transitory computer-readable storage medium of claim 8 , wherein the LLM system executes a LLM that is enabled to analyze visual inputs.
15 . A system, comprising:
a computing device; and
a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for automated visual inspection of products for defects, the operations comprising:
receiving a product image depicting a product that is to-be-inspected for defects;
transmitting a request to a large language model (LLM) system, the request comprising the product image and a reference image;
receiving, from the LLM system, a textual response, the textual response being generated by the LLM system at least partially by processing the product image and the reference image;
processing the textual response and the product image through a referring image segmentation (RIS) model to provide an output image that includes one or more masks, each mask indicating a defect in a product depicted in the product image; and
providing the output image with an indication that the product is defective.
16 . The system of claim 15 , wherein the request to the LLM system further comprises a prompt requesting the LLM system to describe any differences between the product image and the reference image.
17 . The system of claim 15 , wherein the RIS model is provided as a pre-trained model.
18 . The system of claim 17 , wherein operations further comprise executing zero-shot learning of the RIS model using one or more of labeled training images and unlabeled training images.
19 . The system of claim 15 , wherein operations further comprise providing one or more labels with the output image, each label indicating a respective defect type.
20 . The system of claim 15 , wherein processing the textual response and the product image through a RIS model to provide an output image is executed in response to determining that the textual response from the LLM system indicates a difference between the product image and the reference image.