Machine learning system, learning data collection method and storage medium
A machine learning system in the present disclosure includes: an image pickup unit that photographs a product and acquires a product image; a preprocessing unit that generates an inspection target site image by clipping an image of an inspection target site of the product based on a setting file, and saves the generated inspection target site image in an image saving unit; and an inspection processing unit that performs a quality determination process on the inspection target site image of a quality determination object indicated as a quality determination target by production instruction information, in which when the inspection target site image saved in the image saving unit is relevant to a product that is designated by the production instruction information as a learning object that is not the quality determination target, the inspection target site image is accumulated in the image saving unit, as learning data.
1 . A machine learning system comprising:
a camera configured to acquire a product image by imaging a product;
a first memory; and
a processor configured to generate an inspection target site image by clipping, from the product image, images of a plurality of inspection target sites, including an image of an inspection target site of the plurality of inspection target sites, of the product based on a setting file that indicates positions and ranges of the plurality of inspection target sites on the product image, and to save the images of the plurality of inspection target sites in the first memory, the positions and ranges including a position and range of the inspection target site on the product image;
wherein the processor is further configured to:
determine, based on production instruction information in which a specification of the product is described, whether the inspection target site has been learned by any of one or more pre-trained models as any of a plurality of quality determination objects;
perform, based on determining that the inspection target site has been learned by a first model of the one or more pre-trained models as a quality determination object of the plurality of quality determination objects, a quality determination process on the inspection target site image as the quality determination object, the quality determination process being a process to which artificial intelligence is applied by the first model;
determine, based on the production instruction information, whether a second inspection target site, of the plurality of inspection target sites, has been learned by any of the one or more pre-trained models as any of the plurality of quality determination objects; and
generate, based on determining that the second inspection target site has not been learned by any of the one or more pre-trained models, a learning model that is used for quality determination for the second inspection target site as a learning object, the learning model being a model to which learning data, comprising a second image, is input and in which the production instruction information relevant to the learning object is used as teaching data, the second image being of the second inspection target site from the images of the plurality of inspection target sites clipped from the product image acquired by the camera,
wherein the processor is further configured to determine whether to accumulate, in the first memory, the second image of the second inspection target site as learning data of the learning model based on the production instruction information indicating the second inspection target site as a learning object that is not a quality determination target, among the images of the plurality of inspection target sites in the first memory,
wherein the production instruction information includes a plurality of inspection instruction flags, each of the plurality of inspection instruction flags corresponding to a respective inspection target site of the plurality of inspection target sites and indicating the respective inspection target site as one of an inspection target status and an unlearned status, the inspection target status indicating that at least one of the one or more pre-trained models have been pre-trained for inspection of the respective inspection target site, and the unlearned status indicating that none of the one or more pre-trained models have been pre-trained for inspection of the respective inspection target site, and
wherein the processor is further configured to:
read the inspection target site image based on the inspection target site image being saved in the first memory;
determine the first model as corresponding to the inspection target site image;
perform the quality determination process based on determining the first model as corresponding to the inspection target site image; and
skip the quality determination process for the second inspection target site based on determining that an inspection instruction flag, of the plurality of inspection instruction flags, corresponding to the second inspection target site indicates the unlearned status in the production instruction information.
2 . The machine learning system according to claim 1 , wherein the processor saves the inspection target site image in the first memory, in association with the production instruction information.
3 . The machine learning system according to claim 1 , further comprising:
a second memory,
wherein the processor reads the setting file corresponding to the product based on the production instruction information, from the second memory, and saves the inspection target site image in a file path described in the setting file.
4 . The machine learning system according to claim 1 , wherein the product is a vehicle or a part of the vehicle.
5 . The machine learning system according to claim 1 , wherein the machine learning system is set so as to perform the quality determination process on the product on a production line.
6 . The machine learning system according to claim 1 ,
wherein the inspection target site indicates any of a vehicle type of the product, a paint color of the product, an emblem of the product, a side grill of the product, a plate of the product, and a clearance sonar of the product,
wherein the second inspection target site indicates any of a harness of the product, a label of the product, and a sensor of the product,
wherein ones of the plurality of inspection instruction flags respectively correspond to ones of the vehicle type of the product, the paint color of the product, the emblem of the product, the side grill of the product, the plate of the product, the clearance sonar of the product, the harness of the product, the label of the product, and the sensor of the product.
7 . A method for collecting learning data using an inspection device that performs a quality determination process of a product on a production line, the learning data being used for creation of a learning model that is applied to artificial intelligence, the method comprising:
an image pickup process of imaging, by a camera, the product as the product is conveyed on the production line, and acquiring a product image based on imaging the product;
a preprocess of generating an inspection target site image by clipping, from the product image, images of a plurality of inspection target sites, including an image of an inspection target site of the plurality of inspection target sites, of the product based on a setting file that indicates positions and ranges of the plurality of inspection target sites on the product image, and saving the images of the plurality of inspection target sites in a memory, the positions and ranges including a position and range of the inspection target site on the product image;
determining, based on production instruction information in which a specification of the product is described, whether the inspection target site has been learned by any of one or more pre-trained models as any of a plurality of quality determination objects;
performing, based on determining that the inspection target site has been learned by a first model of the one or more pre-trained models as a quality determination object of the plurality of quality determination objects, the quality determination process on the inspection target site image as the quality determination object, the quality determination process being a process to which artificial intelligence is applied by the first model;
determining, based on the production instruction information, whether a second inspection target site, of the plurality of inspection target sites, has been learned by any of the one or more pre-trained models as any of the plurality of quality determination objects;
generating, based on determining that the second inspection target site has not been learned by any of the one or more pre-trained models, a first learning model that is used for quality determination for the second inspection target site as a learning object, the first learning model being a model to which learning data, comprising a second image, is input and in which the production instruction information relevant to the learning object is used as teaching data, the second image being of the second inspection target site from the images of the plurality of inspection target sites clipped from the product image acquired by the camera; and
determining whether to accumulate, in the memory, the second image of the second inspection target site as learning data of the first learning model based on the production instruction information indicating the second inspection target site as a learning object that is not a quality determination target, among the images of the plurality of inspection target sites in the memory,
wherein the production instruction information includes a plurality of inspection instruction flags, each of the plurality of inspection instruction flags corresponding to a respective inspection target site of the plurality of inspection target sites and indicating the respective inspection target site as one of an inspection target status and an unlearned status, the inspection target status indicating that at least one of the one or more pre-trained models have been pre-trained for inspection of the respective inspection target site, and the unlearned status indicating that none of the one or more pre-trained models have been pre-trained for inspection of the respective inspection target site, and
wherein the method further comprises:
reading the inspection target site image based on the inspection target site image being saved in the memory;
determining the first model as corresponding to the inspection target site image;
performing the quality determination process based on determining the first model as corresponding to the inspection target site image; and
skipping the quality determination process for the second inspection target site based on determining that an inspection instruction flag, of the plurality of inspection instruction flags, corresponding to the second inspection target site indicates the unlearned status in the production instruction information.
8 . A non-transitory storage medium storing a learning data collection program that collects learning data using an inspection device that performs a quality determination process of a product on a production line, the learning data collection program being executed by a computation unit provided in the inspection device, the learning data being used for creation of a learning model that is applied to artificial intelligence, the learning data collection program comprising:
an image pickup process of imaging, by a camera, the product as the product is conveyed on the production line, and acquiring a product image based on imaging the product;
a preprocess of generating an inspection target site image by clipping, from the product image, images of a plurality of inspection target sites, including an image of an inspection target site of the plurality of inspection target sites, of the product based on a setting file that indicates positions and ranges of the plurality of inspection target sites on the product image, and saving the images of the plurality of inspection target sites in a memory, the positions and ranges including a position and range of the inspection target site on the product image;
determining, based on production instruction information in which a specification of the product is described, whether the inspection target site has been learned by any of one or more pre-trained models as any of a plurality of quality determination objects;
performing, based on determining that the inspection target site has been learned by a first model of the one or more pre-trained models as a quality determination object of the plurality of quality determination objects, the quality determination process on the inspection target site image as the quality determination object, the quality determination process being a process to which artificial intelligence is applied by the first model;
determining, based on the production instruction information, whether a second inspection target site, of the plurality of inspection target sites, has been learned by any of the one or more pre-trained models as any of the plurality of quality determination objects;
generating, based on determining that the second inspection target site has not been learned by any of the one or more pre-trained models, a first learning model that is used for quality determination for the second inspection target site as a learning object, the first learning model being a model to which learning data, comprising a second image, is input and in which the production instruction information relevant to the learning object is used as teaching data, the second image being of the second inspection target site from the images of the plurality of inspection target sites clipped from the product image acquired by the camera; and
determining whether to accumulate, in the memory, the second image of the second inspection target site as learning data of the first learning model based on the production instruction information indicating the second inspection target site as a learning object that is not a quality determination target, among the images of the plurality of inspection target sites in the memory,
wherein the production instruction information includes a plurality of inspection instruction flags, each of the plurality of inspection instruction flags corresponding to a respective inspection target site of the plurality of inspection target sites and indicating the respective inspection target site as one of an inspection target status and an unlearned status, the inspection target status indicating that at least one of the one or more pre-trained models have been pre-trained for inspection of the respective inspection target site, and the unlearned status indicating that none of the one or more pre-trained models have been pre-trained for inspection of the respective inspection target site, and
wherein the learning data collection program further comprises:
reading the inspection target site image based on the inspection target site image being saved in the memory;
determining the first model as corresponding to the inspection target site image;
performing the quality determination process based on determining the first model as corresponding to the inspection target site image; and
skipping the quality determination process for the second inspection target site based on determining that an inspection instruction flag, of the plurality of inspection instruction flags, corresponding to the second inspection target site indicates the unlearned status in the production instruction information.