Machine-learning framework for detecting defects or conditions of railcar systems
Systems and methods for detecting defects or conditions of railcar systems are disclosed. Raw images depicting railcars can be received and curated into one or more training datasets each comprising at least some of the raw images. One or more machine-learning algorithms can be trained with the training dataset(s). A field image set comprising field images that depict at least a portion of an operating railcar and were obtained contemporaneously from multiple different viewpoints of the operating railcar can be received. A three-dimensional depiction of at least a portion of a railcar can be generated based on the field images. The machine-learning algorithm(s) can be applied to the field images and/or the three-dimensional depiction, and a condition of the operating railcar can be determined based at least in part on the field images and/or three-dimensional depiction.
1 . A computer-implemented method in which one or more processing devices perform operations comprising:
receiving a plurality of raw images depicting railcars;
curating one or more training datasets each comprising at least some of the plurality of raw images;
training one or more machine-learning algorithms with the one or more training datasets;
receiving a field image set comprising a plurality of field images depicting an operating railcar and obtained contemporaneously from a plurality of different viewpoints of the operating railcar;
applying the one or more machine-learning algorithms to the field image set to produce one or more machine-learning algorithm outputs;
determining a plurality of confidence scores, the plurality of confidence scores comprising a confidence score corresponding to the one or more machine-learning algorithm outputs for at least one field image corresponding to each of the plurality of different viewpoints of the operating railcar, wherein determining the plurality of confidence scores comprises:
for each of the plurality of different viewpoints of the operating railcar, averaging the confidence scores for all field images corresponding to a particular viewpoint of the plurality of different viewpoints; and
selecting a selected viewpoint from the plurality of different viewpoints based on a highest average confidence score; and
identifying a condition of the operating railcar based at least in part on the plurality of confidence scores.
2 . The computer-implemented method of claim 1 , wherein averaging the confidence scores comprises determining a weighted average.
3 . The computer-implemented method of claim 2 , wherein weights are assigned to each of the plurality of different viewpoints based at least on visibility of a particular portion of the operating railcar from each corresponding viewpoint of the plurality of different viewpoints.
4 . The computer-implemented method of claim 1 , wherein the one or more machine-learning algorithms comprises a localization algorithm, a classification algorithm, a pose estimation algorithm, a line segment detection algorithm, or a segmentation algorithm.
5 . A computer-implemented method in which one or more processing devices perform operations comprising:
receiving a plurality of raw images depicting railcars;
curating one or more training datasets by at least:
curating a first training dataset comprising a first grouping of the plurality of raw images and a first plurality of secondary images; and
curating a second training dataset comprising a second grouping of the plurality of raw images;
training one or more machine-learning algorithms by at least:
training a first machine-learning algorithm with the first training dataset; and
training a second machine-learning algorithm with the second training dataset;
receiving a field image set comprising a plurality of field images depicting an operating railcar and obtained contemporaneously from a plurality of different viewpoints of the operating railcar;
applying the one or more machine-learning algorithms to the field image set by at least:
applying the first machine-learning algorithm to the field image set to generate a first set of machine-learning algorithm outputs; and
applying the second machine-learning algorithm to the first set of machine-learning algorithm outputs to generate a second set of machine-learning algorithm outputs;
determining a plurality of confidence scores by at least determining a confidence score for each output of the second set of machine-learning algorithm outputs, wherein the plurality of confidence scores comprises a confidence score for at least one field image corresponding to each of the plurality of different viewpoints of the operating railcar; and
identifying a condition of the operating railcar based at least in part on the plurality of confidence scores.
6 . The computer-implemented method of claim 5 , wherein the second training dataset is different from the first training dataset.
7 . The computer-implemented method of claim 5 , wherein the second machine-learning algorithm is a different category of machine-learning algorithm from the first machine-learning algorithm.
8 . The computer-implemented method of claim 5 , wherein the second training dataset comprises a second plurality of secondary images that is different from the first plurality of secondary images.
9 . The computer-implemented method of claim 5 , further comprising generating a plurality of synthetic images using the plurality of raw images, wherein the first plurality of secondary images comprises the plurality of synthetic images.
10 . A computer-implemented method in which one or more processing devices perform operations comprising:
receiving a plurality of raw images depicting railcars, the plurality of raw images comprising a plurality of raw image sets, each raw image sets comprising images obtained contemporaneously from a plurality of different viewpoints of a given operating railcar;
generating a plurality of three-dimensional (3D) training representations, each corresponding to a railcar portion corresponding to each particular raw image set of the plurality of raw image sets;
curating one or more training datasets by at least:
curating a first training dataset comprising first 3D training representations based at least in part on a first grouping of the plurality of raw images and a first plurality of secondary images; and
curating a second training dataset comprising second 3D training representations based at least in part on a second grouping of the plurality of raw images;
training a first machine-learning algorithm with at least the first training dataset; and
training a second machine-learning algorithm with at least the second training dataset;
receiving a field image set comprising a plurality of field images depicting an operating railcar and obtained contemporaneously from a plurality of different viewpoints of the operating railcar;
generating a 3D field representation of the operating railcar based at least in part on the field image set;
applying the one or more machine-learning algorithms by at least:
applying the first machine-learning algorithm to the 3D field representation to generate a first set of machine-learning algorithm outputs; and
applying the second machine-learning algorithm to the first set of machine-learning algorithm outputs to generate a second set of machine-learning algorithm outputs; and
identifying a condition of the operating railcar based at least in part on the second set of machine-learning outputs.
11 . The computer-implemented method of claim 10 , wherein:
generating the 3D training representation comprises applying one or more photogrammetry methods to the plurality of raw image sets; and
generating the 3D field representation of the operating railcar comprising applying the one or more photogrammetry methods to the field image set.
12 . The computer-implemented method of claim 10 further comprising:
generating a plurality of synthetic image sets using at least some of the plurality of raw image sets; and
generating a plurality of three-dimensional (3D) synthetic training representations, each corresponding to a railcar portion corresponding to each particular synthetic image set of the plurality of synthetic image sets,
wherein the one or more training datasets comprises a first group of training data sets comprising at least some of the 3D training representations and second group of training data sets comprising the 3D synthetic training representations.
13 . The computer-implemented method of claim 10 further comprising selecting a representative image for corresponding to each viewpoint represented in the field image set by:
ranking each field image for a corresponding viewpoint based at least in part on one or more image quality factors; and
selecting a highest ranked field image for the corresponding viewpoint as the representative image for the corresponding viewpoint,
wherein the field image set comprises a plurality of representative images corresponding to each of the plurality of different viewpoints.
14 . The computer-implemented method of claim 10 , wherein the second training dataset is different from the first training dataset.
15 . The computer-implemented method of claim 10 , wherein the second machine-learning algorithm is a different category of machine-learning algorithm from the first machine-learning algorithm.
16 . The computer-implemented method of claim 10 , wherein the second training dataset comprises a second plurality of secondary images that is different from the first plurality of secondary images.
17 . The computer-implemented method of claim 10 , further comprising generating a plurality of synthetic images using the plurality of raw images, wherein the first plurality of secondary images comprises the plurality of synthetic images.
18 . The computer-implemented method of claim 10 , wherein the one or more machine-learning algorithms comprises a localization algorithm, a classification algorithm, a pose estimation algorithm, a line segment detection algorithm, or a segmentation algorithm.