Machine-learning framework for detecting defects or conditions of railcar systems
A computer-implemented method in which one or more processing devices perform operations may include obtaining a field image of a railcar collected from a field camera system and applying a machine-learning algorithm to the field image to generate a machine-learning algorithm output. The method may also include performing a post-processing operation on the machine-learning algorithm output to generate a filtered machine-learning algorithm output. Further, the method may include detecting a defect of the railcar using the filtered machine-learning algorithm output.
1 . A computer-implemented method in which one or more processing devices perform operations comprising:
training a defect detection system to identify defects in moving railcars by at least:
obtaining a plurality of raw images depicting railcars;
generating a first plurality of secondary images using at least the plurality of raw images, wherein the first plurality of secondary images is generated by applying image augmenting operations to the plurality of raw images;
curating a first training dataset comprising images from the plurality of raw images and the first plurality of secondary images;
training a first machine-learning algorithm with the first training dataset; and
training a second machine-learning algorithm with a second training dataset comprising images from the plurality of raw images and a second plurality of secondary images;
capturing, via a field camera system, a plurality of field images of a moving railcar travelling along a railway, at least one of the plurality of field images showing the moving railcar from a first angle and at least one other of the plurality of field images showing the moving railcar from a second angle that is different from the first angle;
for each particular field image of the plurality of field images:
applying the first machine-learning algorithm to the particular field image to generate one or more first machine-learning algorithm outputs;
applying the second machine-learning algorithm to at least one of the one or more first machine-learning algorithm outputs to generate one or more second machine-learning algorithm outputs; and
generating, based at least in part on at least one of the one or more second machine-learning algorithm outputs, a defect determination corresponding to the particular field image;
detecting a defect of the moving railcar based at least in part on a plurality of defect determinations corresponding to the plurality of field images; and
in response to detecting the defect, performing one or more remediation operations comprising at least one of:
instructing a reduction of a travel speed of the moving railcar; or
instructing a re-routing of the moving railcar.
2 . The computer-implemented method of claim 1 , wherein at least one of the first angle or the second angle corresponds to a side view of the moving railcar, a view of one or more wheels of the moving railcar, a view of a body of the moving railcar, a view of a coupling system between the moving railcar and an adjacent moving railcar, a view of one or more air hoses, a view of an undercarriage of the moving railcar, a view of one or more axles of the moving railcar, a view of one or more springs of the moving railcar, or a view of one or more braking systems of the moving railcar.
3 . The computer-implemented method of claim 1 , wherein at least one of the first angle or the second angle corresponds to aerial imaging collected by a drone system.
4 . The computer-implemented method of claim 1 , wherein the plurality of field images comprises field images collected at different locations along a railway.
5 . The computer-implemented method of claim 4 further comprising:
in response to detecting the defect of the moving railcar, outputting instructions for a subsequent field camera system to collect one or more subsequent field images to confirm detection of the defect.
6 . The computer-implemented method of claim 5 , wherein the instructions to the subsequent field camera system include instructions to focus imaging on a particular defect.
7 . The computer-implemented method of claim 1 , wherein at least one of the first machine-learning algorithm or the second machine-learning algorithm comprises a localization algorithm, a classification algorithm, a pose estimation algorithm, a line segment detection algorithm, or a segmentation algorithm.
8 . The computer-implemented method of claim 1 , further comprising:
generating a plurality of synthetic images using the plurality of raw images,
wherein at least one of the first training dataset or the second training dataset further comprises the plurality of synthetic images.
9 . The computer-implemented method of claim 1 , wherein:
the operations further comprise performing a post-processing operation on at least one of the one or more second machine-learning algorithm outputs to generate one or more post-processed machine-learning algorithm outputs; and
detecting the defect of the moving railcar is based at least in part on the one or more post-processed machine-learning algorithm outputs.
10 . The computer-implemented method of claim 1 , wherein the plurality of field images comprises (i) one or more side views of the moving railcar or a portion thereof and (ii) one or more undercarriage views of the moving railcar or a portion thereof.
11 . The computer-implemented method of claim 1 , wherein performing the one or more remediation operations further comprises at least one of:
ordering one or more replacement components associated with the defect;
initiating one or more additional inspections at one or more subsequent inspection locations; or
alerting an emergency response group of the defect and a location of the moving railcar.
12 . A system comprising:
a field camera system;
one or more processors in electrical communication with the field camera system; and
a non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by at least one processor of the one or more processors for performing operations comprising:
training a defect detection system to identify defects in moving railcars by at least:
obtaining a plurality of raw images depicting railcars;
generating a first plurality of secondary images using the plurality of raw images;
curating a first training dataset comprising images from the plurality of raw images and the first plurality of secondary images;
training a first machine-learning algorithm with the first training dataset; and
training a second machine-learning algorithm with a second training dataset comprising images from the plurality of raw images and a second plurality of secondary images;
instructing the field camera system to capture a plurality of field images of a moving railcar travelling along a railway, at least one of the plurality of field images showing the moving railcar from a first angle and at least one other of the plurality of field images showing the moving railcar from a second angle that is different from the first angle;
for each particular field image of the plurality of field images:
applying the first machine-learning algorithm to the particular field image to generate one or more first machine-learning algorithm outputs;
applying the second machine-learning algorithm to at least one of the one or more first machine-learning algorithm outputs to generate one or more second machine-learning algorithm outputs; and
generating, based at least in part on at least one of the one or more second machine-learning algorithm outputs, a defect determination corresponding to the particular field image;
detecting a defect of the moving railcar based at least in part on a plurality of defect determinations corresponding to the plurality of field images; and
in response to detecting the defect, performing one or more remediation operations comprising at least one of:
instructing a reduction of a travel speed of the moving railcar; or
instructing a re-routing of the moving railcar.
13 . The system of claim 12 , wherein at least one of the first angle or the second angle corresponds to a side view of the moving railcar, a view of one or more wheels of the moving railcar, a view of a body of the moving railcar, a view of a coupling system between the moving railcar and an adjacent moving railcar, a view of one or more air hoses, a view of an undercarriage of the moving railcar, a view of one or more axles of the moving railcar, a view of one or more springs of the moving railcar, or a view of one or more braking systems of the moving railcar.
14 . The system of claim 12 , wherein at least one of the first angle or the second angle corresponds to aerial imaging collected by a drone system.
15 . The system of claim 12 , wherein the plurality of field images comprises field images collected at different locations along a railway.
16 . The system of claim 15 , wherein the operations further comprise:
in response to detecting the defect of the moving railcar, outputting instructions for a subsequent field camera system to collect one or more subsequent field images to confirm detection of the defect.
17 . The system of claim 16 , wherein the instructions to the subsequent field camera system include instructions to focus imaging on a particular defect.
18 . The system of claim 12 , wherein at least one of the first machine-learning algorithm or the second machine-learning algorithm comprises a localization algorithm, a classification algorithm, a pose estimation algorithm, a line segment detection algorithm, or a segmentation algorithm.
19 . A non-transitory computer-readable storage medium having program code that is stored thereon, the program code being executable by one or more processing devices for performing operations comprising:
training a defect detection system to identify defects in moving railcars by at least:
obtaining a plurality of raw images depicting railcars;
generating a plurality of synthetic images using the plurality of raw images;
curating a first training dataset comprising images from the plurality of raw images and the plurality of synthetic images;
training a first machine-learning algorithm with the first training dataset; and
training a second machine-learning algorithm with a second training dataset comprising images from the plurality of raw images and the plurality of synthetic images;
instructing a field camera system to capture a plurality of field images of a moving railcar travelling along a railway, at least one of the plurality of field images showing the moving railcar from a first angle and at least one other of the plurality of field images showing the moving railcar from a second angle that is different from the first angle;
for each particular field image of the plurality of field images:
applying the first machine-learning algorithm to the particular field image to generate one or more first machine-learning algorithm outputs;
applying the second machine-learning algorithm to at least one of the one or more first machine-learning algorithm outputs to generate one or more second machine-learning algorithm outputs; and
generating, based at least in part on at least one of the one or more second machine-learning algorithm outputs, a defect determination corresponding to the particular field image;
detecting a defect of the moving railcar based at least in part on a plurality of defect determinations corresponding to the plurality of field images; and
in response to detecting the defect, performing one or more remediation operations comprising at least one of:
instructing a reduction of a travel speed of the moving railcar; or
instructing a re-routing of the moving railcar.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein at least one of the first angle or the second angle corresponds to a side view of the moving railcar, a view of one or more wheels of the moving railcar, a view of a body of the moving railcar, a view of a coupling system between the moving railcar and an adjacent moving railcar, a view of one or more air hoses, a view of an undercarriage of the moving railcar, a view of one or more axles of the moving railcar, a view of one or more springs of the moving railcar, or a view of one or more braking systems of the moving railcar.
21 . The non-transitory computer-readable storage medium of claim 19 , wherein the plurality of field images comprises field images collected at different locations along a railway.
22 . The non-transitory computer-readable storage medium of claim 21 , wherein the operations further comprise:
in response to detecting the defect of the moving railcar, outputting instructions for a subsequent field camera system to collect one or more subsequent field images to confirm detection of the defect.
23 . The non-transitory computer-readable storage medium of claim 22 , wherein the instructions to the subsequent field camera system include instructions to focus imaging on a particular defect.